下载文档原格式
基于TF—IDF算法的研究与应用TF-IDF（Term Frequency-Inverse Document Frequency）算法是一种用于信息检索和文本挖掘的常用算法，它能够帮助我们分析文本中关键词的重要性，并用于文本相似度计算、关键词提取、文本分类等领域。
本文将对TF-IDF算法的原理以及在实际应用中的研究和应用进行介绍。
一、TF-IDF算法原理TF-IDF算法是一种用于衡量一个词在文本中的重要性的指标，其计算公式如下所示：TF（词频）= 某个词在文本中出现的次数 / 该文本的总词数IDF（逆文档频率）= log（语料库中文档总数 / 含有该词的文档数+1）TF-IDF = TF * IDF在这个公式中，TF用于衡量某个词在文本中的重要程度，而IDF用于衡量该词在整个语料库中的重要程度。
通过这个公式，我们可以得到一个词在文本中的TF-IDF值，从而确定其在文本中的重要性。
1. 文本相似度计算TF-IDF算法可以用于计算两个文本之间的相似度，通过比较它们的关键词的TF-IDF 值，我们可以得出它们之间的相似程度。
这对于文本匹配、信息检索等领域非常有用，可以帮助我们快速找到相关的文档。
2. 关键词提取在文本挖掘和自然语言处理领域，我们经常需要从大量的文本中提取关键词。
TF-IDF 算法可以帮助我们确定文本中的关键词，通过计算每个词的TF-IDF值，我们可以找到在文本中最重要的词语，从而实现关键词提取的目的。
3. 文本分类1. 搜索引擎搜索引擎是TF-IDF算法最典型的应用场景之一，它通过分析用户输入的关键词，并在文档集合中计算每个词的TF-IDF值，从而找到最相关的文档并呈现给用户。
通过TF-IDF 算法，搜索引擎可以实现准确的文本匹配和相关性排序，提高搜索结果的质量。
2. 新闻推荐系统在新闻推荐系统中，我们需要根据用户的兴趣推荐相关的新闻文章。
TF-IDF算法可以用于分析用户的浏览历史和新闻文章的内容，通过计算关键词的TF-IDF值来确定用户的兴趣，从而实现个性化的新闻推荐。
NLP之关键词提取（TF-IDF、Text-Rank）1.⽂本关键词抽取的种类：关键词提取⽅法分为有监督、半监督和⽆监督三种，有监督和半监督的关键词抽取⽅法需要浪费⼈⼒资源，所以现在使⽤的⼤多是⽆监督的关键词提取⽅法。
⽆监督的关键词提取⽅法⼜可以分为三类：基于统计特征的关键词抽取、基于词图模型的关键词抽取和基于主题模型的关键词抽取。
2.基于统计特征的有个最简单的⽅法，利⽤TF-IDF效果不错对于未登录词其IDF值的常⽤计算以及TF-IDF的计算3、TD-IDF的主要思想以及优缺点主要思想：tf-idf 模型的主要思想是：如果词w在⼀篇⽂档d中出现的频率⾼，并且在其他⽂档中很少出现，则认为词w具有很好的区分能⼒，适合⽤来把⽂章d和其他⽂章区分开来。
TF-IDF算法的优点是简单快速，结果⽐较符合实际情况。
缺点是，单纯以"词频"衡量⼀个词的重要性，不够全⾯，有时重要的词可能出现次数并不多。
⽽且，这种算法⽆法体现词的位置信息，出现位置靠前的词与出现位置靠后的词，都被视为重要性相同，这是不正确的。
IDF的简单结构并不能有效地反映单词的重要程度和特征词的分布情况，使其⽆法很好地完成对权值调整的功能。
4、基于词图模型的介绍⼀个TextRank具体参考：说到TextRank要先介绍PageRank：PageRank算法 PageRank设计之初是⽤于Google的⽹页排名的，以该公司创办⼈拉⾥·佩奇（Larry Page）之姓来命名。
Google⽤它来体现⽹页的相关性和重要性，在搜索引擎优化操作中是经常被⽤来评估⽹页优化的成效因素之⼀。
PageRank通过互联⽹中的超链接关系来确定⼀个⽹页的排名，其公式是通过⼀种投票的思想来设计的：如果我们要计算⽹页A的PageRank值（以下简称PR值），那么我们需要知道有哪些⽹页链接到⽹页A，也就是要⾸先得到⽹页A的⼊链，然后通过⼊链给⽹页A的投票来计算⽹页A的PR值。
自然语言处理的关键词提取方法自然语言处理（Natural Language Processing, NLP）是人工智能领域的重要分支，旨在使计算机能够理解和处理人类语言。
在NLP中，关键词提取是一项关键任务，它可以帮助我们从大量的文本数据中提取出最具代表性和重要性的关键词，从而更好地理解文本内容和进行后续的分析。
关键词提取方法有很多种，下面将介绍几种常见的方法。
一、基于统计的关键词提取方法基于统计的关键词提取方法是一种常见且有效的方法。
它通过统计文本中词语的频率和分布情况来确定关键词。
其中，TF-IDF（Term Frequency-Inverse Document Frequency）是一种常用的统计指标。
TF指的是词语在文本中的频率，IDF指的是词语在整个语料库中的逆文档频率。
通过计算TF和IDF的乘积，可以得到一个词语的重要性分数，从而确定关键词。
二、基于机器学习的关键词提取方法基于机器学习的关键词提取方法是近年来发展起来的一种方法。
它通过训练机器学习模型来识别和提取关键词。
常用的机器学习算法包括朴素贝叶斯、支持向量机和深度学习等。
这些算法可以通过学习大量的标注数据来建立关键词提取模型，并利用模型对新的文本进行关键词提取。
三、基于语义的关键词提取方法基于语义的关键词提取方法是一种较为高级的方法。
它通过理解词语之间的语义关系来确定关键词。
其中，词向量是一种常用的语义表示方法。
词向量可以将词语表示为一个向量，使得具有相似语义的词语在向量空间中距离较近。
通过计算词语之间的相似度，可以确定关键词。
四、基于图论的关键词提取方法基于图论的关键词提取方法是一种基于网络结构的方法。
它通过构建文本的图模型，将词语作为节点，词语之间的关系作为边，从而建立一个词语网络。
通过分析节点之间的连接关系和节点的重要性，可以确定关键词。
常用的图算法包括PageRank算法和TextRank算法等。
综上所述，关键词提取是自然语言处理中的重要任务之一。
pythonTF-IDF算法实现⽂本关键词提取TF（Term Frequency）词频，在⽂章中出现次数最多的词，然⽽⽂章中出现次数较多的词并不⼀定就是关键词，⽐如常见的对⽂章本⾝并没有多⼤意义的停⽤词。
所以我们需要⼀个重要性调整系数来衡量⼀个词是不是常见词。
该权重为IDF（Inverse Document Frequency）逆⽂档频率，它的⼤⼩与⼀个词的常见程度成反⽐。
在我们得到词频（TF）和逆⽂档频率（IDF）以后，将两个值相乘，即可得到⼀个词的TF-IDF值，某个词对⽂章的重要性越⾼，其TF-IDF值就越⼤，所以排在最前⾯的⼏个词就是⽂章的关键词。
TF-IDF算法的优点是简单快速，结果⽐较符合实际情况，但是单纯以“词频”衡量⼀个词的重要性，不够全⾯，有时候重要的词可能出现的次数并不多，⽽且这种算法⽆法体现词的位置信息，出现位置靠前的词和出现位置靠后的词，都被视为同样重要，是不合理的。
TF-IDF算法步骤：（1）、计算词频：词频 = 某个词在⽂章中出现的次数考虑到⽂章有长短之分，考虑到不同⽂章之间的⽐较，将词频进⾏标准化词频 = 某个词在⽂章中出现的次数/⽂章的总词数词频 = 某个词在⽂章中出现的次数/该⽂出现次数最多的词出现的次数（2）、计算逆⽂档频率需要⼀个语料库（corpus）来模拟语⾔的使⽤环境。
逆⽂档频率 = log(语料库的⽂档总数/(包含该词的⽂档数 + 1))（3）、计算TF-IDFTF-IDF = 词频（TF）* 逆⽂档频率（IDF）详细代码如下：#!/usr/bin/env python#-*- coding:utf-8 -*-'''计算⽂档的TF-IDF'''import codecsimport osimport mathimport shutil#读取⽂本⽂件def readtxt(path):with codecs.open(path,"r",encoding="utf-8") as f:content = f.read().strip()return content#统计词频def count_word(content):word_dic ={}words_list = content.split("/")del_word = ["\r\n","/s"," ","/n"]for word in words_list:if word not in del_word:if word in word_dic:word_dic[word] = word_dic[word]+1else:word_dic[word] = 1return word_dic#遍历⽂件夹def funfolder(path):filesArray = []for root,dirs,files in os.walk(path):for file in files:each_file = str(root+"//"+file)filesArray.append(each_file)return filesArray#计算TF-IDFdef count_tfidf(word_dic,words_dic,files_Array):word_idf={}word_tfidf = {}num_files = len(files_Array)for word in word_dic:for words in words_dic:if word in words:if word in word_idf:word_idf[word] = word_idf[word] + 1else:word_idf[word] = 1for key,value in word_dic.items():if key !=" ":word_tfidf[key] = value * math.log(num_files/(word_idf[key]+1))#降序排序values_list = sorted(word_tfidf.items(),key = lambda item:item[1],reverse=True)return values_list#新建⽂件夹def buildfolder(path):if os.path.exists(path):shutil.rmtree(path)os.makedirs(path)print("成功创建⽂件夹！")#写⼊⽂件def out_file(path,content_list):with codecs.open(path,"a",encoding="utf-8") as f:for content in content_list:f.write(str(content[0]) + ":" + str(content[1])+"\r\n")print("well done!")def main():#遍历⽂件夹folder_path = r"分词结果"files_array = funfolder(folder_path)#⽣成语料库files_dic = []for file_path in files_array:file = readtxt(file_path)word_dic = count_word(file)files_dic.append(word_dic)#新建⽂件夹new_folder = r"tfidf计算结果"buildfolder(new_folder)#计算tf-idf,并将结果存⼊txti=0for file in files_dic:tf_idf = count_tfidf(file,files_dic,files_array)files_path = files_array[i].split("//")#print(files_path)outfile_name = files_path[1]#print(outfile_name)out_path = r"%s//%s_tfidf.txt"%(new_folder,outfile_name)out_file(out_path,tf_idf)i=i+1if __name__ == '__main__':main()以上就是本⽂的全部内容，希望对⼤家的学习有所帮助，也希望⼤家多多⽀持。
人工智能与自然语言处理中的关键词提取方法教程关键词提取是自然语言处理中的重要任务之一。
它是通过分析文本的语义和上下文关系来确定文本最重要和具有代表性的单词或短语。
关键词提取方法可以应用于各种领域，如文本摘要、信息检索和情感分析等。
本篇文章将介绍人工智能与自然语言处理中常用的关键词提取方法，并提供相应的教程，帮助读者了解和应用这些方法。
1. 基于频率统计的关键词提取基于频率统计的关键词提取方法是最简单直接的方法之一。
它通过统计文本中词语出现的频率来确定关键词。
常用的统计指标包括词频（TF，Term Frequency）、文档频率（DF，Document Frequency）和逆文档频率（IDF，Inverse Document Frequency）等。
教程：首先，将要处理的文本进行分词，将文本划分为词语的序列。
然后，统计每个词语在文本中的出现次数，并计算词频。
接下来，计算每个词语的文档频率，即在多少个文档中出现过该词语。
最后，综合考虑词频和文档频率，计算每个词语的TF-IDF值，选取TF-IDF值较高的词语作为关键词。
2. 基于词性标注的关键词提取词性标注是指将词语按照其在句子中的语法和语义功能进行分类的过程。
基于词性标注的关键词提取方法是通过分析词性来确定文本中的关键词。
例如，在新闻报道中，名词往往是最具有信息量和代表性的词语。
教程：首先，对文本进行分词和词性标注。
这可以使用中文分词工具和词性标注工具来实现。
接下来，筛选出具有代表性的词性类别，如名词、动词、形容词等。
最后，根据筛选出的词性类别，选择具有代表性的词语作为关键词。
3. 基于机器学习的关键词提取基于机器学习的关键词提取方法是利用训练数据构建模型，通过模型学习数据中的关键词特征，然后对新文本进行预测得到关键词。
常用的机器学习算法包括支持向量机（SVM）、朴素贝叶斯（Naive Bayes）和深度学习等。
教程：首先，准备训练数据集。
训练数据集应包含已标注好的文本和对应的关键词。
关键词提取方法关键词提取是信息检索、文本挖掘和自然语言处理等领域一个重要的任务。
在大量的文本数据中，提取关键词可以帮助人们快速了解文本的主题和内容，从而更高效地进行信息查找和分析。
本文将介绍几种常见的关键词提取方法，并探讨它们的优缺点。
1. TF-IDF（词频-逆文档频率）TF-IDF是一种经典的关键词提取方法，它根据词在文档中的出现频率和在整个文集中的逆文档频率来计算每个词的权重。
TF-IDF的核心思想是，一个词在当前文档中出现次数较多，并且在其他文档中出现较少，那么它很可能是关键词。
TF-IDF的计算公式如下：TF-IDF = TF * IDF其中，TF表示词频，即某个词在当前文档中出现的次数。
IDF表示逆文档频率，它衡量了一个词的普遍重要性。
IDF的计算公式如下：IDF = log(N / (n + 1))其中，N表示文档总数，n表示包含该词的文档数。
使用TF-IDF方法可以得到每个词的权重，根据权重进行排名即可得到关键词。
2. TextRank（基于图的排名算法）TextRank是一种基于图的关键词提取方法，它是PageRank算法在文本中的应用扩展。
TextRank通过构建词语之间的共现关系图，并利用图的节点之间的关系进行关键词提取。
TextRank的基本思路是，将文本分为若干个单词或短语作为节点，然后根据它们之间的关系构建图。
共现关系指的是两个单词在文本中同时出现的次数。
利用共现关系，可以计算出每个单词的重要性。
重要性的计算可以使用PageRank算法，即根据每个节点与其他节点之间的连接关系进行迭代计算。
TextRank方法的优点是可以在不依赖于外部语料库的情况下进行关键词提取，而且可以捕捉到文本中的词义和上下文信息。
然而，TextRank方法也有一些限制，例如对于长文本的处理效果不如短文本，以及对于同义词和多义词的处理较为困难。
3. LDA（潜在狄利克雷分配）LDA是一种概率图模型，常用于主题建模和文档相似度计算。
（三）基于tfidf和textrank关键字提取前⾔关键词提取就是从⽂本⾥⾯把跟这篇⽂章意义最相关的⼀些词语抽取出来。
这个可以追溯到⽂献检索初期，关键词是为了⽂献标引⼯作，从报告、论⽂中选取出来⽤以表⽰全⽂主题内容信息的单词或术语，在现在的报告和论⽂中，我们依然可以看到关键词这⼀项。
因此，关键词在⽂献检索、⾃动⽂摘、⽂本聚类/分类等⽅⾯有着重要的应⽤，它不仅是进⾏这些⼯作不可或缺的基础和前提，也是互联⽹上信息建库的⼀项重要⼯作。
关键词抽取从⽅法来说主要有两种：第⼀种是关键词分配：就是给定⼀个已有的关键词库，对于新来的⽂档从该词库⾥⾯匹配⼏个词语作为这篇⽂档的关键词。
第⼆种是关键词提取：针对新⽂档，通过算法分析，提取⽂档中⼀些词语作为该⽂档的关键词。
⽬前⼤多数应⽤领域的关键词抽取算法都是基于后者实现的，从逻辑上说，后者⽐前者在实际应⽤中更准确。
下⾯介绍⼀些关于关键词抽取的常⽤和经典的算法实现。
基于 TF-IDF 算法进⾏关键词提取在信息检索理论中，TF-IDF 是 Term Frequency - Inverse Document Frequency 的简写。
TF-IDF 是⼀种数值统计，⽤于反映⼀个词对于语料中某篇⽂档的重要性。
在信息检索和⽂本挖掘领域，它经常⽤于因⼦加权。
TF-IDF 的主要思想就是：如果某个词在⼀篇⽂档中出现的频率⾼，也即 TF ⾼；并且在语料库中其他⽂档中很少出现，即 DF 低，也即IDF ⾼，则认为这个词具有很好的类别区分能⼒。
TF 为词频（Term Frequency），表⽰词 t 在⽂档 d 中出现的频率，计算公式：其中，ni,j 是该词 ti 在⽂件 dj 中的出现次数，⽽分母则是在⽂件 dj 中所有字词的出现次数之和。
IDF 为逆⽂档频率（Inverse Document Frequency），表⽰语料库中包含词 t 的⽂档的数⽬的倒数，计算公式：其中，|D|表⽰语料库中的⽂件总数，|{j:ti∈dj}| 包含词 ti 的⽂件数⽬，如果该词语不在语料库中，就会导致被除数为零，因此⼀般情况下使⽤ 1+|{j:ti∈dj}|。
使用Python和TF-IDF算法进行关键词提取TF-IDF是一种文本分析和信息检索中广泛使用的技术，可以帮助我们自动提取文本中的关键词，从而更好地理解文本内容。
本文将介绍TF-IDF算法的原理、计算公式和实际应用，帮助您理解并应用这一强大的文本分析工具。
TF-IDF算法的计算公式TF-IDF算法的核心思想是根据一个词在文档中的频率（Term Frequency，TF）和在整个语料库中的逆文档频率（Inverse Document Frequency，IDF）来衡量词的重要性。
TF衡量了一个词在文档中的重要性，而IDF衡量了一个词在整个语料库中的重要性。
TF-IDF的计算公式如下：TF（词频）计算公式TF(t, d) = (词t在文档d中出现的次数) / (文档d中的总词数)IDF（逆文档频率）计算公式IDF(t) = log(语料库中的文档总数/ 包含词t的文档数+ 1)TF-IDF计算公式TF-IDF(t, d) = TF(t, d) * IDF(t)其中，t代表关键词，d代表文档。
实例代码下面是一个使用Python编写的TF-IDF算法的示例代码：from sklearn.feature_extraction.text import TfidfVectorizer# 语料库corpus = ["TF-IDF是一种用于文本分析的重要算法。
","通过TF-IDF，我们可以提取文本中的关键词。
","关键词提取有助于文本的信息检索和摘要生成。
"]# 创建TF-IDF向量化器tfidf_vectorizer = TfidfVectorizer()# 对文本进行向量化tfidf_matrix = tfidf_vectorizer.fit_transform(corpus)# 输出关键词和对应的TF-IDF值feature_names = tfidf_vectorizer.get_feature_names_out() for i, doc in enumerate(corpus):print(f"文档{i + 1}:")for j, feature in enumerate(feature_names):print(f"{feature}: {tfidf_matrix[i, j]}")详解TF（词频）计算TF计算是基于文档内部的，它衡量了一个词在文档中的出现频率。
自然语言处理（Natural Language Processing, NLP）是人工智能领域的一个重要分支，旨在让计算机能够理解、处理和生成自然语言。
在NLP中，关键词提取算法是一类重要的技术，它可以帮助计算机从文本中抽取出最具代表性和关键性的词语，对文本进行自动化的摘要和分析。
本文将介绍几种常见的关键词提取算法，并分析它们的特点和应用场景。
TF-IDF算法是一种常用的关键词提取算法，它基于词频-逆文档频率（Term Frequency-Inverse Document Frequency）的统计方法。
该算法认为一个词语的重要性和它在文本中的频次成正比，但与它在语料库中的出现频率成反比。
TF-IDF算法计算每个词语的TF-IDF值，然后根据这个值来确定文本中的关键词。
TF-IDF算法简单高效，适用于大规模文本的关键词提取，但它对于一些特定领域的文本可能不够精准。
另一种常见的关键词提取算法是TextRank算法，它是一种基于图的排序算法，源自于PageRank算法。
TextRank算法将文本中的词语作为节点，词语之间的共现关系作为边，构建一个带权有向图。
然后利用图上的节点之间的连接强度来计算每个词语的重要性，最终确定文本中的关键词。
TextRank算法不依赖于领域知识和语料库规模，且能够很好地处理长文本和文档集合，因此在文本摘要和关键词提取方面有着广泛的应用。
除了TF-IDF和TextRank算法外，主题模型（Topic Model）也是一种常见的关键词提取算法。
主题模型可以将文本中的词语映射到潜在的主题空间中，从而挖掘出文本中隐藏的主题信息和关键词。
常用的主题模型包括潜在语义分析（Latent Semantic Analysis, LSA）和潜在狄利克雷分配（Latent Dirichlet Allocation, LDA）等。
这些主题模型能够发现文本中的话题结构和相关性，对于文本主题分析和关键词提取有着较好的效果。
基于改进TF-IDF算法的情报关键词提取方法张瑾【摘要】Being usually entirely based on word frequency, TF-IDF neglects the influence of other features of words on keywords. The essay proposes a automatic keywords extracting method based on TF-IDF, word position and word span. Word position weight value and word span value are added to the traditional TF-IDF keywords term weighting algorithm in order to avoid deviation. The experiment find-ings show that the proposed method, with higher accuracy, lower recall rate and F1 values, is more applicable in the extraction of Chinese intelligence keywords.%传统的TF-IDF完全基于词频，忽略了词语的其它特征项对关键词的影响。
本文提出基于TF-IDF、词位置和词跨度的关键词自动提取的方法。
该方法通过在传统的TF-IDF关键词权重计算方法中，加入位置权值及词跨度权值，避免单纯采用TF-IDF算法产生的偏差。
实验结果表明，该方法在情报关键词提取中有广泛的应用价值，其准确率、召回率及F1值与传统方法相比有明显提升。
【期刊名称】《情报杂志》【年(卷),期】2014(000)004【总页数】3页(P153-155)【关键词】提取;TF-IDF;位置权值;词跨度值【作者】张瑾【作者单位】郑州轻工业学院郑州 450002【正文语种】中文【中图分类】TP3910 引言随着大数据时代的到来，信息和现实生活密不可分，如此海量的数据使得寻找所需的情报的难度加大。
I.J. Education and Management Engineering, 2019, 4, 11-19Published Online July 2019 in MECS ()DOI: 10.5815/ijeme.2019.04.02Available online at /ijemKey Term Extraction using a Sentence based Weighted TF-IDFAlgorithmT. Vetriselvi a, N. P. Gopalan b, G. Kumaresan b,*a, Department of Computer Science and Engineering, K. Ramakrishnan College of Technology,Tiruchirappalli, Indiab Department of Computer Applications, National Institute of Technology, Tiruchirappalli, IndiaReceived: 06 November 2018; Accepted: 15 February 2019; Published: 08 July 2019AbstractKeyword ranking with similarity identification is an approach to find the significant Keywords in a corpus using a Variant Term Frequency Inverse Document Frequency (VTF-IDF) algorithm. Some of these may have same similarity and they get reduced to a single term when WordNet is used. The proposed approach that does not require any test or training set, assigns sentence based Weightage to the keywords(terms) and it is found to be effective. Its suitability is analyzed with several data sets using precision and recall as metrics.Index Terms: Similarity Matrix, Term Count, WordNet.© 2019 Published by MECS Publisher. Selection and/or peer review under responsibility of the Research Association of Mode rn Education and Computer Science1.IntroductionEmerging techniques in Natural Language Processing (NLP) are employed in areas like Information Retrieval, Semantic Analysis, Sentiment Analysis and Review processing systems. The underlying crux of all the above is extracting key phrases from a given document or corpus often leading to further analytics like clustering, classification etc. The keywords are set of important words which give good description about the contents of a given document. All text mining applications at the outset involve preliminary processing like stemming and stop word removal over the extracted keywords.* Corresponding author.E-mail address: kumareshtce@.The conventional TF-IDF algorithm is used for extracting term frequency in a document. Frequency of the terms in documents is counted as TF. Likewise, the occurrence of the same term in different documents is denoted by IDF [1] multiplying them.A Variant of TF-IDF with enhanced efficiency is also used in this context. Sentences of a document play a major role while its contents are delivered. In the present work, assigning weightage to the terms based on their locations in a sentence is found to improve the efficiency of the procedure. The number of occurrences of terms may not necessarily imply that they are keywords for a document. In WordNet, a term may have several synonyms and it is possible that they may be related in some way or other. In such occasions, similarity among these words needs to be considered while extracting terms to calculate similarity measures such as Cosine, Inverse Euclidean Distance and Pearson Correlation Coefficient similarities [2]The key term extraction is a basic requirement for all text processing techniques and there exist several methods for finding keywords and similarity. These are presented in the literature survey. The proposed method is discussed in §2 and the experimental evaluation using LT4EL.eu is presented in §3. The conclusion and the possible extension of this work are discussed in §4.1.1. Keyword ExtractionThe Keyword Extraction is done by various ways using linguistic, statistical and graphical approaches. Machine learning methods such as supervised learning and unsupervised learning are also explored under NLP processes. Further, few linguistic approaches like verb and noun phrase extraction are also used in this context but usage of statistical approaches are simple in nature. The supervised machine learning techniques involve training and testing, requiring a dataset for it. In case of unsupervised machine learning the number of clusters formed has to be mentioned at the earliest.The graphical representations of text documents show the relationship among the terms. To name a few, semantic rank, and HITS are some of the popular methods employed in text processing. One of the most widely used statistical approaches is Traditional TF_IDF, for enumerating the relevant terms of the document. Sungjicj Lee [16] used a two-step process for extracting keywords. In the first step, they are extracted by TF-IDF method and ranked using candidate comparison in the next step. It is evaluated with a statistical measure to bring out the importance of a term in a document [1].The following identities are used calculating TF- IDF.,,,i ji j i jn tf k n =⨯∑ (1)log ()i j j j Didf d t d =∈ (2),,i j i j i tfidf tf idf =* (3) The subscripts i, j mean i th term in j th document.It is further classifieds into two types - a Feature based TF-IDF and a variant TF-IDF. In the former, the location of the term and its relevant information and in the later Basic Term Frequency (BTF), two Normalized Term Frequencies NTF1, and NTF2 are used for extraction. In the above normalization, the repeated and synonymous terms are considered to be identical and ranked using domain filtering.1.2.Semantic GraphsGeorge et al [4] constructed a weighted semantic graph using words as nodes and the relation between them as edges. The length of the path between words in the semantic net is weight of the edge connecting them in the semantic graph. It is called weight assignment in distributional measure. Weights are also calculated with omiotis [3] in thesaurus based method.1.3.Key Phrase RankingA keyword ranking is to find the highly relevant terms in one or more document(s) and syntactic, semantic, and statistical approaches may be employed to accomplish it. In statistical method the weight based arrangement of keywords are helpful to identify the rank of the keywords.Fig. 1. Various Ranking MethodsThe high ranking term is a word which is connected with a maximal number of words in a semantic graph. In syntactic method the noun and verb phrases are counted and ranked which is shown in Fig. 1.2.Related WorkThe description of a document can be inferred from a set of highly significant keywords in it and their extraction is crucial task. Such keywords can be extracted by identifying frequent items in the document in which duplication is removed by a suitable stemming algorithm. Then, VTF-IDF method is used to measure the frequency of words. Semantic similarity among words can be obtained from WordNet measures and subsequently a Similarity matrix is formed for further analysis [5].2.1 Initial Pre-ProcessingTokenization, stemming and stop word removal are set of activities in pre-processing. Tokenization spilt the sentences in to token i.e. small lexical units. Stemming is the process of identifying the root word. The supporting words {is, and, the, of, for,} have to be removed, hence those are removed by the process of stop word removal.2.2TF-IDF variantsAfter pre-processing, each term was assigned by certain weight using several variants in TF-IDF.Term Count (tc):The tc represents frequency of the term that is how often a given term occurs in a document collection.,1Di i j j termcount n ==∑Where n is the total number of terms in the document.2.3 Normalized term frequency (ntf1 and ntf2)The first bias of Term Count (tc) that we need to be removed is a bias towards tf value is even larger than idf value. To remove this bias, we use the first variant that normalized by the Maximum TC in a given document collection. The second bias of tc is that the words appeared in a long documents may have larger frequency and may be regarded as more important words. Hence, we want to reduce such weight of document’s length, which results in the second normalized tc, ntf1. The ntf2 is given as input for IDF variant. First normalized form: 11(1,2,3)tc ntf MAX tc tc tc T = (4) Second normalized form:,1,12j D i j T j k jk n ntf n=-=∑∑ (5) ntf2 value for the terms are multiplied by idf, hence it form ntfidf. The value of ntfidf supports to arrange the key terms in descending order based on the weight calculated. Those keywords are stored in SimMat, to find similarity between terms. SimMat is nothing but a vector Space between the ({t1,t2}) terms. Terms are act as row as well as columns. The function Synset in NLTK tool kit is used to find the values of the SimMat [13]. The normalized similarity values always fall in between 0-1.The major role played here is more similar terms are removed and rest of the terms stored in Term Set. From the Term Set. One constraint to remove the term from the term set is that the similarity value more than the threshold.Here the threshold was fixed as 0.6.SimMat(t1,t1)will be one for all terms hence it is diagonal, SimMat(t1,t2)will be from 0.1 to 0.9,The SimMat values above o.6 are removed from the Term set. Hence the Term Set contains the reduced number of keywords which are most relevant. The weight is still improved by our proposed algorithm and is a way to improve the quality of keyword extraction. Sentences are ranked only by the keyword ranking. In This method length of the sentences are analyzed for extracting keyword. All the scientific articles are simple in nature. The length of the sentence varies from 6 to 21. Our model gives Weightage to the keywords based on the length of the sentences.It comprise of three steps.Step 1: Find the length of the sentencesS = {s1,s2,s3,s4,...sn}SL = {sl1,sl2,sl3,sl4.sl5....sln}Step 2: Find the Average sentence lengthASL=SL N i n =÷∑Step 3: Ranked keywords are updated by its weight.Step 4: Re-ranking2.4 Sentence Processing AlgorithmA New Approach:Fig. 2. Proposed Approach3.Experimental SetupNLTK tools to identify the similarities between words and Python are used to develop SimMat for keyword extraction. The data set are taken from Learning technology for E-Learning (LT4EL) [15] for English. It consists of 45 documents and the number of keywords extracted by traditional method is 1174. Fig. 2. gives the pictorial view of the proposed model.From the data set a sample document is taken for evaluation. And variant TF-IDF (formula 1)is applied. The parameters count (Tc), NTF1 and NTF2 are calculated and it gives still reduced number of terms for a given document.Fig. 3. Number of Terms variantThe above Fig. 3. clearly depicts the terms related to frequency from the E-Learning material provided in the website LT4eL.SimMat created based on the available terms in Term Set.Synset in NLTK help to find similarity between the terms. The terms which are not available in WordNet can be assumed as 0 value .Here it proceeds to do the process of reduction, it is a kind of second filtering .The similarity values between the terms are fall in between 0-1, ultimately all diagonal values are 1, so make it aszero first. Then remove the terms which has more than 0.7 as similarity value. In the analysis five different documents of various sizes are taken for experimentation.Table 1. Comparison with other ideasFig. 4. Comparison of Recall and PrecisionThe set of all 45 documents is taken for the present analysis based on STF-IDF. This produced 1095 keywords which is more by 2 % in comparison with other conventional methods and consequently from Fig. 4. It can be seen that there is an improvement in precision and recall metrics. Also, from Table 1 the present method can be observed to result in a better harmonic mean in comparison with earlier studies and the term set is most descriptive of the documents taken.4.ConclusionsThis System creates a useful TermSet containing the most descriptive and highly related terms for any the given document. The shortcoming in traditional TF-IDF algorithm has been overcome with the usage of attributes tc, ntf1 and ntf2. The newly employed sentence based processing on keywords helps to improve the precision and recall. Further, this may also improve WordNet supported word similarity values from those in E-Learning corpus. Hence, the present approach proves to be efficient in comparison with the earlier traditional key term extraction methods.References[1] S.Akter, AS.Asa and MP.Uddin, MD Hossain”An extractive text summarization technique for Bengalidocument (s) using K-means clustering algorithm “on IEEE International Conference Imaging, Vision & Pattern Recognition (icIVPR), pp 1-6 , 2017.[2] R.Silveira, V.Furtado, and V.Pinheiro “ Ranking Keyphrases from Semantic and Syntactic Features o fTextual Terms”, Brazilian Conference on Intelligent Systems (BRACIS), pp 134-139, , 2015[3] M.Litvak and st “Graph based keyword extraction for single –document summarization” onMMIES '08 Proceedings of the Workshop on Multi-source Multilingual Information Extraction and Summarization pp:17-24,2008.[4] P.Alireza and K.Mohadesh,”A Probabilistic Relational Model for Keyword Extraction” InternationalConference on Statistics in Science, Business and Engineering (ICSSBE) ,pp 1-5,2012.[5] Sneha .S Des ai, and xmonarayana ”WordNet and Semantic Similarity based Approach forDocument Clustering”International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS), pp 312-317 ,2016[6] A .Guo, and T .Yang “Re search And Improvement Of Feature Words Weight Based On TfidfAlgorithm,” Information Technology, Networking, Electronic and Automation Control Conference, IEEE 2016 ,pp 415-419,2016[7] C.Clifton, R.Cooley and J.Rennie “Topcat: Data Mining For Topic Identification In A Text Corpus”IEEE Transactions on Knowledge and Data Engineering Vol 16, pp 949-964,Issue: 8, Aug. 2004 [8] L.Suanmali and N.Salim“ Fuzzy Genetic Semantic Based Text Summarization.IEEE NinthInternational Conference on Dependable, Autonomic and Secure Computing (DASC), pp 1184-1191,2014[9] A. Kiani, and MR. Akbarzadeh Automatic Text Summarization Using: Hybrid Fuzzy GA-GP “IEEEInternational Conference on Fuzzy Systems Sheraton Vancouver Wall Centre Hotel, Vancouver, BC, Canada ,pp 977-983,2006[10] P.Arora and O.Vikas ” Semantic Searching and Ranking of Documents using Hybrid Learning Systemand WordNet” (IJACSA) International Journal of Advanced Computer Science and Applications, Vol.3,pp 113-120,2011[11] L. Lemnitzer and P. Monach e” Extraction and evaluation of keywords from Learning Objects –amultilingual approachs” Language Resources and Evaluation Conference LREC, pp 112-120,2008 [12] YA.Jaradat and AT.Al-Taani “Hybrid-based Arabic Single-Document Text Summarization ApproachUsing Genatic Algorithm “7th International Conference on Information and Communication Systems (ICICS), pp 85-91 ,2016[13] Porter M.F., “An Algorithm for Suffix Stripping”, MCB UP Ltd Program, Vol. 14, no. 3, pp. 130-137,1980.[14] /howto/wordnet.html[15] http://www.lt4el.eu/review_luxembourg.phpAuthors’ ProfilesT. Vetriselvi: Part-Time Research Scholar at Department of Computer Applications,National Institute of Technology Tiruchirappalli. She is currently an assistant professor atthe K. Ramakrishnan College of Technology, Tiruchirappalli. She has 11 years ofteaching experience and 3 years of industry experience. She is the author of 4 journalscientific papers, 2 books and 3 proceedings. Her areas of interest include Data Miningand Programming Languages.N. P. Gopalan:Professor of Computer Applications Department, National Institute ofTechnology, Tiruchirappalli, Tamil Nadu, India. He obtained his PhD. from IndianInstitute of Science, Bangalore. Interested in Data Mining, Cryptography, DistributedComputing and Theoretical Computer Science.G. Kumaresan: Research Scholar at Department of Computer Applications, NationalInstitute of Technology Tiruchirappalli. He received MCA from Thiagarajar College ofEngineering, Madurai, India. M.Tech from Bharathidasan University, Tiruchirappalli,India and M.Phil. from St. Joseph College, Tiruchirappalli, India. His areas of interestinclude Public Key Cryptography, Cellular Automata and Cloud Security andProgramming Languages.How to cite this paper: T. Vetriselvi, N. P. Gopalan, G. Kumaresan,"Key Term Extraction using a Sentence based Weighted TF-IDF Algorithm", International Journal of Education and Management Engineering(IJEME), Vol.9, No.4, pp.11-19, 2019.DOI: 10.5815/ijeme.2019.04.02。
