完整的中英⽂词频统计步骤：1.准备utf-8编码的⽂本⽂件file2.通过⽂件读取字符串 str3.对⽂本进⾏预处理4.分解提取单词 list5.单词计数字典 set , dict6.按词频排序 list.sort(key=)7.排除语法型词汇，代词、冠词、连词等⽆语义词8.输出TOP(20)⼀、.英⽂歌曲词频统计str2='''I will run, I will climb, I will soarI'm undefeatedJumpiing out of my skin, pull the chordYeah I believe itThe past, is everything we weredon't make us who we areSo I'll dream, until I make it real,and all I see is starsIts not until you fall that you flyWhen your dreams come alive you're unstoppableTake a shot, chase the sun, find the beautifulWe will glow in the dark turning dust to goldAnd we'll dream it possiblepossibleAnd we'll dream it possibleI will chase, I will reach, I will flyUntil I'm breaking, until I'm breakingOut of my cage, like a bird in the nightI know I'm changing, I know I'm changingIn, into something big, better than beforeAnd if it takes, takes a thousand livesThen it's worth fighting forIts not until you fall that you flyWhen your dreams come alive you're unstoppableTake a shot, chase the sun, find the beautifulWe will glow in the dark turning dust to goldAnd we'll dream it possibleit possibleFrom the bottom to the topWe're sparking wild fire'sNever quit and never stopThe rest of our livesFrom the bottom to the topWe're sparking wild fire'sNever quit and never stopIts not until you fall that you flyWhen your dreams come alive you're unstoppableTake a shot, chase the sun, find the beautifulWe will glow in the dark turning dust to goldAnd we'll dream it possiblepossibleAnd we'll dream it possible'''.lower()#aa = '''."?!'''#for word in aa:# str2 =str2.replace('word','')str2 =str2.replace('\n',' ')str2 =str2.replace(',',' ')print(str2)#去除特殊符号str2 = str2.strip()#去掉⾸尾空格str2 = str2.split()#通过指定分隔符对字符串进⾏切⽚print(str2)print('统计每个单词出现的次数为:')for word in str2:print(word,str2.count(word))strSet=set(str2)newSet={'a','will','it','out','of','my','the','i','in','to','when','and'}strSet1=strSet-newSet#去除介词和其他print(strSet1)strdict={} #单词计数字典for word in strSet1:strdict[word] = str2.count(word)print(len(strdict),strdict)strList = list(strdict.items())def takesecond(elem):#定义函数return elem[1]#strList.sort(key=lambda x:x[1],reverse=True)#匿名函数strList.sort(key=takesecond,reverse=True)#按照数值⼤⼩进⾏排序print(strList)for i in range(20):print (strList[i])#前⼆⼗2.中⽂⼩说词频统计import jieba;#准备utf-8编码的⽂本⽂件filef = open('doupo.txt','r',encoding='utf-8')#通过⽂件读取字符串 str,对⽂本进⾏预处理fo=f.read()f.close()print(fo)#⽤字典形式统计每个词的字数doupols = jieba.lcut(fo)doupodict = {}for word in doupols:if len(word)==1:continueelse:doupodict[word]=doupodict.get(word,0)+1print(doupodict)#cutprint(list(jieba.cut(fo))) #精确模式，将句⼦最精确的分开，适合⽂本分析print(list(jieba.cut(fo,cut_all=True))) #全模式，把句⼦中所有的可以成词的词语都扫描出来print(list(jieba.cut_for_search(fo))) #搜索引擎模式，在精确模式的基础上，对长词再次切分，提⾼召回率，适合⽤于搜索引擎分词#以列表返回可遍历的(键, 值) 元组数组wcList = list(doupodict.items())wcList.sort(key = lambda x:x[1],reverse=True) #出现词汇次数由⾼到低排序print(wcList)#第⼀个词循环遍历输出5次for i in range(5):print(wcList[1])。
