MIT-Machine-learning-for-nlp-Michael Collinsl10

人工智能自然语言技术练习(习题卷11)第1部分：单项选择题，共116题，每题只有一个正确答案,多选或少选均不得分。

1.[单选题]语料库加工的主要方式不包括A)人工方式B)交替方式C)自动方式D)半自动方式答案:B解析:2.[单选题]以下几个选项，哪一个常赋值给batch_sizeA)128B)127C)126D)125答案:A解析:3.[单选题]L2正则化项的加入，可以使模型达到什么样的效果A)防止过拟合B)防止欠拟合C)防止完美拟合D)不确定答案:A解析:4.[单选题]在NLP的中多模型当中，N-Gram模型可以用来做什么A)预计或者评估一个句子是否合理B)可以做到文本翻译C)提取文本当中的主题D)可以做问答系统答案:A解析:5.[单选题]实验测得四组(x，y)的值为(1，2)，(2，3)，(3，4)，(4，5)，则y与x之间的回归直线方程为A)y=x+1B)y=x+2C)y=x-1D)y=2x+1答案:A解析:6.[单选题]我们知道在概率图中有很多的节点，这些节点分别有什么意义A)随机变量D)学习率答案:A解析:7.[单选题]使用马尔科夫链的算法有：A)HMMB)SVMC)CRFD)MEMM答案:A解析:8.[单选题]下面关于数据粒度的描述不正确的是:A)粒度是指数据仓库小数据单元的详细程度和级别;B)数据越详细,粒度就越小,级别也就越高;C)数据综合度越高,粒度也就越大,级别也就越高;D)粒度的具体划分将直接影响数据仓库中的数据量以及查询质量.答案:C解析:9.[单选题]修正线性函数指的是哪个函数A)sigmoidB)tanhC)reluD)Leaky relu答案:C解析:10.[单选题]归一化的公式方式为：A)通过中值和均值进行确定B)通过平均值和最小值确定C)通过方差和均值确定D)通过标准差和均值确定答案:C解析:11.[单选题]LDA中的五个分布其中的二项分布，是一个什么分布A)离散的随机分布B)连续的随机分布C)连续的分布D)不确定答案:A解析:12.[单选题]（）是指用NLP、文本挖掘和计算机语言学等方法对带有情感色彩的主观性文本进行分析、处理、归纳和推理的过程。

Machine Learning
input output
input
output
Heavily Relying on Observations
One Conclusion from Past Research
• Hybrid is most realistic and effective for natural language processing, astic View on Language
• Language is cognitive activities • Language concept is based on prototype • Language is metaphoric and metonymic
• A computer system must be constructed based on math • Open question: whether it is possible to process natural language as humans, using math models • Natural language processing is believed to be AI complete
– Logic – Facts – Their combinations
• “Whereof one cannot speak, thereof one must be silent”
Philosophical Investigations
(Ludwig Wittgenstein, 1953)
• “The meaning of a word is its use in the language” • Language game: speaking of language is part of an activity, or a form of life, which gives language its meaning • Family resemblance: e.g., the notion of “game” • To understand the meaning of word, we should travel with the word’s uses through “a complicated network of similarities overlapping and criss-crossing” • Paradox: no course of action could be determined by a rule

Part-of-Speech Tagging of Transcribed SpeechMargot Mieskes,Michael StrubeEML Research gGmbH,Heidelberg,Germanyhttp://www.eml-research.de/nlpAbstractWe used four Part-of-Speech taggers,which are available for research purposes and were originally trained on text to tag a corpus of transcribed multiparty spoken dialogues.The assigned tags were then manually corrected.The correction wasfirst used to evaluate the four taggers,then to retrain them.Despite limited resources in time,money and annotators we reached results comparable to those reported for the taggers on text.Based on our experience we present guidelines to produce reliably POS tagged corpora of new domains.1.IntroductionPart-of-Speech(POS)tagging is a prerequisite for many high-level Natural Language Processing(NLP)tasks.A number of POS taggers have been developed and made available to the research community.The majority of them has been trained on written texts,mostly on newspaper texts.Only in few instances POS tagging was applied to transcribed speech.Examples for this can be found in God-frey et al.(1992),Heeman&Allen(1999)and Zechner (2001).All three mainly deal with dialogues.Only Zech-ner(2001)reports results for multiparty dialogues as well. Some work has been done to apply POS taggers to new do-mains with little or with no manual annotation.A small amount of manually annotated training data was used by Clark et al.(2003)and by Collins(2002),who used small amounts of data to do co-training and explore the perfor-mance of a Hidden Markov Model based perceptron re-spectively.Nakagawa et al.(2002)and van Halteren et al. (1998)used no manually annotated data for retraining but multiple POS taggers directly and voting techniques on the results.A third method is applied by Zavrel&Daelemans (2000)who use the results from several taggers trained on a small amount of training data as input for a learner. There are problems in applying the approaches described above to our task,the application of POS taggers to tran-scribed multiparty dialogues.The researchers who tagged transcribed speech did not evaluate the taggers they used before retraining was done.The research exploring ways to retrain taggers with little or no data was performed on written text.But Wermter&Hahn(2004)showed that texts even from different domains can be very similar.They used two POS taggers,which were trained on newspaper texts, and applied them to medical texts.The evaluation on manu-ally annotated medical texts gave good results.The authors explain this by a similarity in uni-,bi-and trigram POS dis-tribution in newspaper and medical texts.In our work we make use of four different POS taggers(see Section2.).We apply them to transcribed multiparty spo-ken dialogues.We report results before(see Section3.)and after the taggers have been retrained on manually annotated data,and specifically we evaluated the behaviour on differ-ent(increasing)amounts of data(see Section4.).This led us to compile guidelines on how to efficiently create data for retraining taggers to be used on a new domain.2.TaggersFour taggers were considered.The TnT tagger1(Brants, 2000)uses a Hidden Markov Model(HMM)based on n-grams and lexical information.Two taggers from the Stan-ford Java library for tagging2(left3words(Toutanova& Manning,2000)and bidirectional(Toutanova et al.,2003)). Thefirst uses a maximum entropy model and the context of three words to the left.The second considers the context to the left and to the right by applying contextual HMMs (CHMM).Finally the Brill(TBL)tagger3(Brill,1994)uses transformation-based learning.Each of these automatic taggers was originally trained and tested on the Wall Street Journal portion of the Penn Tree-bank(Marcus et al.,1993),which only consists of written text.The results reported for the taggers on this corpus vary between96.5%and97.24%.For TnT only the perfor-mance on known and unknown words is given separately with97.7and89.0%.Therefore,using these taggers on the transcribed speech in the ICSI Meeting Recorder Corpus (ICSI Corpus)(Janin et al.,2003)will inevitably result in considerably lower accuracy rates.Some of the reasons that account for this are:First,the vocabulary offinancial news is different from that of dialogues which mostly deal with speech and language technology.Second,the style is differ-ent between newspaper text and colloquial speech.Exam-ples for these differences include disfluencies and explicit paragraph separation,but also sentence length and sentence complexity.Finally in the meetings non-native speakers are also involved.It seemed reasonable though to do the POS annotation semi-automatically and using these taggers as the basis for the manual correction.In order to get one tag for each token,a majority decision over the four automatic taggers(Maj4in the following)was reached.This was used as input for the human annotators,who corrected the data.3.Manual AnnotationThe whole ICSI Corpus contains75meetings.12were randomly chosen to be annotated by three human annota-tors.In addition to the12meetings we used one meeting to train the human annotators,which was not considered 1http://www.coli.uni-saarland.de/˜thorsten/tnt/2/software/ tagger.shtml3/˜brill/in the evaluation.We used the MMAX2annotation tool4, which allows easy access to and manipulation of the tags. The tagset was based on the Penn Treebank tagset(San-torini,1990),which was also used for the Switchboard POS annotation(Godfrey et al.,1992).We added some tags to deal with phenomena that are of specific interest to the project in which this evaluation was carried out-specifi-cally RELP,which is used to distinguish relative pronouns from wh-determiners(WDT).Additionally,we introduced one tag to deal with all interpunctuation signs–INP.The12meetings were used to check inter-rater agreement for the three annotators.From the remaining62meetings 25were annotated individually by one of the three annota-tors.Based on the manual annotation a gold standard was created by assigning a majority decision tag to each token in the meetings.This majority decision was manually cor-rected by a senior annotator.Inter-rater reliability was very high(κ=.96),showing that the automatically assigned tags can be manually corrected highly reliable.Therefore, we assume that the quality of the individually annotated meetings are of equally high quality as the gold standard. Gathering the data took about two months,with three an-notators working for about240h in total.The costs were reasonable with about EUR5000total.Nine meetings were used for evaluating the automatic an-notation.Three meetings were taken from the gold standard data(Test1in the following)and six from the individually annotated data(Test2).Data TBL TnT Left3Bidirect Maj4Test111.311.211.111.110.5Test213.814.313.613.213.4 Table1:Error Rates for automatically annotated data Table1shows the results for the automatically tagged data. The results for Test1,which is part of the gold standard data but not used for retraining are better than those for Test2, which was also not used in retraining but was not part of the gold standard either.In this evaluation we did not consider tags that were unknown to the taggers,because they were introduced by our annotation scheme.4.RetrainingNine Meetings from the gold standard were used for train-ing.Since they are spread across the whole corpus they contain a large variety of words and language used.Ad-ditionally,we had15meetings which were manually an-notated.Retraining was done based on6different setups. These setups contained increasing amounts of data.Setup1 contained the data from the gold standard and consisted of 124,158tokens.In every setup3meetings,all of them an-notated individually by one of the three annotators,were added.Thefinal setup contained24meetings and282,686 tokens in total.Our aim was tofind a good trade-off between good anno-tation results and reasonable effort for the manual annota-tion.It is also important that for most taggers,the amount of time needed for retraining increased with increasing data 4http://mmax.eml-research.de set size.Additionally,the relationship between the amount of training data and the results is unknown.It is assumed that the more data is available,the better the results will get. The fastest to train was the TnT tagger,which took only a few minutes.The TBL and Stanford taggers took hours. Despite of this difference,TnT’s results were comparable to the other three taggers.The training parameters for each of the taggers remained unchanged throughout the different training setups.Espe-cially the TBL Tagger would allow for several parameters to be set according to data set size and desired results.We left these parameters as they were suggested by the author. Test1contains about40K token and Test2contains about 77K tokens.Error Rates in%Set1Set2Set3Set4Set5Set6 tokens K124162197221253283TnTTest1 3.4 3.4 3.3 3.3 3.4 3.4Test2 5.4 5.1 4.9 4.6 4.5 4.5TBLTest1 3.9 3.5 3.5 3.5 3.6 3.5Test28.4 5.5 5.0 4.7 4.4 4.4Left3Test1 3.2 3.0 3.0 3.2 3.2 3.2Test2 5.2 4.7 4.5 4.3 4.2 4.1BidirectTest1 3.2 3.0 3.0 3.2 3.2 3.2Test2 5.2 4.7 4.5 4.3 4.2 4.1 Table2:Average error rates for all taggers in each of the setupsTable2shows the results for all taggers after they have been trained on the manual data in different setups.Thefirst part shows the results for TnT,after being trained on each of the setups.The results are very good and improve by about1% in total.The gain in each step is rather small,the biggest is about0.3%.In the last three steps the gain is very small and finally non-existent.The last noticeable step is from Set3 to Set4.This indicates that a training set size of between 197K and221K tokens gives good tagging results with rea-sonable effort for manual data.The second part shows the results for TBL.The results are similar to TnT but the gain for the various setups is higher. Especially from thefirst to the second setup in Test2the error rate decreases by2.9%.The later steps are smaller and level out towards the end.Here,the last noticeable step is from Set4to Set5.This indicates that TBL needs more training data than TnT.The third and fourth parts should be considered together be-cause the results are very similar if not identical.Again the first few steps are bigger than the last few steps.After Set4 the error rate does not decrease much.This suggests that the even-odd point of decreasing error rate and increasing training data size is somewhere between Set3and Set4. The improvement for all taggers presented here from the original tagging(Table1)is in the range of8−10%for each tagger.In addition,we were interested in whether the improve-ment can also be demonstrated in the majority decision or whether the majority decision even requires less trainingdata,while keeping the error rate low.Since the two Stan-ford Taggers had a similar performance,the majority over all four taggers would show very similar results to these taggers.Therefore,we considered only three taggers in the final evaluation.We removed the Bidirectional tagger from further consideration.This is motivated by the fact that this tagger takes longer to train and to tag and the results are very similar to those by Left3.Set1Set2Set3Set4Set5Set6 Test1 3.0 2.9 2.9 3.0 3.0 2.9Test2 5.1 4.7 4.4 4.1 4.0 3.9 Table3:Average error rates for Maj3on each of the Setups Table3shows the results for the majority decision based on three different taggers(Maj3in the following).As can be seen in Set4the results are as good as for any of the single taggers in all setups.This is also the last noticeable step.Although Set5and Set6show the best overall results, the gain in Set5and Set6is not as remarkable as in the steps before.With this amount of training data Maj3outperforms all single taggers in all setups.In general,one can observe that the gain throughout the setups is about1%.TBL improves by about4%,but this is mainly due to the difference betweenfirst and second setup(2.9%).Between the second and the last setup the difference is only1.1%,which is the same as with the other taggers.Maj3gained slightly more(1.2%),but also outper-forms the single taggers by0.2%.Furthermore,the individ-ual results are very close to those that have been reported for the taggers on text(see Section2.).Maj3achieves 97.1%on the best setup,which is close to the best tagger on text(97.2%).5.DiscussionThe work presented here,was done on transcribed speech from multiparty meetings,which so far has not been ex-plored in detail.We used four common POS taggers to automatically annotate the transcripts.These results were manually corrected by human annotators,which is consid-erably faster than assigning POS tags from scratch.The re-sults from the manual annotation were then used to retrain the POS taggers.It turned out,that about221K tokens are sufficient to get results that are comparable to those reported for the POS taggers applied to text.Redoing the majority decision im-proved the results on this amount of data by about0.2%. Using the full amount of training data improved the best re-sults by0.3%also compared to the best results of the single taggers,which were achieved on texts.It has been argued in the past,that in some cases retrain-ing is not necessary to do POS tagging(Wermter&Hahn, 2004).The authors report an analysis of uni-,bi and tri-grams of the data on which the taggers were trained(news texts)and of the data on which the taggers were tested (medical texts).They found that the n-grams were very similar.Following this analysis we compared the WSJ cor-pus with the ICSI corpus.Table4shows thefive most common Uni-,Bi-and Tri-grams for Wall Street Journal(WSJ)and the Meetingx-gram Num WSJ%ICSI% uni1NN14.01INP19.00 2INP11.24PRP9.203IN10.51DT7.664NNP9.83UH7.545DT8.67NN7.186IN7.1412PRP 2.69······19······NNP 1.0933UH0.01······bi1DT+NN 4.13UH+INP 4.95 2NNP+NNP 3.68INP+UH 4.933NN+IN 3.50PRP+VBP 3.294IN+DT 3.46INP+PRP 3.295JJ+NN 2.91DT+NN 2.742$+CD+CD0.81INP+PRP+VBP 1.633DT+NN+NN0.78UH+INP+UH 1.464.+DT+NN0.66CD+CD+CD 1.395DT+NN+,0.65INP+RB+INP0.01 Table4:Differences between WSJ and ICSI Uni-,Bi-and Trigrams distributionRecorder Data(ICSI).Thefive most common unigrams for WSJ are NN,INP,IN,NNP and DT,whereas for ICSI theyare INP,PRP,DT,UH and NN.UH is very rare in the WSJ corpus.These differences also appear in the analysis of bi-and trigram.For WSJ the dominant tags are DT and NN in various combinations and variations,whereas for ICSI the dominant tags are UH,PRP and INP,also in various combi-nations.For the Unigrams we also show at which positionin the frequency table the tags that are most often in WSJ occur in ICSI and vice versa.The Bi-and Trigrams under-line the difference between these two corpora.For the Uni-grams three tags are shared in thefive most frequent tags.Inthe Bigrams only one combination is left and for Trigrams none.It has to be noted that the combination of CD+CD+CDis an artefact of the data in ICSI,as most meetings start orfinish with all speaker recording a sequence of numbers. The same accounts for the combination$+CD+CD in WSJ. Table5shows those categories which benefit the most from retraining.Most other categories improve as well,but on a smaller scale.Only few categories do not improve at all or even achieve worse results after training.The categories in Table5can be characterized as either occurring rarely inthe original training data(WSJ)as e.g.UH,FW and PDT or they form a very large group,as e.g.NN.Some categories achieved good results(≥95%correct)with the original POS taggers.Among those categories are CC,CD,MD,NNS, PRP,PRP$,TO,VBP,VBZ and WRB.These categories ei-ther belong to afixed group of words()or are ruledby certain(fixed)rules(e.g.VBZ).Among the categories that have been unreliably tagged are particles(RP),which achieve a precision of about80%and recall of about72%,proper singular nouns(NNP)with a precision of about86%and recall of about90%,but alsowh-determiner(WDT),which only achieve a precision of about66%and a recall of about52%.A more detailed dis-cussion will be provided after further analysis of the data.Tag before afterFW52.185.2JJ78.689.2NN83.994.4NNP46.088.9PDT33.690.52POS33.984.5RP77.580.12UH56.699.0VB89.494.5WDT14.379.0WP86.294.2Table5:Categories which improve most through training6.ConclusionsIn Section1we presented some approaches to perform POS tagging with as little manual work as possible and ap-proaches to improve the results of POS tagging in general (Clark et al.,2003;Collins,2002;Nakagawa et al.,2002). Several remarks should be made here:First these works were based on text,like the Wall Street Journal portion of the Penn Treebank.Second,all of them are computation-ally very demanding.Finally,only the results presented by Clark et al.(2003)have been better than the results we re-port here,but on a considerably larger amount of data. The results presented in our work give several method-ological implications for further approaches to POS tag-ging of new domains.Human annotation is very expen-sive,in time and money.It is therefore desirable to ex-plore,how much manually annotated data is necessary to get good/comparable results.Furthermore,the computa-tional effort needed to achieve these results should be rea-sonable,too.Two main results were found in our work:first,a good trade-off point between the effort put into man-ually annotated data and the results of retraining POS tag-gers based on this data.Second,we applied a computation-ally cheap method for getting good results automatically. In future work more sophisticated methods to merge the results of different taggers in order to improve the results could be applied,like e.g.those mentioned by van Halteren et al.(1998),who used a pairwise voting system or Zavrel &Daelemans(2000)who used a learning system based on the results from the POS taggers.Data Availability.The manually annotated data as well as the automatically tagged data can be downloaded from the projects homepage http://www.eml-research. de/nlp/diana-summ.php.Acknowledgments.This work has been supported by the DFG under grant STR545/2-1within the DIANA-Summ project and by the Klaus Tschira Foundation.ReferencesBrants,T.(2000).TnT-a statistical part-of-speech tagger.In Pro-ceedings of the6th Conference on Applied Natural Language Processing,Seattle,Wash.,29April–4May2000,pp.224–231.Brill,E.(1994).Some advances in transformation based part-of-speech tagging.In Proceedings of the12th National Con-ference on Artificial Intelligence,Seattle,Wash.,1–4August 1994,pp.722–727.Clark,S.,J.R.Curran&M.Osborne(2003).Bootstrapping POS taggers using unlabelled data.In Proceeings of the Seventh CoNLL conference held at HLT-NAACL2003,Edmonton,Al-berta,Canada,27May–1June,2003,pp.49–55.Collins,M.(2002).Discriminative training methods for Hidden Markov Models:Theory and experiments with Perceptron al-gorithms.In Proceedings of the2002Conference on Empirical Methods in Natural Language Processing,Philadelphia,Pa., 6–7July2002.Godfrey,J.J.,E.Holliman&J.McDaniel(1992).Switchboard: Telephone speech corpus for research and development.In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing San Francisco,Cal.,USA,pp.517–520.Heeman,P.A.&J.F.Allen(1999).Speech repairs,intonational phrases,and discourse markers:Modeling speakers’utterances in spoken putational Linguistics,25(4):527–571.Janin,A.,D.Baron,J.Edwards,D.Ellis,D.Gelbart,N.Mor-gan,B.Peskin,T.Pfau,E.Shriberg,A.Stolcke&C.Wooters (2003).The ICSI meeting corpus.In Proceedings of IEEE In-ternational Conference on Acoustics,Speech and Signal Pro-cessing Hong Kong,China,6–10April2003,pp.364–367. Marcus,M.P., B.Santorini&M.A.Marcinkiewicz(1993).Building a large annotated corpus of English:The Penn putational Linguistics,19(2):313–330. Nakagawa,T.,T.Kudo&Y.Matsumoto(2002).Revision learning and its application to part-of-speech tagging.In Proceedings of the40th Annual Meeting of the Association for Computational Linguistics,Philadelphia,Penn.,7–12July2002,pp.497–504. Santorini,B.(1990).Part of Speech Tagging Guidelines for the Penn Treebank Project./tree-bank/home.html.Toutanova,K.,D.Klein,C.D.Manning&Y.Singer(2003).Feature-rich part-of-speech tagging with a cyclic dependency network.In Proceedings of the Human Language Technolgy Conference of the North American Chapter of the Association for Computational Linguistics,Edmonton,Alberta,Canada,27 May–1June,2003,pp.252–259.Toutanova,K.&C.D.Manning(2000).Enriching the knowl-edge sources used in a maximum entropy part-of-speech tagger.In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Cor-pora,Hong Kong,pp.63–70.van Halteren,H.,J.Zavrel&W.Daelemans(1998).Improv-ing data driven wordclass tagging by system combination.In Proceedings of the17th International Conference on Computa-tional Linguistics and36th Annual Meeting of the Association for Computational Linguistics,Montr´e al,Qu´e bec,Canada,10–14August1998,pp.491–497.Wermter,J.&U.Hahn(2004).Really,is medical sublanguage that different?Experimental counter-evidence from tagging medical and newspaper corpora.In Medinfo’0411th World Congress on Medical Informatics,pp.560–564.Zavrel,J.&W.Daelemans(2000).Bootstrapping a tagged cor-pus through combination of existing heterogeneous taggers.In Proceedings of the2nd International Conference on Language Resources and Evaluation,Athens,Greece,May,2000,pp.1–4.Zechner,K.(2001).Automatic Summarization of Spoken Di-alogues in Unrestricted Domains,(Ph.D.thesis).Language Technology Institute,School of Computer Science,Carnegie Mellon University,Pittsburgh,USA:School of Computer Sci-ence.。

A. Carry school flags. B. Wear fun clothes. C. Design fancy masks.
19. Where can the parents stand and watch?
A. On the sidewalk. B. On the playground. C. In front of the office.
2.Which program will you join if you're interested in drawing pictures?
A.Light Awash in Watercolor.B.Spineless Artists: Invertebrate Creativity.
C.Wheel Throwing.D.Knitting and Pom-Pom Making.
A. A new restaurant. B. A convenience store. C. Their office.
2. When is the class presentation according to Vicky?
A On Thursday. B. On Wednesday. C. On Tuesday.
20. What should the students do if they don't take part?
A. Let their teacher know. B. Stay in the lecture hall. C. Do some reading.
第二部分阅读（共两节,满分50分）
第一节（共15小题;每小题2. 5分,满分37. 5分）
Spineless Artists: Invertebrate Creativity