下载文档原格式
如何阅读与写作一篇科研论文-以Spark作者两篇经典论文1为例——剖析英文论文阅读与写作陈敏刚博士副研究员2016年6月13日1.第一遍阅读题目、摘要、引言和结论第一步是：快速浏览整篇论文，对其有个大概的了解。
你也可以自己决定是否需要进一步阅读。
这一步大概需要五到十分钟，有下面几个小节组成：1)认真阅读题目、摘要、引言。
2)阅读标题和段落开头，除此之外一概不看。
3)瞥一眼数学部分（如果有的话），以确定其基本理论概况。
4)阅读结论部分。
5)瞥一眼引用，找一下是否有你读过的论文。
1.1题目：Spark：Cluster Computing with Working Sets1.2 摘要【实例分析】This paper focuses on one such class of applications: those that reuse a working set of data across multiple parallel operations.We propose a new framework called Spark that supports these applications while retaining the scalability and fault tolerance of MapReduce.【英文写作】1Spark：Cluster Computing with Working Sets，2010Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing, 2012.1.3 引言问题所在——论文首先分析了现有分布式计算系统（如MapReduce 、Dryad ）的两个问题，即应对迭代式作业（机器学习算法）、交互式分析（SQL 查询）时导致效率损失。
本文贡献（本文主要工作）——提出并实现了弹性分布式数据集（RDD ）、RDD 基于Lineage （血缘）的容错，基于Scala 解释器修改的交互式操作接口等。
各位同学：下面这篇文章讲解了如何细读科研类文章，大家看一下，把握主要阅读方法。
当然，我们在泛读过程中，有些可粗读，有些细读。
希望大家把下文介绍的方法与我们课本的文章相结合并进行取舍，掌握科研类文章阅读方法。
How to Read a Scientific Research Paper--- a four-step guide for students and for facultyAnn McNeal, School of Natural Science, Hampshire College, Amherst MA 01002Reading research papers ("primary articles") is partly a matter of experience and skill, and partly learning the specific vocabulary of a field. First of all, DON'T PANIC! If you approach it step by step, even an impossible-looking paper can be understood.1. Skimming. Skim the paper quickly, noting basics like headings, figures and the like. This takes just a few minutes. You're not trying to understand it yet, but just to get an overview.2. Vocabulary. Go through the paper word by word and line by line, underlining or highlighting every word and phrase you don't understand. Don't worry if there are a lot of underlinings; you're still not trying to make sense of the article.Now you have several things you might do with these vocabulary and concept questions, depending upon the kind of question each is. You cana.Look up simple words and phrases. Often the question is simplyvocabulary--what's a lateral malleolus, or a christa, or the semilunar valve. A medical or biological dictionary is a good place to look for definitions. A textbook of physiology or anatomy may be a good source, because it gives more complete explanations. Your ordinary shelf dictionary is not a good source, because the definitions may not be precise enough or may not reflect the way in which scientists use a word (for example "efficiency" has a common definition, but the physical definition is much more restricted.)b.Get an understanding from the context in which it is used.Often words that are used to describe the procedures used in an experiment can be understood from the context, and may be very specific to the paper you are reading. Examples are the "lithium-free control group" in a rat experiment or the "caroteneextraction procedure" in a biochemical experiment. Of course, you should be careful when deciding that you understand a word from its context, because it might not mean what you think.c.Flag this phrase as belonging to one of the major concepts ofthe paper--it's bigger than a vocabulary question. For example, a paper about diet and cancer might refer to "risk reduction," which you would need to understand in context and in some depth.3. Comprehension, section by section. Try to deal with all the words and phrases, although a few technical terms in the Methods section might remain. Now go back and read the whole paper, section by section, for comprehension.In the Introduction, note how the context is set. What larger question is this a part of? The author should summarize and comment on previous research, and you should distinguish between previous research and the actual current study. What is the hypothesis of the paper and the ways this will be tested?In the Methods, try to get a clear picture of what was done at each step. What was actually measured? It is a good idea to make an outline and/or sketch of the procedures and instruments. Keep notes of your questions; some of them may be simply technical, but others may point to morefundamental considerations that you will use for reflection and criticism below.In Results, look carefully at the figures and tables, as they are the heart of most papers. A scientist will often read the figures and tables before deciding whether it is worthwhile to read the rest of the article! What does it mean to "understand" a figure? You understand a figure when you can redraw it and explain it in plain English words.The Discussion contains the conclusions that the author would like to draw from the data. In some papers, this section has a lot of interpretation and is very important. In any case, this is usually where the author reflects on the work and its meaning in relation to other findings and to the field in general.4. Reflection and criticism.After you understand the article and can summarize it, then you can return to broader questions and draw your own conclusions. It is very useful to keep track of your questions as you go along, returning to see whether they have been answered. Often, the simple questions may contain the seeds of very deep thoughts about the work--for example, "Why did the authors use a questionnaire at the end of the month to find out about premenstrual tension? Wouldn't subjects forget or have trouble recalling?"Here are some questions that may be useful in analyzing various kinds of research papers:Introduction:What is the overall purpose of the research?How does the research fit into the context of its field? Is it,for example, attempting to settle a controversy? show thevalidity of a new technique? open up a new field of inquiry?Do you agree with the author's rationale for studying thequestion in this way?Methods:Were the measurements appropriate for the questions theresearcher was approaching?Often, researchers need to use "indicators" because theycannot measure something directly--for example, usingbabies' birthweight to indicate nutritional status. Were themeasures in this research clearly related to the variables inwhich the researchers (or you) were interested?If human subjects were studied, do they fairly represent thepopulations under study?ResultsWhat is the one major finding?Were enough of the data presented so that you feel you can judge for yourself how the experiment turned out?Did you see patterns or trends in the data that the author did not mention? Were there problems that were not addressed? DiscussionDo you agree with the conclusions drawn from the data? Are these conclusions over-generalized or appropriately careful?Are there other factors that could have influenced, or accounted for, the results?What further experiments would you think of, to continue the research or to answer remaining questions?。
如何阅读学术文章更高效在现代学术研究中，面对浩瀚如海的学术文章，如何高效地阅读并提取有价值的信息成为每位研究者必须掌握的技能。
阅读学术文章并不仅仅是为了获取知识，还需要判断其可信度、应用性以及对自身研究的启发作用。
本文将从选文、快速阅读、深度分析和整理笔记几个方面，探讨如何更加高效地阅读学术文章。
一、选文：选择适合自己的文章高效阅读的第一步是选择合适的文献。
面对大量的学术资源，我们需要根据自身的研究主题和需求进行有效筛选。
1. 明确研究主题在开始搜索文献之前，首先要明确自己的研究主题或问题。
有了清晰的目标，才能更准确地筛选出相关文献。
可以通过关键字搜索、参考已有的文献综述以及与领域内专家的讨论来明确主题。
2. 使用科学数据库利用学术搜索引擎和数据库（如Google Scholar、PubMed、JSTOR等）进行文献检索，这些平台能够提供丰富的文献资源。
在选择文章时，要关注文章的影响因子、引用次数等指标，以判断其学术价值。
3. 查看摘要和关键词在找到相关文献后，首先阅读摘要和关键词。
摘要通常概述了研究的问题、方法和结果，可以帮助你判断这篇文章是否值得深入阅读。
如果摘要中提到的方法或结论与你的研究方向密切相关，那么可以进一步研读完整文章。
二、快速阅读：提高效率的方法当选定了要深入阅读的文章后，可以运用一些技巧来提高阅读效率，使自己更快地获取关键信息。
1. 通读标题、引言和结论在文章的初步阅读中，可以快速浏览标题、引言和结论部分。
标题提供了主题的直观信息，引言通常包含背景知识和研究目的，而结论则总结了主要发现。
这部分内容能够让你迅速捕捉到研究的核心思想和结论。
2. 浏览小节标题和图表在快速浏览过程中，可以扫描文章中的小节标题。
这些标题通常会提示各个部分的重要内容。
此外，图表和数据是理解研究成果的重要部分，通过分析图表，能够得到更加直观的信息，并理解数据背后的含义。
3. 标记关键信息在快速阅读中使用标记工具（如荧光笔或注释工具）来突出重点信息，如重要观点、数据支持和关键引用。
How to Read a PaperAugust2,2013S.KeshavDavid R.Cheriton School of Computer Science,University of WaterlooWaterloo,ON,Canadakeshav@uwaterloo.caABSTRACTResearchers spend a great deal of time reading research pa-pers.However,this skill is rarely taught,leading to much wasted effort.This article outlines a practical and efficient three-pass method for reading research papers.I also de-scribe how to use this method to do a literature survey. 1.INTRODUCTIONResearchers must read papers for several reasons:to re-view them for a conference or a class,to keep current in theirfield,or for a literature survey of a newfield.A typi-cal researcher will likely spend hundreds of hours every year reading papers.Learning to efficiently read a paper is a critical but rarely taught skill.Beginning graduate students,therefore,must learn on their own using trial and error.Students waste much effort in the process and are frequently driven to frus-tration.For many years I have used a simple‘three-pass’approach to prevent me from drowning in the details of a paper be-fore getting a bird’s-eye-view.It allows me to estimate the amount of time required to review a set of papers.Moreover, I can adjust the depth of paper evaluation depending on my needs and how much time I have.This paper describes the approach and its use in doing a literature survey.2.THE THREE-PASS APPROACHThe key idea is that you should read the paper in up to three passes,instead of starting at the beginning and plow-ing your way to the end.Each pass accomplishes specific goals and builds upon the previous pass:The first pass gives you a general idea about the paper.The second pass lets you grasp the paper’s content,but not its details.The third pass helps you understand the paper in depth.2.1Thefirst passThefirst pass is a quick scan to get a bird’s-eye view of the paper.You can also decide whether you need to do any more passes.This pass should take aboutfive to ten minutes and consists of the following steps:1.Carefully read the title,abstract,and introduction2.Read the section and sub-section headings,but ignoreeverything else3.Glance at the mathematical content(if any)to deter-mine the underlying theoretical foundations4.Read the conclusions5.Glance over the references,mentally ticking offtheones you’ve already readAt the end of thefirst pass,you should be able to answer thefive Cs:1.Category:What type of paper is this?A measure-ment paper?An analysis of an existing system?A description of a research prototype?2.Context:Which other papers is it related to?Whichtheoretical bases were used to analyze the problem?3.Correctness:Do the assumptions appear to be valid?4.Contributions:What are the paper’s main contribu-tions?5.Clarity:Is the paper well written?Using this information,you may choose not to read fur-ther(and not print it out,thus saving trees).This could be because the paper doesn’t interest you,or you don’t know enough about the area to understand the paper,or that the authors make invalid assumptions.Thefirst pass is ade-quate for papers that aren’t in your research area,but may someday prove relevant.Incidentally,when you write a paper,you can expect most reviewers(and readers)to make only one pass over it.Take care to choose coherent section and sub-section titles and to write concise and comprehensive abstracts.If a reviewer cannot understand the gist after one pass,the paper will likely be rejected;if a reader cannot understand the high-lights of the paper afterfive minutes,the paper will likely never be read.For these reasons,a‘graphical abstract’that summarizes a paper with a single well-chosenfigure is an ex-cellent idea and can be increasingly found in scientific jour-nals.2.2The second passIn the second pass,read the paper with greater care,but ignore details such as proofs.It helps to jot down the key points,or to make comments in the margins,as you read. Dominik Grusemann from Uni Augsburg suggests that you “note down terms you didn’t understand,or questions you may want to ask the author.”If you are acting as a paper referee,these comments will help you when you are writing your review,and to back up your review during the program committee meeting.1.Look carefully at thefigures,diagrams and other illus-trations in the paper.Pay special attention to graphs.Are the axes properly labeled?Are results shown witherror bars,so that conclusions are statistically sig-nificant?Common mistakes like these will separaterushed,shoddy work from the truly excellent.2.Remember to mark relevant unread references for fur-ther reading(this is a good way to learn more aboutthe background of the paper).The second pass should take up to an hour for an expe-rienced reader.After this pass,you should be able to grasp the content of the paper.You should be able to summarize the main thrust of the paper,with supporting evidence,to someone else.This level of detail is appropriate for a paper in which you are interested,but does not lie in your research speciality.Sometimes you won’t understand a paper even at the end of the second pass.This may be because the subject matter is new to you,with unfamiliar terminology and acronyms. Or the authors may use a proof or experimental technique that you don’t understand,so that the bulk of the pa-per is incomprehensible.The paper may be poorly written with unsubstantiated assertions and numerous forward ref-erences.Or it could just be that it’s late at night and you’re tired.You can now choose to:(a)set the paper aside,hoping you don’t need to understand the material to be successful in your career,(b)return to the paper later,perhaps after reading background material or(c)persevere and go on to the third pass.2.3The third passTo fully understand a paper,particularly if you are re-viewer,requires a third pass.The key to the third pass is to attempt to virtually re-implement the paper:that is, making the same assumptions as the authors,re-create the work.By comparing this re-creation with the actual paper, you can easily identify not only a paper’s innovations,but also its hidden failings and assumptions.This pass requires great attention to detail.You should identify and challenge every assumption in every statement. Moreover,you should think about how you yourself would present a particular idea.This comparison of the actual with the virtual lends a sharp insight into the proof and presentation techniques in the paper and you can very likely add this to your repertoire of tools.During this pass,you should also jot down ideas for future work.This pass can take many hours for beginners and more than an hour or two even for an experienced reader.At the end of this pass,you should be able to reconstruct the entire structure of the paper from memory,as well as be able to identify its strong and weak points.In particular,you should be able to pinpoint implicit assumptions,missing citations to relevant work,and potential issues with experimental or analytical techniques.3.DOING A LITERATURE SURVEYPaper reading skills are put to the test in doing a literature survey.This will require you to read tens of papers,perhaps in an unfamiliarfield.What papers should you read?Here is how you can use the three-pass approach to help.First,use an academic search engine such as Google Scholar or CiteSeer and some well-chosen keywords tofind three to five recent highly-cited papers in the area.Do one pass on each paper to get a sense of the work,then read their re-lated work sections.You willfind a thumbnail summary ofthe recent work,and perhaps,if you are lucky,a pointer toa recent survey paper.If you canfind such a survey,youare done.Read the survey,congratulating yourself on your good luck.Otherwise,in the second step,find shared citations and repeated author names in the bibliography.These are thekey papers and researchers in that area.Download the key papers and set them aside.Then go to the websites of thekey researchers and see where they’ve published recently. That will help you identify the top conferences in thatfield because the best researchers usually publish in the top con-ferences.The third step is to go to the website for these top con-ferences and look through their recent proceedings.A quick scan will usually identify recent high-quality related work. These papers,along with the ones you set aside earlier,con-stitute thefirst version of your survey.Make two passes through these papers.If they all cite a key paper that youdid notfind earlier,obtain and read it,iterating as neces-sary.4.RELATED WORKIf you are reading a paper to do a review,you should also read Timothy Roscoe’s paper on“Writing reviews for sys-tems conferences”[3].If you’re planning to write a technical paper,you should refer both to Henning Schulzrinne’s com-prehensive web site[4]and George Whitesides’s excellent overview of the process[5].Finally,Simon Peyton Joneshas a website that covers the entire spectrum of research skills[2].Iain H.McLean of Psychology,Inc.has put together a downloadable‘review matrix’that simplifies paper review-ing using the three-pass approach for papers in experimen-tal psychology[1],which can probably be used,with minor modifications,for papers in other areas.5.ACKNOWLEDGMENTSThefirst version of this document was drafted by my stu-dents:Hossein Falaki,Earl Oliver,and Sumair Ur Rahman.My thanks to them.I also benefited from Christophe Diot’s perceptive comments and Nicole Keshav’s eagle-eyed copy-editing.I would like to make this a living document,updating itas I receive comments.Please take a moment to email meany comments or suggestions for improvement.Thanks to encouraging feedback from many correspondents over the years.6.REFERENCES[1]I.H.McLean,“Literature Review Matrix,”/[2]S.Peyton Jones,“Research Skills,”/en-us/um/people/simonpj/papers/giving-a-talk/giving-a-talk.htm[3]T.Roscoe,“Writing Reviews for Systems Conferences,”http://people.inf.ethz.ch/troscoe/pubs/review-writing.pdf[4]H.Schulzrinne,“Writing Technical Articles,”/∼hgs/etc/writing-style.html[5]G.M.Whitesides,“Whitesides’Group:Writing a Paper,”/∼rlake/Whitesides writing res paper.pdf。
How to read a research paper.Later in the semester,we will talk about how to write a research paper.To begin the course,however,we consider how to read a research paper.This discussion presupposes that you have a good reason to carefully read a research paper–for example,the fact that I assign a paper is(probably)a good reason for you to read it.You may also need to carefully read a paper if you are asked to review it,or if it is relevant to your own research.We might also later discuss how to skim a paper,so that you can decide whether a paper is worth a careful reading.When you read a research paper,your goal is to understand the scientific contributions the authors are making.This is not an easy task.1It may require going over the paper several times.Expect to spend several hours to read a paper.Here are some initial guidelines for how to read a paper:Read critically:Reading a research paper must be a critical process.You should not assume that the authors are always correct.Instead,be suspicious.Critical reading involves asking appropriate questions.If the authors attempt to solve a problem,are they solving the right problem?Are there simple solutions the authors do not seem to have considered?What are the limitations of the solution(including limitations the authors might not have noticed or clearly admitted)?Are the assumptions the authors make reasonable?Is the logic of the paper clear and justifiable,given the assumptions,or is there aflaw in the reasoning?If the authors present data,did they gather the right data to substantiate their argument,and did they appear to gather it in the correct manner?Did they interpret the data in a reasonable manner?Would other data be more compelling?Read creatively:Reading a paper critically is easy,in that it is always easier to tear something down than to build it up.Reading creatively involves harder,more positive thinking.What are the good ideas in this paper?Do these ideas have other applications or extensions that the authors might not have thought of?Can they be generalized further?Are there possible improvements that might make important practical differences?If you were going to start doing research from this paper,what would be the next thing you would do?Make notes as you read the paper:Many people cover the margins of their copies of papers with e whatever style you prefer.If you have questions or criticisms,write them down so you do not forget them.Underline key points the authors make.Mark the data that is most important or that appears questionable.Such efforts help the first time you read a paper and pay big dividends when you have to re-read a paper after several months.1It would be easier if more research papers were well written...but again,we will discuss writing later on.After thefirst read-through,try to summarize the paper in one or two sentences.Almost all good research papers try to provide an answer a specific question.(Sometimes the question isa natural one that people specifically set out to answer;sometimes a good idea just ends up answering aworthwhile question.)If you can succinctly describe a paper,you have probably recognized the question the authors started with with and the answer they provide.Once you have focused on the main idea,you can go back and try to outline the paper to gain insight into more specific details.Indeed,if summarizing the paper in one or two sentences is easy,go back and try to deepen your outline by summarizing the three or four most important subpoints of the main idea.If possible,compare the paper to other works.Summarizing the paper is one way to try to determine the scientific contribution of a paper.But to really guage the scientific merit,you must compare the paper to other works in the area.Are the ideas really novel,or have they appeared before?(Of course we do not expect you to be experts and know the areas ahead of time in this class!)It is worth mentioning that scientific contributions can take on many forms.Some papers offer new ideas;others implement ideas,and show how they work;others bring previous ideas together and unite them under a novel framework.Knowing other work in the area can help you to determine which sort of contribution a paper is actually making.For this class,I will often ask you to provide a short,one page review of a paper.Although this may sound like a simple assignment,I expect that it will take a significant amount of time,especially in the beginning. (Remember,I am expecting it to take several hours just to read the paper!)Keeping the above in mind as you read the paper should make the process easier.Your one page review should include the following:a one or two sentence summary of the paper.a deeper,more extensive outline of the main points of the paper,including for example assumptionsmade,arguments presented,data analyzed,and conclusions drawn.any limitations or extensions you see for the ideas in the paper.your opinion of the paper;primarily,the quality of the ideas and its potential impact.。
HOW TO READ A RESEARCH PAPERby Spencer RugaberAmong the questions that you should ask yourself when reading a researchpaper are the following.1. What is the research paradigm that the author is using? Example paradigmsare psychological experiments, formalization and theorem proving, andartifact design and construction. If the paper is part of a wellestablished field, you should describe the field and its current state.2. What is the problem area with which the paper is concerned? For example,"Automatic Generation of Compilers from Denotational Semantic Descriptionsof the Source Code" would describe a research paper on compilation.3. What is the author's thesis? That is, what is he/she trying to convinceyou of?4. Summarize the author's argument. That is, how does the author go abouttrying to convince you of the thesis?5. Does the author describe other work in the field? If so, how does theresearch described in the paper differ from the other work?6. Does the paper succeed? Are you convinced of the thesis by the time thatyou have finished reading the paper?7. Does the author indicate how the work should be followed up on? Does thepaper generate new ideas.8. Some papers implicitly or explicitly provide a new way of doing things orof thinking about problems. If your paper does so, describe the approach.。
HowtoReadaScientificArticle（如何阅读科学论文）How to Critically Read a Scientific PaperA general strategy for reading and understanding a scientific paper is to read the material critically. There are two underlying themes to this strategy. First, one must ask his/her own questions about the material (i.e. about the methods, data, figures, concept, etc.), and attempt to answer them after careful analysis of the paper. Second, one must integrate the information from the paper into his/her own larger body of knowledge. This second theme is most suc cessfully accomplished by drawing a graphic model of the “new reality” which the paper reveals.Step 1. Looking at the PicturesBefore you begin reading the paper look at and try to understand each figure, table and graph. This is the focal point of the paper. The text will eventually help you to understand them completely, but this step will put you in “Question Asking Mode” and will help you to get started on your critical examination of the paper.1.Look at each figure, graph, or table and try to understand what is being presented.Read the figure legends and notes associated with each piece of data.2.Make brief notes on what you do understand about each figure (i.e. what is beingmeasured? What technique was used to generate the data? What were theindependent variables, dependent variables and controls?). Determine if youunderstand the concept behind the different techniquesused to generate the data.If you do not then make sure you learn about them before you even attempt toread the paper (i.e. go to your textbooks).3.For anything (and everything) that you don’t understand, write an explicitquestion (i.e. What is that smallest band in Lane 3 on this protein gel? or Whydoes the mutant strain have the highest enzyme activity?). Try to answer yourown questions (take a stab at it) based on what you know.4.After you finish looking at the pictures, tables and graphs make a guess aboutwhat methods you expect to see described in the materials and methods section.Write these down. Once again, if you do not understand any of these techniquesthen do some research before you begin reading the paper. It is very difficult totry to understand research wh en you don’t know how it was done.Step 2. Read through once.1.Read the paper from beginning to end.2.Mark or highlight every place where one of your earlier questions is answered.3.Formulate new questions about things you do not understand. Make sure you arevery specific in your questions (i.e. don’t just write, huh?).Step 3. Critical AnalysisEvery scientific paper is like a narrative of things that theresearchers thought and did. The formal structure of a modern paper (i.e. Title, Abstract, Materials and Methods, Results, Discussion) obscures the thought process that directed the research, but this process can be rediscovered by careful examination. This step is extremely important in order to understand the material.Steps in the critical analysis:1.Significant prior knowledge/current ignorance. What specific information(from earlier work) formed the basis for the experiments reported in the paper.You can usually find this information in the introduction. In your own wordsstate what is known about the research and what is still unknown.2.The Main Hypothesis and Alternatives. All research is directed by a hypothesisand one or more alternative hypotheses. In your own words, state all of thehypotheses directing the research.3.Assumptions (Explicit and Implicit). Often the authors will tell you that theyassumed something would (or would not) affect their experimental results. Thisis an explicit assumption. Authors also imply some assumptions (implicitassumptions). These are rather subtle. An example might be that they assumeOhm’s Law was working when they ran the electrophoresis, or that there are noplasmids in a strain if plasmid DNA cannot be isolated from the strain. Identifyall of the assumptions (implicit and explicit) made and state why you agree ordisagree with them.Note: Hypotheses are things that are tested in the research and assumptions are things that are not tested. Learn to distinguish between them.4.Elements of Support or non-support. Questions are answered and hypothesesare either supported or not supported using experimental procedures (i.e. by thedata). Look at the data again and re-read the materials and methods section andthe results section. Were the methods appropriate? Did they offer evidence tosupport the hypothesis? Was the data interpreted correctly? For example, was the gel band seen in lane 1 really brighter than the control or would you interpretsomething differently. It is important to not assume that just because a paper was published that it is the gospel truth. Remember that you are training to be ascientist; therefore, you need to be convinced that the data supports the hypothesis.5.Change in Reality? Did the data support any of the hypotheses? Be able toexplain why or why not. Was this work a significant contribution to science orjust a small contribution? Can this research be applicable to some greater cause?How does this work change our understanding? Draw a graphical model showing what was considered correct before the work was done and draw a model showing what is now considered correct after the work was done. Examine how the twomodels differ (if at all).6.The Next Step. When a hypothesis is supported there is always another questionthat arises. This is a natural occurrence in scientific research. State one or a fewnew questions that come to your mind. Predict what new set of experiments canbe done to answer the new questions. Are there any new experiments can be done to give further support for the previous hypothesis?。
How to Read a PaperAugust2,2013S.KeshavDavid R.Cheriton School of Computer Science,University of WaterlooWaterloo,ON,Canadakeshav@uwaterloo.caABSTRACTResearchers spend a great deal of time reading research pa-pers.However,this skill is rarely taught,leading to much wasted effort.This article outlines a practical and efficient three-pass method for reading research papers.I also de-scribe how to use this method to do a literature survey. 1.INTRODUCTIONResearchers must read papers for several reasons:to re-view them for a conference or a class,to keep current in theirfield,or for a literature survey of a newfield.A typi-cal researcher will likely spend hundreds of hours every year reading papers.Learning to efficiently read a paper is a critical but rarely taught skill.Beginning graduate students,therefore,must learn on their own using trial and error.Students waste much effort in the process and are frequently driven to frus-tration.For many years I have used a simple‘three-pass’approach to prevent me from drowning in the details of a paper be-fore getting a bird’s-eye-view.It allows me to estimate the amount of time required to review a set of papers.Moreover, I can adjust the depth of paper evaluation depending on my needs and how much time I have.This paper describes the approach and its use in doing a literature survey.2.THE THREE-PASS APPROACHThe key idea is that you should read the paper in up to three passes,instead of starting at the beginning and plow-ing your way to the end.Each pass accomplishes specific goals and builds upon the previous pass:The first pass gives you a general idea about the paper.The second pass lets you grasp the paper’s content,but not its details.The third pass helps you understand the paper in depth.2.1Thefirst passThefirst pass is a quick scan to get a bird’s-eye view of the paper.You can also decide whether you need to do any more passes.This pass should take aboutfive to ten minutes and consists of the following steps:1.Carefully read the title,abstract,and introduction2.Read the section and sub-section headings,but ignoreeverything else3.Glance at the mathematical content(if any)to deter-mine the underlying theoretical foundations4.Read the conclusions5.Glance over the references,mentally ticking offtheones you’ve already readAt the end of thefirst pass,you should be able to answer thefive Cs:1.Category:What type of paper is this?A measure-ment paper?An analysis of an existing system?A description of a research prototype?2.Context:Which other papers is it related to?Whichtheoretical bases were used to analyze the problem?3.Correctness:Do the assumptions appear to be valid?4.Contributions:What are the paper’s main contribu-tions?5.Clarity:Is the paper well written?Using this information,you may choose not to read fur-ther(and not print it out,thus saving trees).This could be because the paper doesn’t interest you,or you don’t know enough about the area to understand the paper,or that the authors make invalid assumptions.Thefirst pass is ade-quate for papers that aren’t in your research area,but may someday prove relevant.Incidentally,when you write a paper,you can expect most reviewers(and readers)to make only one pass over it.Take care to choose coherent section and sub-section titles and to write concise and comprehensive abstracts.If a reviewer cannot understand the gist after one pass,the paper will likely be rejected;if a reader cannot understand the high-lights of the paper afterfive minutes,the paper will likely never be read.For these reasons,a‘graphical abstract’that summarizes a paper with a single well-chosenfigure is an ex-cellent idea and can be increasingly found in scientific jour-nals.2.2The second passIn the second pass,read the paper with greater care,but ignore details such as proofs.It helps to jot down the key points,or to make comments in the margins,as you read. Dominik Grusemann from Uni Augsburg suggests that you “note down terms you didn’t understand,or questions you may want to ask the author.”If you are acting as a paper referee,these comments will help you when you are writing your review,and to back up your review during the program committee meeting.1.Look carefully at thefigures,diagrams and other illus-trations in the paper.Pay special attention to graphs.Are the axes properly labeled?Are results shown witherror bars,so that conclusions are statistically sig-nificant?Common mistakes like these will separaterushed,shoddy work from the truly excellent.2.Remember to mark relevant unread references for fur-ther reading(this is a good way to learn more aboutthe background of the paper).The second pass should take up to an hour for an expe-rienced reader.After this pass,you should be able to grasp the content of the paper.You should be able to summarize the main thrust of the paper,with supporting evidence,to someone else.This level of detail is appropriate for a paper in which you are interested,but does not lie in your research speciality.Sometimes you won’t understand a paper even at the end of the second pass.This may be because the subject matter is new to you,with unfamiliar terminology and acronyms. Or the authors may use a proof or experimental technique that you don’t understand,so that the bulk of the pa-per is incomprehensible.The paper may be poorly written with unsubstantiated assertions and numerous forward ref-erences.Or it could just be that it’s late at night and you’re tired.You can now choose to:(a)set the paper aside,hoping you don’t need to understand the material to be successful in your career,(b)return to the paper later,perhaps after reading background material or(c)persevere and go on to the third pass.2.3The third passTo fully understand a paper,particularly if you are re-viewer,requires a third pass.The key to the third pass is to attempt to virtually re-implement the paper:that is, making the same assumptions as the authors,re-create the work.By comparing this re-creation with the actual paper, you can easily identify not only a paper’s innovations,but also its hidden failings and assumptions.This pass requires great attention to detail.You should identify and challenge every assumption in every statement. Moreover,you should think about how you yourself would present a particular idea.This comparison of the actual with the virtual lends a sharp insight into the proof and presentation techniques in the paper and you can very likely add this to your repertoire of tools.During this pass,you should also jot down ideas for future work.This pass can take many hours for beginners and more than an hour or two even for an experienced reader.At the end of this pass,you should be able to reconstruct the entire structure of the paper from memory,as well as be able to identify its strong and weak points.In particular,you should be able to pinpoint implicit assumptions,missing citations to relevant work,and potential issues with experimental or analytical techniques.3.DOING A LITERATURE SURVEYPaper reading skills are put to the test in doing a literature survey.This will require you to read tens of papers,perhaps in an unfamiliarfield.What papers should you read?Here is how you can use the three-pass approach to help.First,use an academic search engine such as Google Scholar or CiteSeer and some well-chosen keywords tofind three to five recent highly-cited papers in the area.Do one pass on each paper to get a sense of the work,then read their re-lated work sections.You willfind a thumbnail summary ofthe recent work,and perhaps,if you are lucky,a pointer toa recent survey paper.If you canfind such a survey,youare done.Read the survey,congratulating yourself on your good luck.Otherwise,in the second step,find shared citations and repeated author names in the bibliography.These are thekey papers and researchers in that area.Download the key papers and set them aside.Then go to the websites of thekey researchers and see where they’ve published recently. That will help you identify the top conferences in thatfield because the best researchers usually publish in the top con-ferences.The third step is to go to the website for these top con-ferences and look through their recent proceedings.A quick scan will usually identify recent high-quality related work. These papers,along with the ones you set aside earlier,con-stitute thefirst version of your survey.Make two passes through these papers.If they all cite a key paper that youdid notfind earlier,obtain and read it,iterating as neces-sary.4.RELATED WORKIf you are reading a paper to do a review,you should also read Timothy Roscoe’s paper on“Writing reviews for sys-tems conferences”[3].If you’re planning to write a technical paper,you should refer both to Henning Schulzrinne’s com-prehensive web site[4]and George Whitesides’s excellent overview of the process[5].Finally,Simon Peyton Joneshas a website that covers the entire spectrum of research skills[2].Iain H.McLean of Psychology,Inc.has put together a downloadable‘review matrix’that simplifies paper review-ing using the three-pass approach for papers in experimen-tal psychology[1],which can probably be used,with minor modifications,for papers in other areas.5.ACKNOWLEDGMENTSThefirst version of this document was drafted by my stu-dents:Hossein Falaki,Earl Oliver,and Sumair Ur Rahman.My thanks to them.I also benefited from Christophe Diot’s perceptive comments and Nicole Keshav’s eagle-eyed copy-editing.I would like to make this a living document,updating itas I receive comments.Please take a moment to email meany comments or suggestions for improvement.Thanks to encouraging feedback from many correspondents over the years.6.REFERENCES[1]I.H.McLean,“Literature Review Matrix,”/[2]S.Peyton Jones,“Research Skills,”/en-us/um/people/simonpj/papers/giving-a-talk/giving-a-talk.htm[3]T.Roscoe,“Writing Reviews for Systems Conferences,”http://people.inf.ethz.ch/troscoe/pubs/review-writing.pdf[4]H.Schulzrinne,“Writing Technical Articles,”/∼hgs/etc/writing-style.html[5]G.M.Whitesides,“Whitesides’Group:Writing a Paper,”/∼rlake/Whitesides writing res paper.pdf。
Hello, everyone.I am pleased to be able to write you this letter, and introduce you how to read a research paper.Researchers must read papers for several reasons: to review papers for a conference or a class, to keep current in their field, or for a literature survey of a new field. A typical researcher will likely spend hundreds of hours every year reading papers. Learning to efficiently read a paper is a critical but rarely taught skill. Beginning graduate students, therefore, must learn on their own using trial and error. Students waste much effort in the process and are frequently driven to frustration. I will share my personal experience may save you time.First of all, before beginning to read a paper, consider why you are doing it. What is the target? What do you wish to get from it? Different target means different level of effort for doing it. If you only need an overview, a brief skim is enough. If you will present the paper to others, i.e. introducing it in group discussion, you will need to dig deeper. If you mean to use the information later, taking a note will help you remember it. BUT if you don’t know what you hope to gain from the paper, don’t do it at all, it is a complete waste of time.Then, when you first approach a paper, find the answer to the most important question: What did the author do? To answer this question, you should only need to examine the title and the abstract. When you find the answer, you can decide if the paper is useful to you now. If so, read it. If not, another question: Is it will useful in the future? If so, file it, or skip it. Figure 1 illustrates this simple decision-making process.Figure 1. Deciding to Read a PaperWhen you decide to read a paper. You should know the purpose of the authors. Writing a research paper is to declare that the authors did a very scientific contribution to the research. Reading the paper is to find out what the contribution is.To describe the contribution, the authors should define the problem which they devote to, and highlight the importance and broad usage. And then, they describe the proposed solution. And often they will cite others’ work which to solve the same problem, and compare their solution to others’ to announce their work is new and different from others’. To support the correctness, effectiveness and appropriateness of their solution, they will provide evidence. Those evidence maybe theoretical proof or experiment data analysis. Then they arrive at the conclusion. The work is finished. You can imagine when writing the paper, the authors actually say to you:：That’s a problem! You see, it is very, very important!：You know that is a difficult problem. Many other researchers attempt to solve it, but failed!：Here! Here is my solution.：Maybe, I should explain it in some more details ……：Now, here, my experiment proves it! Ok?：Oh, you see, it is a great job!：Praise me!Figure 2. What is the Paper SaysSo that, research paper writing in a very purposeful way. The authors will try their best to make you believe they do a great contribution. So you should read the paper critically. Reading a researchpaper must be a critical process. You should not assume that the authors are always correct. Instead, be suspicious. You may notice:If the authors state the problem is important, is it really an important problem?If the authors attempt to solve a problem, are they solving the right problem? Are there simple solutions the authors do not seem to have been considered?If the authors state other researchers did not solve it correctly, are the authors give a fair judgment?What are the limitations of the solution (including limitations the authors might not have noticed or clearly admitted)?Is the logic of the paper clear and justifiable? Is the conclusion comes out naturally? Is there a flaw in the reasoning?If the authors present experiment data, are the data substantiate their argument? Did they perform the experiment in a correct manner? Did they interpret the data in a reasonable manner?Figure 3. Be Suspicious When Reading PapersReading critically are not enough, you should also read the paper creatively. Reading creatively involves you harder, more positive thinking. You should think:What are the good ideas in this paper?Do these ideas have other applications or extensions?Can they be generalized further?Are there possible improvements that might make important practical differences?If you were going to start doing research from this paper, what would be the next thing you would do?Figure 4. Thinking When Reading PapersFinding out those answers is all things which should be issued once you read a paper. Let’s make a checklist:What are message you take-away from this paper?What is the problem which the paper focusses on?What is the proposed solution?What are their contribution?What are future directions for this research?What question are you left with?Figure 5. Paper Reading ChecklistOK, you know the goal of reading, then I will show the trick to reach the goal.My method is:Reading the paper several times,at different level of detail.Research paper is written in a heavy repetitive style. We can get the whole idea at different level of detail.The first level: the title only. The title is the most important. The authors state the most important thing which they believe to be in the title. To understand the paper you should understand the thing state in the title. For example, in the Attachment 1, the title of this paper is Deep BeliefNetworks for phone recognition. This title state 2 thing: firstly, the Deep Belief Networks seems a model’s name, it may be the proposed solution. Secondly, for phone recognition, it is the problem. The title tells us this paper will use deep belief networks to solve the phone recognition problem. If you have the basic background, you know the phone recognition problem is important and useful. If this paper gives a good result, it will be a contribution to the research. And you need to find out: 1, what is deep belief network? And 2, how to use it for phone recognition?The second level: the abstract. Abstract will give the whole idea of this paper, almost every question listed in our checklist (Figure 5) can be solved according to the abstract. We use the same example as before. The abstract of that paper is:1,Hidden Markov Models (HMMs) have been the state-of-the-art techniques for acoustic modeling despite their unrealistic independence assumptions and the very limited representational capacity of their hidden states. 2,There are many proposals in the research community for deeper models that are capable of modeling the many types of variability present in the speech generation process. 3,Deep Belief Networks (DBNs) have recently proved to be very effective for a variety of machine learning problems and this paper applies DBNs to acoustic modeling. 4, On the standard TIMIT corpus, DBNs consistently outperform other techniques and the best DBN achieves a phone error rate (PER) of 23.0% on the TIMIT core test set.Figure 6. AbstractAs you see, in the abstract there are only 4 sentences. 1st sentence says HMMs have disadvantages. This may be the concrete problem. Phone recognition is a big problem and this may be the concrete problem: HMMs have the disadvantages, how can we overcome those disadvantages (with deep belief networks)? 2nd sentence says use deeper model may solve it. Yes, it is the solution. 3rd sentence says in this paper the deep belief networks be used to solve the problem. It is the solution, too, but in more detail. And 4th sentence the authors present experiment data to support their proposed solution.In the abstract there may be some terminology you don’t understand, you can first search it in the paper, look for more detail about the terminology (i.e. use search function in your PDF browser). Oh, maybe it is not to be explained in the text. It often means the terminology is common sense in the research field which the paper devotes to. For those terminologies, Wikipedia will be a good information source. And if you still can’t understand what means it is with Wikipedia help, it turns out that you are very unfamiliar with this field, a basic book will be a good start point.The third level: the introduction and conclusion. In the introduction, it often describes the problem in more detail and describes the proposed solution briefly. In this pass, you can get further understand about the problem. And you can imagine if you are asked to solve this problem, how to do it? And follow the brief description on the proposed solution, how to get it done? In the introduction, there may be a comparison between this work and ot hers’ work. If you familiar with the relevant literature, you can use this information to place the work into your whole understand about the field. Then go to the conclusion. In this part, the authors will give summaries of their work and state the contributions they have done. Maybe a future work also be listed here. It may value to read.The next level is the proposed solution itself. This part is free to write, and has no fixed format, and probably difficult to understand. To read this part, you can read the section title first. In this approach, you can first get the outline of the proposed solution, and the roadway of the authors explains their work to you. For example, the section title in the before paper is:2 Deep belief networks2.1 Restricted Boltzmann machines2.2 RBM training2.2.1 Generative training of an RBM2.2.2 Discriminative training of a DBN using back propagation2.2.3 Hybrid training of an RBM2.3 DBN structure2.4 Using DBNs for phone recognition2.5 Generalized softmax (GSM) output layerFigure 7. Outline of ExplanationFrom the section title, as we think, it will answer the question: What is the deep belief network? And how to use it for phone recognition? Section 2.1 introduces the restricted Boltzmann machines. These (RBMs) may have relationships with the deep belief network. What is the relationship? You should find out. Then section 2.2 explains how to train the RBM. From the subsection title, the training process will split into separate aspects or different steps. Then section 2.3 describes the deep belief network. Section 2.4 describes how to use the deep belief network for phone recognition. And how about section 2.5, it may be a special concept which used in this work. When you get the outline you can easier understand what are those sections said. And you are not easier to be stuck in the technical detail.This part probably long, and seems like a mammoth task. You can break it into small tasks, read a section or a subsection once. Those will be easier tasks and you will be more confident of finishing those tasks. When you get the meaning of every section, then you can fill them into your entire framework, and come up with a complete understand about the paper.Someone may be afraid of reading mathematical formulas, and always stick in the formulas. I tell you. Those formulas are often not as important as you think. Formulas just a form to explain the idea. The important idea always be explained several times, in different forms. You can understand the idea by comparing the different expression forms.And again, the detail does not usually as important as you think. Even some detail absent from your understanding, you can understand the entire paper as well. You can safely skip those details which difficult to obscure when you go through the paper. And when you get a high level concept of the paper, then go back, you can decide which detail is decisive one and which is nothing else matter. You should dedicate to the decisive one, and ignore others.The last topic is how to read experiment data. The experiment can skim quickly, but focus on the tables and charts. Tables and charts are compiled by experiment data. These include a lot of information. You should notice the title of the table or chart. The title tells you what data this table or chart care for. For example in the figure on the next page, which is extracted from the Attachment 1’s figure 3, it title is The effect of the model depth on PER, we will get known about the trend of PER changes with the mode depth. The next thing to note is the axis of the chart, you should getknown what the means of the axis is. Then you can look for how the authors interpret the data, is it reasonable and coincided with you?Figure 8. ChartAfter reading a paper, I ask you to provide a short, one page review of this paper. I named it cover because it covers the main idea of the paper. Write a cover is not only a way to share your reading work to the group, but also a useful memo to recall the memory of the paper when you need it later.A cover may include the paper summary, a deeper and more extensive outline of the main points of the paper, any limitations or extensions you see the ideas in the paper, and your opinion of the paper. Also you can just fill the checklist more carefully.Good luckYoursHui ZhangＡｔｔａｃｈｍｅｎｔ １Deep Belief Networks for phone recognitionAbdel-rahman Mohamed,George Dahl,and Geoffrey HintonDepartment of Computer ScienceUniversity of Toronto{asamir,gdahl,hinton}@AbstractHidden Markov Models(HMMs)have been the state-of-the-art techniques foracoustic modeling despite their unrealistic independence assumptions and the verylimited representational capacity of their hidden states.There are many proposalsin the research community for deeper models that are capable of modeling themany types of variability present in the speech generation process.Deep BeliefNetworks(DBNs)have recently proved to be very effective for a variety of ma-chine learning problems and this paper applies DBNs to acoustic modeling.Onthe standard TIMIT corpus,DBNs consistently outperform other techniques andthe best DBN achieves a phone error rate(PER)of23.0%on the TIMIT core testset.1IntroductionA state-of-the-art Automatic Speech Recognition(ASR)system typically uses Hidden Markov Mod-els(HMMs)to model the sequential structure of speech signals,with local spectral variability mod-eled using mixtures of Gaussian densities.HMMs make two main assumptions.Thefirst assumption is that the hidden state sequence can be well-approximated using afirst order Markov chain where each state S t at time t depends only on S t−1.Second,observations at different time steps are as-sumed to be conditionally independent given a state sequence.Although these assumptions are not realistic,they enable tractable decoding and learning even with large amounts of speech data.Many methods have been proposed for relaxing the very strong conditional independence assumptions of standard HMMs(e.g.[1,2,3,4]).Substantial research effort has been devoted to going beyond the“beads-on-a-string”view of speech to representing structure in speech above the level of the phonetic segment[5,6,7].This has led to promising results using segment-and landmark-based methods for phone recognition(e.g,[8,9]).The limited representational capacity of HMMs prevents them from modeling streams of interact-ing knowledge sources in the speech signal which may require deeper architectures with multiple layers of representations.The work in[10]proposes a hierarchical framework where each layer is designed to capture a set of distinctive feature landmarks.For each feature,a specialized acoustic representation is constructed in which that feature best expresses itself.In[11],a probabilistic gen-erative model is introduced where the dynamic structure in the hidden vocal tract resonance space is used to characterize long-span contextual influence across phonetic units.Feedforward neural networks were used in other multilayer frameworks,such as the TRAP architecture[12].The TRAP architecture uses a one second long feature vector that describes segments of temporal evolution of critical-band spectral densities within a single critical band.Sub-word posterior probabilities are estimated using feedforward neural networks for each critical band which are merged to produce thefinal estimation of posterior probabilities using another feedforward neural network in the last layer.In[13],the split temporal context system is introduced which modifies the TRAP system by including splits over time as well as over frequency bands in the middle layer of the system before thefinal merger neural network.1In this work,we propose using Deep Belief Networks(DBNs)[14]to model the spectral variabil-ities in speech.DBNs are probabilistic generative models that are composed of multiple layers of stochastic latent variables with Restricted Boltzmann Machines(RBMs)as their building blocks. DBNs have a greedy layer-wise unsupervised learning algorithm as well as a discriminativefine-tuning procedure for optimizing performance on classification tasks.DBNs and related models have been used successfully for hand-written character recognition[14, 15],3-D object recognition[16],information retrieval[17,18],motion capture data modeling[19, 20],and machine transliteration[21].2Deep belief networksLearning is difficult in densely connected,directed belief nets that have many hidden layers because it is difficult to infer the posterior distribution over the hidden variables,when given a data vector, due to the phenomenon of explaining away.Markov chain Monte Carlo methods[22]can be used to sample from the posterior,but they are typically very time-consuming.In[14]complementary priors were used to eliminate the explaining away effects producing a train-ing procedure which is equivalent to training a sequence of restricted Boltzmann machines(RBMs) [23].An RBM is a bipartite graph in which visible units that represent observations are connected to binary,stochastic hidden units using undirected weighted connections.They are restricted in the sense that there are no visible-visible or hidden-hidden connections.RBMs have an efficient training procedure which makes them suitable as building blocks for Deep Belief Networks(DBNs).2.1Restricted Boltzmann machinesAn RBM[figure1-(a)]is a particular type of Markov Random Field(MRF)that has one layer of binary stochastic hidden units and one layer of binary stochastic visible units,although the units need not be Bernoulli random variables and can in fact have any distribution in the exponential family [24].Typically,all visible units are connected to all hidden units.The weights on the connections and the biases of the individual units define a probability distribution over the binary state vectors v of the visible units via an energy function.The energy of the joint configuration(v,h)is given by [24]:E(v,h;θ)=−Vi=1Hj=1w ij v i h j−Vi=1b i v i−Hj=1a j h j(1)whereθ=(w,b,a)and w ij represents the symmetric interaction term between visible unit i and hidden unit j while b i and a j are their bias terms.V and H are the numbers of visible and hidden units.The probability that the model assigns to a visible vector v is:p(v;θ)=h e−E(v,h)u h e−E(u,h)(2)Since there are no hidden-hidden or visible-visible connections,the conditional distributions p(v|h) and p(h|v)are factorial and are given by:p(h j=1|v;θ)=σ(Vi=1w ij v i+a j)p(v i=1|h;θ)=σ(Hj=1w ij h j+b i),(3)whereσ(x)=(1+e−x)−1.To train an RBM to model the joint distribution of data and class labels, the visible vector is concatenated with a binary vector of class labels.The energy function becomes:2E(v,l,h;θ)=−Vi=1Hj=1w ij h j v i−Ly=1Hj=1w yj h j l y−Hj=1a j h j−Ly=1c y l y−Vi=1b i v i(4)p(l y=1|h;θ)=softmax(Hj=1w yj h j+c y).(5)Furthermore,p(l|v)can be computed exactly using:p(l|v)=h e−E(v,l,h)l h e−E(v,l,h).(6)The value of p(l|v)can be computed efficiently by exploiting the conditional independence of the hidden units,which allows the hidden units to be marginalized out in a time that is linear in theFigure1:The DBN is composed of RBMs.2.2RBM training2.2.1Generative training of an RBMFollowing the gradient of the joint likelihood function of data and labels,the update rule for the visible-hidden weights is∆w ij= v i h j data− v i h j model(7) The expectation v i h j data is the frequency with which the visible unit v i and the hidden unit h j areon together in the training set and v i h j model is that same expectation under the distribution defined by the model.The term . model takes exponential time to compute exactly so the Contrastive Divergence(CD)approximation to the gradient is used instead[15].The new update rule becomes:∆w ij= v i h j data− v i h j 1(8)where . 1represents the expectation with respect to the distribution of samples from running the Gibbs sampler initialized at the data for one full step.2.2.2Discriminative training of a DBN using backpropagationThe RBM pretraining procedure of a DBN can be used to initialize the weights of a deep neural network,which can then be discriminativelyfine-tuned by backpropagating error derivatives.The “recognition”weights of the DBN become the weights of a standard neural network.32.2.3Hybrid training of an RBMIn cases where the RBM models the joint distribution of visible data and class labels,a hybrid train-ing procedure can be used tofine-tune the generatively trained parameters.Since the log conditional probability,log p(l|v),can be computed exactly,the gradient can also be computed exactly.The update rule for the visible-hidden weights is∆w ij=Hj=1σ(a j+w jy+Vi=1w ij v i)v i−Ll=1Hj=1σ(a j+w jl+Vi=1w ij v i)p(l|v)v i(9)To avoid model overfitting,we follow the gradient of a hybrid function f(v,l)which contains both generative and discriminative components:f(v,l)=αp(l|v)+p(v|l)(10) In this case p(v|l)works as a regularizer and is learned by using the original labels with the recon-structed data to infer the states of the hidden units at the end of the sampling step.Theαparameter controls the emphasis given to the discriminative component of the objective function.Since the original labels are used during hidden layer reconstruction for evaluating p(v|l),the label biases are updated using the gradient of p(l|v)only.2.3DBN structureEach layer of hidden units learns to represent features that capture higher order correlations in the original input data[figure1-(b)].The key idea behind training a deep belief net by training a se-quence of RBMs is that the model parameters,θ,learned by an RBM define both p(v|h,θ)and the prior distribution over hidden vectors,p(h|θ),so the probability of generating a visible vector,v, can be written as:p(v)= h p(h|θ)p(v|h,θ)(11) After learningθ,p(v|h,θ)is kept while p(h|θ)can be replaced by a better model that is learned by treating the hidden activity vectors produced from the training data as the training data for another RBM.This replacement improves a variational lower bound on the probability of the training data under the composite model[14].So a DBN can be viewed as an RBM that defines a prior over the top layer of hidden variables in a directed belief net,combined with a set of“recognition”weights to perform fast approximate inference.2.4Using DBNs for phone recognitionIn order to apply DBNs withfixed input and output dimensionality to phone recognition,a context window of n successive frames of feature vectors is used to set the states of the visible units of the lower layer of the DBN which produces a probability distribution over the possible labels of the the central frame.To generate phone sequences,a sequence of probability distributions over the possible labels for each frame are fed into a standard Viterbi decoder.We employed two general types of DBN architectures.Both types use greedy layer-wise Contrastive Divergence(CD)pretraining for initializing weights.Thefirst architecture[figure2-(a)]adds afinal layer of variables that represent the desired outputs then performs a purely discriminativefine-tuning phase using backpropagation.We refer to this architecture as“BP-DBN.”The second type used an RBM associative memory for thefinal layer to model the joint density of the labels and inputs [figure2-(b)].Forfine-tuning,derivatives of the hybrid objective function in10are followed.Only the discriminative component of the weight updates is propagated back through the earlier layers in the network during thefine-tuning stage.We refer to this architecture as“AM-DBN.”2.5Generalized softmax(GSM)output layerWhen the number of possible classes is very large and the distribution of frequencies for differ-ent classes is far from uniform,it may sometimes be advantageous to use a different encoding for4Figure2:The DBN architectures used in this work.the class targets than the standard one-of-K softmax encoding[25].It is quite straightforward to use an arbitraryfixed binary code for each class.Suppose we represent each class with its own q-dimensional code vector and thus we have q output units for our model.Let z be the q-dimensional column vector generated by the network for a certain input speech segment;z needs to be trans-formed into a vector of class posterior probabilities.If C j is the row vector holding the code for class j,then the expression for the probability the model assigns to class t given z becomesP(t|θ,z)=e C t zj e C j z(12)If we allow C j to be the j th row of the identity matrix,we recover the normal softmax expression. 3Experimental setup3.1TIMIT corpusPhone recognition experiments were performed on the TIMIT corpus1.The462speaker training set was used.All SA records(i.e.,identical sentences for all speakers in the database)were removed as they could bias the results.A development set of50speakers was used for model tuning.Results are reported using the24-speaker core test set.The speech was analyzed using a25-ms Hamming win-dow with10-ms between the left edges of successive frames.In all the experiments,we represented the speech using12th-order Mel frequency cepstral coefficients(MFCCs)and energy,along with theirfirst and second temporal derivatives.The data were normalized to have zero mean and unit variance over the entire corpus.All experiments used a context window of11frames as the visible states.We used183target class labels(i.e.,3states for each one of the61phones).After decoding, starting and ending silences were removed and the61phone classes were mapped to a set of39 classes as in[26]for scoring.All of our experiments used a bigram language model over phones, estimated from the training set.The decoder parameters were tuned to optimize performance on the development set for each run using grid search.3.2Computational setupTraining DBNs of the sizes used in this paper is quite computationally expensive.Training was accelerated by exploiting a graphics processor.A single pass over the entire training set during pretraining took about5minutes.An epoch offine-tuning with backpropagation took around13 minutes.The discriminative gradient computation for hybrid training was substantially more ex-pensive.Each epoch of hybridfine-tuning took around an hour.These time estimates represent the 1/Catalog/CatalogEntry.jsp?catalogId=LDC93S1.5largest architecture running on one of the GPUs in a NVIDIA Tesla S1070system,using the library in[27].4ExperimentsFor all experiments,the Viterbi decoder parameters(i.e.the word insertion probability,the language model scale factor)were optimized on the development set and then the best performing setting was used to compute the phone error rate(PER)for the core test set.Figure3explores the effect of varying the number of hidden layers in the model.The BP-DBN architecture is used with2048hidden units per layer.All models use the samefixed random binary code matrix to convert the128network output units into probabilities over the target183states’labels.Figure3:The effect of the model depth on PER.Adding a second layer significantly reduces the PER as shown infigure3.By adding more layers, the PER on the development set starts to plateau while the PER on the test stays between23%and 24%after adding the4th layer.This motivates the decision to restrict most of the experiments to four andfive layer models.Phone error rates(PER)for four-layer architectures with different hidden layer sizes are presented in table1.Table1:The effect of layer size on PERModel devset testset1024units21.94%23.46%2048units22.00%23.36%3072units21.74%23.54%We generally focused our computational resources on experiments with2048units per hidden layer since the performance was not significantly different for different numbers of units per layer.To check the effect of using the Generalized softmax(GSM)in the output layer of the network, we compared its performance to the standard183-way softmax output layer.Both a4hidden layer model using a128-dimensional GSM and the same architecture using a standard softmax achieved 22%PER on the development set.On the core test set,The PER of the GSM model is23.36% while the standard Softmax PER is23.9%.The128-dimensional GSM model can be viewed as a5 layer DBN with afinal layer offixed weights.To be clear on the main source of improvement,we compared the128-dimensional GSM model to a5layer DBN with afinal layer of128hidden units6。
