for the generation

  • 格式:pdf
  • 大小:192.89 KB
  • 文档页数:3

ProceedingsoftheFourthInternationalNaturalLanguageGenerationConference,pages121–123,Sydney,July2006.c󰀁2006AssociationforComputationalLinguisticsBuildingasemanticallytransparentcorpus

forthegenerationofreferringexpressions

KeesvanDeemterandIelkavanderSluisandAlbertGatt

DepartmentofComputingScience

UniversityofAberdeen

{kvdeemte,ivdsluis,agatt}@csd.abdn.ac.uk

Abstract

Thispaperdiscussestheconstructionof

acorpusfortheevaluationofalgorithms

thatgeneratereferringexpressions.Itis

arguedthatsuchanevaluationtaskre-

quiresasemanticallytransparentcorpus,

andcontrolledexperimentsarethebest

waytocreatesucharesource.Weaddress

anumberofissuesthathaveariseninan

ongoingevaluationstudy,amongwhichis

theproblemofjudgingtheoutputofGREalgorithmsagainstahumangoldstandard.

1CreatingandusingacorpusforGRE

Adecadeago,DaleandReiter(1995)published

aseminalpaperinwhichtheycomparedanum-

berofGREalgorithms.Thesealgorithmsincluded

aFullBrevity(FB)algorithmwhichgeneratesde-

scriptionsofminimallength,agreedyalgorithm

(GA),andanIncrementalAlgorithm(IA).The

authorsarguedthatthelatterwasthebestmodel

ofhumanreferentialbehaviour,andversionsof

theIAhavesincecometorepresentthestate

oftheartinGRE.DaleandReiter’shypothe-

siswasmotivatedbypsycholinguisticfindings,

notablythatspeakerstendtoinitiatereferences

beforetheyhavecompletelyscannedadomain.

However,thisfindingaffordsdifferentalgorithmic

interpretations.Similarly,thefindingthatbasic-

leveltermsinreferringexpressionsallowhearers

toformapsychologicalgestaltcouldbeincorpo-

ratedintopracticallyanyGREalgorithm.1

WedecidedtoputDaleandReiter’shypothesis

tothetestbyanevaluationoftheoutputofdif-

1AseparateargumentforIAinvolvestractability,butal-thoughsomealternatives(suchasFB)areintractable,others(suchasGA)areonlypolynomial,andcanthereforenoteas-ilybedismissedonpurelycomputationalgrounds.ferentGREalgorithmsagainsthumanproduction.

However,itisnotoriouslydifficulttoobtainsuit-

ablecorporaforataskthatisassemanticallyin-

tensiveasContentDetermination(forGRE).Al-

thoughexistingcorporaarevaluableresources,

NLGoftenrequiresinformationthatisnotavail-

ableintext.Suppose,forexample,thatacorpus

containedarticlesaboutpolitics,howwouldthe

outputofaGREalgorithmbeevaluatedagainstthe

corpus?Itwouldbedifficulttoinferfromanar-

ticleexactlywhichrepresentativesintheBritish

HouseofCommonsareLiberalDemocrats,or

Scottish.Combiningmultipletextsishazardous,

sincefactscouldalteracrosssourcesandtime.

Moreover,theconditionsunderwhichsuchtexts

wereproduced(e.g.fault-criticalornot,asex-

plainedbelow)arehardtodetermine.

ArecentGREevaluationbyGuptaandStent

(2005)focusedondialoguecorpora,usingMAP-

TASKandCOCONUT,bothofwhichhaveanas-

sociateddomain.Theirresultsshowthatreferent

identificationinMAPTASKoftenrequiresnomore

thanaTYPEattribute,sothatnoneofthealgo-

rithmsperformedbetterthanabaseline.Incon-

trasttoMAPTASK,COCONUThasamoreelabo-

ratedomain,butitischaracterisedbyacollabora-

tivetask,andreferencesfrequentlygobeyondthe

identificationcriterionthatistypicallyinvokedin

GRE2.Mindfulofthelimitationsofexistingcor-

pora,andoftheextenttowhichevaluationde-

pendsonthecorpusunderstudy,weareusing

controlledexperimentstocreateacorpuswhose

constructionwillensurethatexistingalgorithms

canbeadequatelydifferentiatedonanidentifica-

tiontask.

2JordanandWalker(2000)havedemonstratedasignifi-cantlybettermatchtothehumandatawhentask-relatedcon-straintsaretakenintoaccount.2Setupoftheexperiment

LikeDaleandReiter(1995),wefocusedonfirst-

mentiondescriptions.However,wedecidedtoin-

cludesimple‘disjunctive’referencestosets(as

in‘theredchairandtheblacktable’),inaddi-

tiontoconjunctionsofatomicproperties,since

thesecanbehandledbyessentiallythesameal-

gorithms(vanDeemter,2002).Forgenerality,we

lookedattwoverydifferentdomains.Oneofthese

involvedartificiallyconstructedpicturesoffurni-

ture,wheretheavailableattributesandvaluesare

relativelyeasytodetermine.Theotherinvolved

realphotographsofindividuals,whichprovidea

richerrangeofoptionstosubjects.Todate,data

hasbeencollectedfrom19participants,andanal-

ysisisinprogress.

Ourfirstchallengewastomaketheexperiment

naturalistic.Subjectswereshown38randomised

trials,eachdepictingasetofobjects,oneortwo

ofwhichwerethetargets,surroundedby6dis-

tractors(Figure1).Ineachcase,aminimaldistin-

guishingdescriptionofthetargetswasavailable.

Subjectswereledtobelievethattheywouldbe

describingthetargetsforaninterlocutor.Oncea

descriptionwastyped,thesystemremovedfrom

thescreenwhatittooktobethereferents.

Figure1:Astimulusexamplefromthefurnituredomain.

Threegroupsperformedthetaskindifferent

conditions,namely:󰀄±FaultCritical󰀅,where

halfthesubjectsinthe󰀄+FaultCritical󰀅case

coulduselocation(‘inthetopleftcorner’).The

󰀄+FaultCritical󰀅groupwastold:‘Ourprogram

willeventuallybeusedinsituationswhereitis