for the generation
- 格式:pdf
- 大小:192.89 KB
- 文档页数:3
ProceedingsoftheFourthInternationalNaturalLanguageGenerationConference,pages121–123,Sydney,July2006.c2006AssociationforComputationalLinguisticsBuildingasemanticallytransparentcorpus
forthegenerationofreferringexpressions
KeesvanDeemterandIelkavanderSluisandAlbertGatt
DepartmentofComputingScience
UniversityofAberdeen
{kvdeemte,ivdsluis,agatt}@csd.abdn.ac.uk
Abstract
Thispaperdiscussestheconstructionof
acorpusfortheevaluationofalgorithms
thatgeneratereferringexpressions.Itis
arguedthatsuchanevaluationtaskre-
quiresasemanticallytransparentcorpus,
andcontrolledexperimentsarethebest
waytocreatesucharesource.Weaddress
anumberofissuesthathaveariseninan
ongoingevaluationstudy,amongwhichis
theproblemofjudgingtheoutputofGREalgorithmsagainstahumangoldstandard.
1CreatingandusingacorpusforGRE
Adecadeago,DaleandReiter(1995)published
aseminalpaperinwhichtheycomparedanum-
berofGREalgorithms.Thesealgorithmsincluded
aFullBrevity(FB)algorithmwhichgeneratesde-
scriptionsofminimallength,agreedyalgorithm
(GA),andanIncrementalAlgorithm(IA).The
authorsarguedthatthelatterwasthebestmodel
ofhumanreferentialbehaviour,andversionsof
theIAhavesincecometorepresentthestate
oftheartinGRE.DaleandReiter’shypothe-
siswasmotivatedbypsycholinguisticfindings,
notablythatspeakerstendtoinitiatereferences
beforetheyhavecompletelyscannedadomain.
However,thisfindingaffordsdifferentalgorithmic
interpretations.Similarly,thefindingthatbasic-
leveltermsinreferringexpressionsallowhearers
toformapsychologicalgestaltcouldbeincorpo-
ratedintopracticallyanyGREalgorithm.1
WedecidedtoputDaleandReiter’shypothesis
tothetestbyanevaluationoftheoutputofdif-
1AseparateargumentforIAinvolvestractability,butal-thoughsomealternatives(suchasFB)areintractable,others(suchasGA)areonlypolynomial,andcanthereforenoteas-ilybedismissedonpurelycomputationalgrounds.ferentGREalgorithmsagainsthumanproduction.
However,itisnotoriouslydifficulttoobtainsuit-
ablecorporaforataskthatisassemanticallyin-
tensiveasContentDetermination(forGRE).Al-
thoughexistingcorporaarevaluableresources,
NLGoftenrequiresinformationthatisnotavail-
ableintext.Suppose,forexample,thatacorpus
containedarticlesaboutpolitics,howwouldthe
outputofaGREalgorithmbeevaluatedagainstthe
corpus?Itwouldbedifficulttoinferfromanar-
ticleexactlywhichrepresentativesintheBritish
HouseofCommonsareLiberalDemocrats,or
Scottish.Combiningmultipletextsishazardous,
sincefactscouldalteracrosssourcesandtime.
Moreover,theconditionsunderwhichsuchtexts
wereproduced(e.g.fault-criticalornot,asex-
plainedbelow)arehardtodetermine.
ArecentGREevaluationbyGuptaandStent
(2005)focusedondialoguecorpora,usingMAP-
TASKandCOCONUT,bothofwhichhaveanas-
sociateddomain.Theirresultsshowthatreferent
identificationinMAPTASKoftenrequiresnomore
thanaTYPEattribute,sothatnoneofthealgo-
rithmsperformedbetterthanabaseline.Incon-
trasttoMAPTASK,COCONUThasamoreelabo-
ratedomain,butitischaracterisedbyacollabora-
tivetask,andreferencesfrequentlygobeyondthe
identificationcriterionthatistypicallyinvokedin
GRE2.Mindfulofthelimitationsofexistingcor-
pora,andoftheextenttowhichevaluationde-
pendsonthecorpusunderstudy,weareusing
controlledexperimentstocreateacorpuswhose
constructionwillensurethatexistingalgorithms
canbeadequatelydifferentiatedonanidentifica-
tiontask.
2JordanandWalker(2000)havedemonstratedasignifi-cantlybettermatchtothehumandatawhentask-relatedcon-straintsaretakenintoaccount.2Setupoftheexperiment
LikeDaleandReiter(1995),wefocusedonfirst-
mentiondescriptions.However,wedecidedtoin-
cludesimple‘disjunctive’referencestosets(as
in‘theredchairandtheblacktable’),inaddi-
tiontoconjunctionsofatomicproperties,since
thesecanbehandledbyessentiallythesameal-
gorithms(vanDeemter,2002).Forgenerality,we
lookedattwoverydifferentdomains.Oneofthese
involvedartificiallyconstructedpicturesoffurni-
ture,wheretheavailableattributesandvaluesare
relativelyeasytodetermine.Theotherinvolved
realphotographsofindividuals,whichprovidea
richerrangeofoptionstosubjects.Todate,data
hasbeencollectedfrom19participants,andanal-
ysisisinprogress.
Ourfirstchallengewastomaketheexperiment
naturalistic.Subjectswereshown38randomised
trials,eachdepictingasetofobjects,oneortwo
ofwhichwerethetargets,surroundedby6dis-
tractors(Figure1).Ineachcase,aminimaldistin-
guishingdescriptionofthetargetswasavailable.
Subjectswereledtobelievethattheywouldbe
describingthetargetsforaninterlocutor.Oncea
descriptionwastyped,thesystemremovedfrom
thescreenwhatittooktobethereferents.
Figure1:Astimulusexamplefromthefurnituredomain.
Threegroupsperformedthetaskindifferent
conditions,namely:±FaultCritical,where
halfthesubjectsinthe+FaultCriticalcase
coulduselocation(‘inthetopleftcorner’).The
+FaultCriticalgroupwastold:‘Ourprogram
willeventuallybeusedinsituationswhereitis