Automatic Data Extraction from Data-rich Web Pages
- 格式:pdf
- 大小:80.86 KB
- 文档页数:12
AutomaticDataExtractionfromData-richWebPages
DongdongHuandXiaofengMeng
SchoolofInformationRenminUniversityofChina{hudd,xfmeng}@ruc.edu.cn
Abstract.Extractingdatafromwebpagesusingwrappersisafundamentalprob-lemarisinginalargevarietyofapplicationsofvastpracticalinterests.Inthispaper,weproposeanoveltechniquetotheproblemofdifferentiatingrolesofdataitemsfromWebpages,whichisoneofthekeyproblemsinourautomaticextractionapproach.Theproblemisresolvedatvariouslevels:semanticblocks,sectionsanddataitems,andseveralapproachesareproposedtoeffectivelyiden-tifythemappingbetweendataitemshavingthesamerole.Intensiveexperimentsonrealwebsitesshowthattheproposedtechniquecaneffectivelyhelpextractingdesireddatawithhighaccuraciesinmostofthecases.
1Introduction
TheWorldWideWebhasbecomeoneofthemostimportantconnectionsofvarious
informationsources.Alargeproportionofdataonthewebisembeddedinvarious
HTMLdocuments.TheHTMLlanguageservesthevisualpresentationofdatainWeb
browsers,whileitisnotsuitableforautomated,computer-assistedinformationman-
agementsystems.Thisisnotexpectedtochangesoon,evenwhenXMLismoreand
morepopulartoday.Thusifdatafromdifferentsourcesneedstobeintegrated,itisnec-
essarytodevelopspecialandoftencomplexprogramstoextractdatafromWebpages.
Toachievethisgoal,peoplehavedevelopedwrappers,whicharespecializedprograms
thatcanautomaticallyextractdatafromwebpagesandconverttheinformationintoa
structuredformat.
Therehavebeenmanyworksonsemi-automaticandmanualdataextraction[1][2][3]
[4][5],inthepastyears.Theseapproachesrequirehumaninteractionstobuildsample
mappingsbetweentheoutputresults(orschemas)anditemsintheHTMLpages,after
thatextractionruleswillbeinducedforextractingpageshavingsimilarstructures.Be-
sidestheflexibilityoftheseapproaches,therearestillmanychallengesinconstructing
suchwrappers.Firstly,theusershouldhavegoodknowledgeofcontentsintheweb
pagesandshouldtrytoselectmorepropersamplestocovermorepossiblesituations;
also,anotherimportantproblemishowtomaintainexistingwrappersifthecorrespond-
ingwebpagestakechangesontheirlayouts.
Therestillexistsseveralotherautomaticapproaches,Exalg[6]andRoadrunner
[7],whichcanautomaticallyextractdatafromdata-intensive[7]websites.Pages
fromdata-intensivesitesarecreatedthroughencodingvalues,fromtheunderground
database,intowebpagesusingtemplates[6].Inotherwords,suchpagesusuallyhave
thesameschemaandaccordinglysimilarstructures.EitherRoadrunnerorExalgusethestructureinformationofsamplepagestoinduceHTMLtags-basedtemplates(see
Section5).AtemplatecoverstheconstantpartsoftheHTMLsources’stringsequence,
andtheleftpartswhicharevariantareviewedastherightdatatobeextracted.
(a)Sample1
(b)Sample2
(c)Sample3
Fig.1.SampleHTMLfragmentsfromBuy
Unlikethepreviousworkdescribedabove,weproposeanovelautomaticapproach
togeneratewrapperswithoutanyuserinteractions.Firstlywebreakuptheproblemof
automaticallyconstructingawrapperintofoursubproblems:
1.Discoveringblocks(calledsemanticblockinthispaper,whichcorrespondstoan
instanceoftheschema)containingdatatobeextractedfromsamplepages;
2.Differentiatingrolesofdataitemsinthesemanticblocks;
3.Inducingschemadescribingthecontentsinthepages;
4.Computingextractionruleandgeneratingwrappers.
Amongthesefoursteps,asimplewaytotackleproblem(1)istouseatechniqueof
treecomparingfordiscoveringdata-richsections[8].Forexample,inFig.1arethree
extractedblocksfromsamplepagesfromwww.buy.com.Problem(3)isintroduced
in[9],andproblem(4)iswidelydiscussedinalmostallthepreviousworkofdata
extraction.Inthepreviouswork[3]weproposedanXQuery[10]basedextraction
rulesystem.Ourworkfocusesonproblem(2),whichservesasthekeystepforthe
extractionproblem.Oncewehavegotasetofsemanticblocks,whichcanbeviewedas
asetofinstancesoftheschema,whichdescribesthecontentsinthepage,tobeinduced
inthenextstep.Toinducetheschemafromtheinstances,wefirstlydifferentiatethe
rolesofdataitemsinthem.Inotherwords,foradataiteminasematicblock,we
wanttofindoutthedataitemstakingthesamerole,correspondingtothesameschema
element,inallothersemanticblocks.Forexample,referringtoFig.1,theitemsof
“HeadFirstJava”and“ThinkinginJavawithCDROM”arebothcorrespondedtothe
schemaelementTitle.TheprocessofdifferentiatingrolesarenamedIdentificationin
thispaper.Afterallthedataitemsareidentified,eachoneisassignedalabelindicatingitsrole.Dataitemshavingthesameroleshavethesamelabels.Notethatthelabelhere
doesnotprovideanysemanticinformation,butasymbolrepresentingtheroleofadata
item.
Thispaperisstructuredasfollows.Inthenextsectionthechallengesofwefaces
andsomepreliminaryknowledgeisdescribed,insection3weintroducethetechnique
ofsimpleidentification.Section4discussesthecomplexidentification,andsection5
comparesourworktotherelatedwork.Insection6wereporttheexperimentalresults.
Finallyinsection7wehighlighttheconclusionandfutureresearchdirection.