Automatic Data Extraction from Data-rich Web Pages

  • 格式:pdf
  • 大小:80.86 KB
  • 文档页数:12

AutomaticDataExtractionfromData-richWebPages

DongdongHuandXiaofengMeng

SchoolofInformationRenminUniversityofChina{hudd,xfmeng}@ruc.edu.cn

Abstract.Extractingdatafromwebpagesusingwrappersisafundamentalprob-lemarisinginalargevarietyofapplicationsofvastpracticalinterests.Inthispaper,weproposeanoveltechniquetotheproblemofdifferentiatingrolesofdataitemsfromWebpages,whichisoneofthekeyproblemsinourautomaticextractionapproach.Theproblemisresolvedatvariouslevels:semanticblocks,sectionsanddataitems,andseveralapproachesareproposedtoeffectivelyiden-tifythemappingbetweendataitemshavingthesamerole.Intensiveexperimentsonrealwebsitesshowthattheproposedtechniquecaneffectivelyhelpextractingdesireddatawithhighaccuraciesinmostofthecases.

1Introduction

TheWorldWideWebhasbecomeoneofthemostimportantconnectionsofvarious

informationsources.Alargeproportionofdataonthewebisembeddedinvarious

HTMLdocuments.TheHTMLlanguageservesthevisualpresentationofdatainWeb

browsers,whileitisnotsuitableforautomated,computer-assistedinformationman-

agementsystems.Thisisnotexpectedtochangesoon,evenwhenXMLismoreand

morepopulartoday.Thusifdatafromdifferentsourcesneedstobeintegrated,itisnec-

essarytodevelopspecialandoftencomplexprogramstoextractdatafromWebpages.

Toachievethisgoal,peoplehavedevelopedwrappers,whicharespecializedprograms

thatcanautomaticallyextractdatafromwebpagesandconverttheinformationintoa

structuredformat.

Therehavebeenmanyworksonsemi-automaticandmanualdataextraction[1][2][3]

[4][5],inthepastyears.Theseapproachesrequirehumaninteractionstobuildsample

mappingsbetweentheoutputresults(orschemas)anditemsintheHTMLpages,after

thatextractionruleswillbeinducedforextractingpageshavingsimilarstructures.Be-

sidestheflexibilityoftheseapproaches,therearestillmanychallengesinconstructing

suchwrappers.Firstly,theusershouldhavegoodknowledgeofcontentsintheweb

pagesandshouldtrytoselectmorepropersamplestocovermorepossiblesituations;

also,anotherimportantproblemishowtomaintainexistingwrappersifthecorrespond-

ingwebpagestakechangesontheirlayouts.

Therestillexistsseveralotherautomaticapproaches,Exalg[6]andRoadrunner

[7],whichcanautomaticallyextractdatafromdata-intensive[7]websites.Pages

fromdata-intensivesitesarecreatedthroughencodingvalues,fromtheunderground

database,intowebpagesusingtemplates[6].Inotherwords,suchpagesusuallyhave

thesameschemaandaccordinglysimilarstructures.EitherRoadrunnerorExalgusethestructureinformationofsamplepagestoinduceHTMLtags-basedtemplates(see

Section5).AtemplatecoverstheconstantpartsoftheHTMLsources’stringsequence,

andtheleftpartswhicharevariantareviewedastherightdatatobeextracted.

(a)Sample1

(b)Sample2

(c)Sample3

Fig.1.SampleHTMLfragmentsfromBuy

Unlikethepreviousworkdescribedabove,weproposeanovelautomaticapproach

togeneratewrapperswithoutanyuserinteractions.Firstlywebreakuptheproblemof

automaticallyconstructingawrapperintofoursubproblems:

1.Discoveringblocks(calledsemanticblockinthispaper,whichcorrespondstoan

instanceoftheschema)containingdatatobeextractedfromsamplepages;

2.Differentiatingrolesofdataitemsinthesemanticblocks;

3.Inducingschemadescribingthecontentsinthepages;

4.Computingextractionruleandgeneratingwrappers.

Amongthesefoursteps,asimplewaytotackleproblem(1)istouseatechniqueof

treecomparingfordiscoveringdata-richsections[8].Forexample,inFig.1arethree

extractedblocksfromsamplepagesfromwww.buy.com.Problem(3)isintroduced

in[9],andproblem(4)iswidelydiscussedinalmostallthepreviousworkofdata

extraction.Inthepreviouswork[3]weproposedanXQuery[10]basedextraction

rulesystem.Ourworkfocusesonproblem(2),whichservesasthekeystepforthe

extractionproblem.Oncewehavegotasetofsemanticblocks,whichcanbeviewedas

asetofinstancesoftheschema,whichdescribesthecontentsinthepage,tobeinduced

inthenextstep.Toinducetheschemafromtheinstances,wefirstlydifferentiatethe

rolesofdataitemsinthem.Inotherwords,foradataiteminasematicblock,we

wanttofindoutthedataitemstakingthesamerole,correspondingtothesameschema

element,inallothersemanticblocks.Forexample,referringtoFig.1,theitemsof

“HeadFirstJava”and“ThinkinginJavawithCDROM”arebothcorrespondedtothe

schemaelementTitle.TheprocessofdifferentiatingrolesarenamedIdentificationin

thispaper.Afterallthedataitemsareidentified,eachoneisassignedalabelindicatingitsrole.Dataitemshavingthesameroleshavethesamelabels.Notethatthelabelhere

doesnotprovideanysemanticinformation,butasymbolrepresentingtheroleofadata

item.

Thispaperisstructuredasfollows.Inthenextsectionthechallengesofwefaces

andsomepreliminaryknowledgeisdescribed,insection3weintroducethetechnique

ofsimpleidentification.Section4discussesthecomplexidentification,andsection5

comparesourworktotherelatedwork.Insection6wereporttheexperimentalresults.

Finallyinsection7wehighlighttheconclusionandfutureresearchdirection.