BIOINFORMATICS ORIGINAL PAPER

  • 格式:pdf
  • 大小:87.84 KB
  • 文档页数:7

BIOINFORMATICSORIGINALPAPERVol.21no.152005,pages3301–3307doi:10.1093/bioinformatics/bti499

Dataandtextmining

Predictionerrorestimation:acomparisonofresampling

methods

AnnetteM.Molinaro1,3,∗,RichardSimon2andRuthM.Pfeiffer1

1BiostatisticsBranch,DivisionofCancerEpidemiologyandGeneticsand2BiometricResearchBranch,Divisionof

CancerTreatmentandDiagnostics,NCI,NIH,Rockville,MD20852USAand3DepartmentofEpidemiologyand

PublicHealth,YaleUniversitySchoolofMedicine,NewHaven,CT06520,USA

ReceivedonApril6,2005;revisedonApril28,2005;acceptedonMay12,2005

AdvanceAccesspublicationMay19,2005

ABSTRACT

Motivation:Ingenomicstudies,thousandsoffeaturesarecollected

onrelativelyfewsamples.Oneofthegoalsofthesestudiesistobuild

classifierstopredicttheoutcomeoffutureobservations.Thereare

threeinherentstepstothisprocess:featureselection,modelselection

andpredictionassessment.Withafocusonpredictionassessment,

wecompareseveralmethodsforestimatingthe‘true’predictionerror

ofapredictionmodelinthepresenceoffeatureselection.

Results:Forsmallstudieswherefeaturesareselectedfromthou-

sandsofcandidates,theresubstitutionandsimplesplit-sampleestim-

atesareseriouslybiased.Inthesesmallsamples,leave-one-out

cross-validation(LOOCV),10-foldcross-validation(CV)andthe.632+

bootstraphavethesmallestbiasfordiagonaldiscriminantanalysis,

nearestneighborandclassificationtrees.LOOCVand10-foldCVhave

thesmallestbiasforlineardiscriminantanalysis.Additionally,LOOCV,

5-and10-foldCV,andthe.632+bootstraphavethelowestmean

squareerror.The.632+bootstrapisquitebiasedinsmallsample

sizeswithstrongsignal-to-noiseratios.Differencesinperformance

amongresamplingmethodsarereducedasthenumberofspecimens

availableincrease.

Contact:annette.molinaro@yale.edu

SupplementaryInformation:Acompletecompilationofresultsand

RcodeforsimulationsandanalysesareavailableinMolinaroetal.

(2005)(http://linus.nci.nih.gov/brb/TechReport.htm).

1INTRODUCTION

Ingenomicexperimentsonefrequentlyencountershighdimensional

dataandsmallsamplesizes.Microarrayssimultaneouslymonitor

expressionlevelsforseveralthousandsofgenes.Proteomicprofil-

ingstudiesusingSELDI-TOF(surface-enhancedlaserdesorption

andionizationtime-of-flight)measuresizeandchargeofproteins

andproteinfragmentsbymassspectroscopy,andresultinupto

15000intensitylevelsatprespecifiedmassvaluesforeachspectrum.

Samplesizesinsuchexperimentsaretypically<100.

Inmanystudies,observationsareknowntobelongtopredeter-

minedclassesandthetaskistobuildpredictorsorclassifiersfor

newobservationswhoseclassisunknown.Decidingwhichgenesor

proteomicmeasurementstoincludeinthepredictioniscalledfea-

tureselectionandisacrucialstepindevelopingaclasspredictor.

∗Towhomcorrespondenceshouldbeaddressed.Includingtoomanynoisyvariablesreducesaccuracyofthepredic-

tionandmayleadtoover-fittingofdata,resultinginpromisingbut

oftennon-reproducibleresults(Ransohoff,2004).

Anotherdifficultyismodelselectionwithnumerousclassification

modelsavailable.Animportantstepinreportingresultsisassessing

thechosenmodel’serrorrate,orgeneralizability.Intheabsenceof

independentvalidationdata,acommonapproachtoestimatingpre-

dictiveaccuracyisbasedonsomeformofresamplingtheoriginal

data,e.g.cross-validation.Thesetechniquesdividethedataintoa

learningsetandatestset,andrangeincomplexityfromthepopu-

larlearning-testsplittov-foldcross-validation,Monte-Carlov-fold

cross-validationandbootstrapresampling.Fewcomparisonsof

standardresamplingmethodshavebeenperformedtodate,andallof

themexhibitlimitationsthatmaketheirconclusionsinapplicableto

mostgenomicsettings.Earlycomparisonsofresamplingtechniques

intheliteraturearefocussedonmodelselectionasopposedtopredic-

tionerrorestimation(BreimanandSpector,1992;Burman,1989).In

tworecentassessmentsofresamplingtechniquesforerrorestimation

(Braga-NetoandDougherty,2004;Efron,2004),featureselection

wasnotincludedaspartoftheresamplingprocedures,causingthe

conclusionstobeinappropriateforthehigh-dimensionalsetting.

Wehaveperformedanextensivecomparisonofresamplingmeth-

odstoestimatepredictionerrorusingsimulated(largesignal-to-noise

ratio),microarray(intermediatesignaltonoiseratio)andproteomic

data(lowsignal-to-noiseratio),encompassingincreasingsample

sizeswithlargenumbersoffeatures.Theimpactoffeatureselection

ontheperformanceofvariouscross-validationmethodsishigh-

lighted.Theresultselucidatethe‘best’resamplingtechniquesfor

futureresearchinvolvinghighdimensionaldatatoavoidoverly

optimisticassessmentoftheperformanceofamodel.

2METHODS

Inthepredictionproblem,oneobservesnindependentandidenticallydis-

tributed(i.i.d.)randomvariablesO1,...,OnwithunknowndistributionP.

EachobservationinOconsistsofanoutcomeYwithrangeYandanl-vector

ofmeasuredcovariates,orfeatures,XwithrangeX,suchthatOi=(Xi,Yi),

i=1,...,n.InmicroarrayexperimentsXincludesgeneexpressionmeas-