BIOINFORMATICS ORIGINAL PAPER
- 格式:pdf
- 大小:87.84 KB
- 文档页数:7
BIOINFORMATICSORIGINALPAPERVol.21no.152005,pages3301–3307doi:10.1093/bioinformatics/bti499
Dataandtextmining
Predictionerrorestimation:acomparisonofresampling
methods
AnnetteM.Molinaro1,3,∗,RichardSimon2andRuthM.Pfeiffer1
1BiostatisticsBranch,DivisionofCancerEpidemiologyandGeneticsand2BiometricResearchBranch,Divisionof
CancerTreatmentandDiagnostics,NCI,NIH,Rockville,MD20852USAand3DepartmentofEpidemiologyand
PublicHealth,YaleUniversitySchoolofMedicine,NewHaven,CT06520,USA
ReceivedonApril6,2005;revisedonApril28,2005;acceptedonMay12,2005
AdvanceAccesspublicationMay19,2005
ABSTRACT
Motivation:Ingenomicstudies,thousandsoffeaturesarecollected
onrelativelyfewsamples.Oneofthegoalsofthesestudiesistobuild
classifierstopredicttheoutcomeoffutureobservations.Thereare
threeinherentstepstothisprocess:featureselection,modelselection
andpredictionassessment.Withafocusonpredictionassessment,
wecompareseveralmethodsforestimatingthe‘true’predictionerror
ofapredictionmodelinthepresenceoffeatureselection.
Results:Forsmallstudieswherefeaturesareselectedfromthou-
sandsofcandidates,theresubstitutionandsimplesplit-sampleestim-
atesareseriouslybiased.Inthesesmallsamples,leave-one-out
cross-validation(LOOCV),10-foldcross-validation(CV)andthe.632+
bootstraphavethesmallestbiasfordiagonaldiscriminantanalysis,
nearestneighborandclassificationtrees.LOOCVand10-foldCVhave
thesmallestbiasforlineardiscriminantanalysis.Additionally,LOOCV,
5-and10-foldCV,andthe.632+bootstraphavethelowestmean
squareerror.The.632+bootstrapisquitebiasedinsmallsample
sizeswithstrongsignal-to-noiseratios.Differencesinperformance
amongresamplingmethodsarereducedasthenumberofspecimens
availableincrease.
Contact:annette.molinaro@yale.edu
SupplementaryInformation:Acompletecompilationofresultsand
RcodeforsimulationsandanalysesareavailableinMolinaroetal.
(2005)(http://linus.nci.nih.gov/brb/TechReport.htm).
1INTRODUCTION
Ingenomicexperimentsonefrequentlyencountershighdimensional
dataandsmallsamplesizes.Microarrayssimultaneouslymonitor
expressionlevelsforseveralthousandsofgenes.Proteomicprofil-
ingstudiesusingSELDI-TOF(surface-enhancedlaserdesorption
andionizationtime-of-flight)measuresizeandchargeofproteins
andproteinfragmentsbymassspectroscopy,andresultinupto
15000intensitylevelsatprespecifiedmassvaluesforeachspectrum.
Samplesizesinsuchexperimentsaretypically<100.
Inmanystudies,observationsareknowntobelongtopredeter-
minedclassesandthetaskistobuildpredictorsorclassifiersfor
newobservationswhoseclassisunknown.Decidingwhichgenesor
proteomicmeasurementstoincludeinthepredictioniscalledfea-
tureselectionandisacrucialstepindevelopingaclasspredictor.
∗Towhomcorrespondenceshouldbeaddressed.Includingtoomanynoisyvariablesreducesaccuracyofthepredic-
tionandmayleadtoover-fittingofdata,resultinginpromisingbut
oftennon-reproducibleresults(Ransohoff,2004).
Anotherdifficultyismodelselectionwithnumerousclassification
modelsavailable.Animportantstepinreportingresultsisassessing
thechosenmodel’serrorrate,orgeneralizability.Intheabsenceof
independentvalidationdata,acommonapproachtoestimatingpre-
dictiveaccuracyisbasedonsomeformofresamplingtheoriginal
data,e.g.cross-validation.Thesetechniquesdividethedataintoa
learningsetandatestset,andrangeincomplexityfromthepopu-
larlearning-testsplittov-foldcross-validation,Monte-Carlov-fold
cross-validationandbootstrapresampling.Fewcomparisonsof
standardresamplingmethodshavebeenperformedtodate,andallof
themexhibitlimitationsthatmaketheirconclusionsinapplicableto
mostgenomicsettings.Earlycomparisonsofresamplingtechniques
intheliteraturearefocussedonmodelselectionasopposedtopredic-
tionerrorestimation(BreimanandSpector,1992;Burman,1989).In
tworecentassessmentsofresamplingtechniquesforerrorestimation
(Braga-NetoandDougherty,2004;Efron,2004),featureselection
wasnotincludedaspartoftheresamplingprocedures,causingthe
conclusionstobeinappropriateforthehigh-dimensionalsetting.
Wehaveperformedanextensivecomparisonofresamplingmeth-
odstoestimatepredictionerrorusingsimulated(largesignal-to-noise
ratio),microarray(intermediatesignaltonoiseratio)andproteomic
data(lowsignal-to-noiseratio),encompassingincreasingsample
sizeswithlargenumbersoffeatures.Theimpactoffeatureselection
ontheperformanceofvariouscross-validationmethodsishigh-
lighted.Theresultselucidatethe‘best’resamplingtechniquesfor
futureresearchinvolvinghighdimensionaldatatoavoidoverly
optimisticassessmentoftheperformanceofamodel.
2METHODS
Inthepredictionproblem,oneobservesnindependentandidenticallydis-
tributed(i.i.d.)randomvariablesO1,...,OnwithunknowndistributionP.
EachobservationinOconsistsofanoutcomeYwithrangeYandanl-vector
ofmeasuredcovariates,orfeatures,XwithrangeX,suchthatOi=(Xi,Yi),
i=1,...,n.InmicroarrayexperimentsXincludesgeneexpressionmeas-