North Carolina State University Scientific Data Management Center for
Enabling Technologies
Project Report (DE-FC02-07) ER25809
Period: 10/1/2006 – 11/14/2012
11/15/2006-11/14/2012
NC State PI:
Mladen A. Vouk
Computer Science, Box 8206 North Carolina State University Raleigh, NC 27695
vouk@https://www.doczj.com/doc/3c7504952.html,
919-515-7886
-XO\ -XQH
DQG 1&68 H[W -XO 1RY
KWWS VGPFHQWHU OEO JRY
7DEOH RI &RQWHQWV
6'0 FHQWHU PLVVLRQ DQG JRDOV??? ???????????????? 2UJDQL]DWLRQ RI WKH UHSRUW ?????????????????????? 6HOHFWHG +LJKOLJKWV RI $FKLHYHPHQWV ????????????????? 'HWDLOHG VHFWLRQV ????????????????????????? 6WRUDJH (IILFLHQW $FFHVV 6($ ?????????????????? )LOH 6\VWHP %HQFKPDUNLQJ DQG $SSOLFDWLRQ , 2 %HKDYLRU ?????
3DUDOOHO , 2 ,QIUDVWUXFWXUH (YROXWLRQ ?????????????
$SSOLFDWLRQ ,QWHUIDFHV DQG 'DWD 0RGHOV ????????????
7KH $GDSWDEOH , 2 6\VWHP $',26 ???????????? 1H[W *HQHUDWLRQ , 2 6RIWZDUH 7HFKQRORJLHV ?????????? 2XWUHDFK ???????????????????????? 6FLHQWLILF 'DWD 0LQLQJ DQG $QDO\WLFV '0$ ???????????? +LJK SHUIRUPDQFH SDUDOOHO VWDWLVWLFDO FRPSXWLQJ ????????? (IILFLHQW VHDUFKLQJ DQG ILOWHULQJ LQ GDWD LQWHQVLYH DSSOLFDWLRQV ??? )HDWXUH H[WUDFWLRQ DQG WUDFNLQJ IRU VFLHQWLILF DSSOLFDWLRQV????? 7UDLQLQJ DQG 2XWUHDFK ??????????????????? 6FLHQWLILF 3URFHVV $XWRPDWLRQ 63$ ??????????????? :RUNIORZ GHYHORSPHQW ?????????????????? *HQHULF ZRUNIORZ FRPSRQHQWV DQG WHPSODWHV ?????????? 'DVKERDUG GHYHORSPHQW ?????????????????? 3URYHQDQFH FROOHFWLRQ DQG DQDO\VLV ?????????????? :RUNIORZ UHOLDELOLW\ DQG IDXOW WROHUDQFH ???????????? )UDPHZRUN IRU ,QWHJUDWHG (QG WR HQG 6'0 7HFKQRORJLHV
DQG $SSOLFDWLRQV ),(67$ ???????????????? 3XEOLFDWLRQV DQG UHIHUHQFHV ????????????????????? $SSHQGLFHV ???????????????????????????? $SSHQGL[ 7XWRULDOV WUDLQLQJ WKHVLV RXWUHDFK LQYLWHG SUHVHQWDWLRQV ??? $SSHQGL[ 7DEOH RI &ROODERUDWLRQ ZLWK $SSOLFDWLRQ 3URMHFWV
DQG RWKHU &HQWHUV DQG ,QVWLWXWHV ??????????????? $SSHQGL[ 'HWDLOV RI DSSOLFDWLRQV YV WHFKQRORJLHV LQ DSSHQGL[ WDEOH ?? $SSHQGL[ 1&68 H[WHQVLRQ UHVXOWV
2FWREHU -XQH
KWWS VGPFHQWHU OEO JRY
SDM center Mission and goals
0DQDJLQJ VFLHQWLILF GDWD KDV EHHQ LGHQWLILHG E\ WKH VFLHQWLILF FRPPXQLW\ DV RQH RI WKH PRVW LPSRUWDQW HPHUJLQJ QHHGV EHFDXVH RI WKH VKHHU YROXPH DQG LQFUHDVLQJ FRPSOH[LW\ RI GDWD EHLQJ FROOHFWHG (IIHFWLYHO\ JHQHUDWLQJ PDQDJLQJ DQG DQDO\]LQJ WKLV LQIRUPDWLRQ UHTXLUHV D FRPSUHKHQVLYH HQG WR HQG DSSURDFK WR GDWD PDQDJHPHQW WKDW HQFRPSDVVHV DOO RI WKH VWDJHV IURP WKH LQLWLDO GDWD DFTXLVLWLRQ WR WKH ILQDO DQDO\VLV RI WKH GDWD )RUWXQDWHO\ WKH GDWD PDQDJHPHQW SUREOHPV HQFRXQWHUHG E\ PRVW VFLHQWLILF GRPDLQV DUH FRPPRQ HQRXJK WR EH DGGUHVVHG WKURXJK VKDUHG WHFKQRORJ\ VROXWLRQV %DVHG RQ FRPPXQLW\ LQSXW ZH KDYH LGHQWLILHG WKUHH VLJQLILFDQW UHTXLUHPHQWV )LUVW PRUH HIILFLHQW DFFHVV WR VWRUDJH V\VWHPV LV QHHGHG ,Q SDUWLFXODU SDUDOOHO ILOH V\VWHP DQG , 2 V\VWHP LPSURYHPHQWV DUH QHHGHG WR ZULWH DQG UHDG ODUJH YROXPHV RI GDWD ZLWKRXW VORZLQJ D VLPXODWLRQ DQDO\VLV RU YLVXDOL]DWLRQ HQJLQH 7KHVH SURFHVVHV DUH FRPSOLFDWHG E\ WKH IDFW WKDW VFLHQWLILF GDWD DUH VWUXFWXUHG GLIIHUHQWO\ IRU VSHFLILF DSSOLFDWLRQ GRPDLQV DQG DUH VWRUHG LQ VSHFLDOL]HG ILOH IRUPDWV 6HFRQG VFLHQWLVWV UHTXLUH WHFKQRORJLHV WR IDFLOLWDWH EHWWHU XQGHUVWDQGLQJ RI WKHLU GDWD LQ SDUWLFXODU WKH DELOLW\ WR HIIHFWLYHO\ SHUIRUP FRPSOH[ GDWD DQDO\VLV DQG VHDUFKHV RYHU H[WUHPHO\ ODUJH GDWD VHWV 6SHFLDOL]HG IHDWXUH GLVFRYHU\ DQG VWDWLVWLFDO DQDO\VLV WHFKQLTXHV DUH QHHGHG EHIRUH WKH GDWD FDQ EH XQGHUVWRRG RU YLVXDOL]HG )XUWKHUPRUH LQWHUDFWLYH DQDO\VLV UHTXLUHV WHFKQLTXHV IRU HIILFLHQWO\ VHOHFWLQJ VXEVHWV RI WKH GDWD )LQDOO\ JHQHUDWLQJ WKH GDWD FROOHFWLQJ DQG VWRULQJ WKH UHVXOWV NHHSLQJ WUDFN RI GDWD SURYHQDQFH GDWD SRVW SURFHVVLQJ DQG DQDO\VLV RI UHVXOWV LV D WHGLRXV IUDJPHQWHG SURFHVV 7RROV IRU DXWRPDWLRQ RI WKLV SURFHVV LQ D UREXVW WUDFWDEOH DQG UHFRYHUDEOH IDVKLRQ DUH UHTXLUHG WR HQKDQFH VFLHQWLILF H[SORUDWLRQ 7KH JRDOV RI WKH FHQWHU DUH VKRZQ VFKHPDWLFDOO\ LQ WKH )LJXUH EHORZ
7KH 6'0 FHQWHU ZDV HVWDEOLVKHG XQGHU WKH 6FL'$& SURJUDP WR DGGUHVV WKHVH LVVXHV 7KH 6FL'$& 6FLHQWLILF 'DWD 0DQDJHPHQW 6'0 &HQWHU VXFFHHGHG LQ EULQJLQJ DQ LQLWLDO VHW RI DGYDQFHG GDWD PDQDJHPHQW WHFKQRORJLHV WR '2( DSSOLFDWLRQ VFLHQWLVWV LQ DVWURSK\VLFV FOLPDWH IXVLRQ DQG ELRORJ\ (TXDOO\ LPSRUWDQW LW HVWDEOLVKHG FROODERUDWLRQV ZLWK WKHVH VFLHQWLVWV WR EHWWHU XQGHUVWDQG WKHLU VFLHQFH DV ZHOO DV WKHLU IRUWKFRPLQJ GDWD PDQDJHPHQW DQG GDWD DQDO\WLFV FKDOOHQJHV %XLOGLQJ RQ RXU HDUO\ VXFFHVVHV ZH KDYH JUHDWO\ HQKDQFHG UREXVWLILHG DQG GHSOR\HG RXU WHFKQRORJ\ WR WKHVH FRPPXQLWLHV ,Q VRPH FDVHV ZH LGHQWLILHG QHZ QHHGV WKDW KDYH EHHQ DGGUHVVHG LQ RUGHU WR VLPSOLI\ WKH XVH RI RXU WHFKQRORJ\ E\ VFLHQWLVWV 7KLV UHSRUW VXPPDUL]HV RXU ZRUN VR IDU LQ 6FL'$&
2XU DSSURDFK LV WR HPSOR\ DQ HYROXWLRQDU\ GHYHORSPHQW DQG GHSOR\PHQW SURFHVV IURP UHVHDUFK WKURXJK SURWRW\SHV WR GHSOR\PHQW DQG LQIUDVWUXFWXUH $FFRUGLQJO\ ZH KDYH RUJDQL]HG RXU DFWLYLWLHV LQ WKUHH OD\HUV WKDW DEVWUDFW WKH HQG WR HQG GDWD IORZ GHVFULEHG DERYH :H ODEHOHG WKH OD\HUV IURP ERWWRP WR WRS x6WRUDJH (IILFLHQW $FFHVV 6($
x'DWD 0LQLQJ DQG $QDO\VLV '0$
x6FLHQWLILF 3URFHVV $XWRPDWLRQ 63$
7KH 6($ OD\HU LV LPPHGLDWHO\ RQ WRS RI KDUGZDUH RSHUDWLQJ V\VWHPV ILOH V\VWHPV DQG PDVV VWRUDJH V\VWHPV DQG SURYLGHV SDUDOOHO GDWD DFFHVV WHFKQRORJ\ DQG WUDQVSDUHQW DFFHVV WR DUFKLYDO VWRUDJH 7KH '0$ OD\HU ZKLFK EXLOGV RQ WKH IXQFWLRQDOLW\ RI WKH 6($ OD\HU FRQVLVWV RI LQGH[LQJ IHDWXUH LGHQWLILFDWLRQ DQG SDUDOOHO VWDWLVWLFDO DQDO\VLV WHFKQRORJ\ 7KH 63$ OD\HU ZKLFK LV RQ WRS RI WKH '0$ OD\HU SURYLGHV WKH DELOLW\ WR FRPSRVH VFLHQWLILF ZRUNIORZV IURP WKH FRPSRQHQWV LQ WKH '0$ OD\HU DV ZHOO DV DSSOLFDWLRQ VSHFLILF PRGXOHV 7KH RUJDQL]DWLRQ RI WKH FHQWHU DQG WHFKQRORJLHV LQFOXGHG LQ HDFK OD\HU DUH VKRZQ EHORZ
Organization of the report
7KLV UHSRUW FRQVLVWV RI WKH IROORZLQJ VHFWLRQV RUJDQL]HG DFFRUGLQJ WR WKH WKUHH OD\HUV DV IROORZV
7KH 6WRUDJH (IILFLHQW $FFHVV 6($ DUHD LQFOXGHV WKH IROORZLQJ DFWLYLWLHV ILOH V\VWHP EHQFKPDUNLQJ DQG DSSOLFDWLRQ , 2 EHKDYLRU SDUDOOHO , 2 LQIUDVWUXFWXUH HYROXWLRQ DSSOLFDWLRQ LQWHUIDFHV DQG GDWD PRGHOV DQG QH[W JHQHUDWLRQ , 2 VRIWZDUH WHFKQRORJLHV
7KH 'DWD 0LQLQJ DQG $QDO\VLV '0$ DUHD LQFOXGHV WKH IROORZLQJ DFWLYLWLHV KLJK SHUIRUPDQFH VWDWLVWLFDO FRPSXWLQJ HIILFLHQW VHDUFKLQJ DQG ILOWHULQJ LQ GDWD LQWHQVLYH VFLHQWLILF DSSOLFDWLRQV DQG )HDWXUH H[WUDFWLRQ DQG WUDFNLQJ LQ VFLHQWLILF DSSOLFDWLRQV
7KH 6FLHQWLILF 3URFHVV $XWRPDWLRQ 63$ DUHD LQFOXGHV WKH IROORZLQJ DFWLYLWLHV ZRUNIORZ GHYHORSPHQW SURYHQDQFH FROOHFWLRQ JHQHULF DFWRUV DQG ZRUNIORZ IDXOW WROHUDQFH
,Q DGGLWLRQ WR WKH WKUHH VHFWLRQV FRYHULQJ SURJUHVV LQ WKH WKUHH IRFXV DUHDV DERYH ZH LQFOXGH DQ DGGLWLRQDO VHFWLRQ WKDW GHVFULEHV RXU HIIRUWV LQ SURYLGLQJ DQ )UDPHZRUN IRU ,QWHJUDWHG (QG WR HQG 6'0 7HFKQRORJLHV DQG $SSOLFDWLRQV ),(67$ WKDW XVHV PXOWLSOH 6'0 FHQWHU WHFKQRORJLHV IRU D VSHFLILF 6FL'$& )XVLRQ DSSOLFDWLRQ SURMHFW FDOOHG &3(6 &HQWHU IRU 3ODVPD (GJH 6LPXODWLRQ 7KH WHFKQRORJLHV XVHG DUH IURP DOO WKUHH DUHDV DQG LQFOXGH ZRUNIORZ DQDO\VLV , 2 VSHHG XS GDWD PRYHPHQW WHFKQRORJLHV DQG YLVXDO GDWD DQDO\VLV 7KH ),(67$ ZDV GHVLJQHG LQ FROODERUDWLRQ ZLWK DSSOLFDWLRQ XVHUV WR SURYLGH WKHP ZLWK VRSKLVWLFDWHG DQG SRZHUIXO FDSDELOLWLHV DFFHVVHG WKURXJK LQWXLWLYH ZHE LQWHUIDFHV :KLOH ),(67$ ZDV GHVLJQHG LQ UHVSRQVH WR WKH &3(6 SURMHFW LW ZDV GHYHORSHG DV D JHQHUDO IUDPHZRUN WKDW FDQ EH XVHG LQ RWKHU DSSOLFDWLRQ GRPDLQV :H DUH FXUUHQWO\ HQJDJLQJ &RPEXVWLRQ DQG $VWURSK\VLFV VFLHQWLVWV ZKR H[SUHVVHG LQWHUHVW LQ XVLQJ WKLV IUDPHZRUN DV ZHOO
'HWDLOV RI SURJUHVV LQ HDFK RI WKH WKUHH DUHDV 6($ '0$ DQG 63$ DV ZHOO DV ),(67$ IROORZ 7KLV LV IROORZHG E\ D SXEOLFDWLRQV DQG UHIHUHQFHV VHFWLRQ DQG RXWUHDFK WXWRULDOV LQYLWHG WDONV DQG WKHVHV LQ Appendix 1 7KH 6'0 FHQWHU KDV KDYH GHYHORSHG QXPHURXV FROODERUDWLRQV ZLWK DSSOLFDWLRQ SURMHFWV DQG RWKHU FHQWHUV DQG LQVWLWXWHV 7KLV LV VXPPDUL]HG LQ D FRORU FRGHG WDEOH IRUP LQ appendix 2 DV ZHOO DV D VXPPDU\ GHVFULSWLRQ RI WKH FROODERUDWLYH WDVNV %XW ILUVW ZH GHVFULEH VHOHFWHG KLJKOLJKWV WKDW DUH GLVFXVVHG LQ WKHVH VHFWLRQV LQ PRUH GHWDLO
Selected Highlights of Achievements
x7KH ERRN RQ 6FLHQWLILF 'DWD 0DQDJHPHQW ZDV SXEOLVKHG
0HPEHUV RI WKH 6'0 FHQWHU HGLWHG DQG FRQWULEXWHG FKDSWHUV WR WKH
ERRN HQWLWOHG 36FLHQWLILF 'DWD 0DQDJHPHQW &KDOOHQJHV ([LVWLQJ
7HFKQRORJ\ DQG 'HSOR\PHQW′ >65 @ ,Q VL[ RXW RI WKLUWHHQ
FKDSWHUV WKH OHDG DXWKRUV DUH PHPEHUV RI WKH 6'0 FHQWHU DQG
DGGLWLRQDO PHPEHUV FRQWULEXWHG WR WKH FRQWHQW RI WKHVH FKDSWHUV
x$ ERRN RQ 6FLHQWLILF 'DWD 0LQLQJ ZDV SXEOLVKHG 7KH ERRN
36FLHQWLILF 'DWD 0LQLQJ $ 3UDFWLFDO 3HUVSHFWLYH ′ ZDV DXWKRUHG E\
D PHPEHU RI WKH 6'0 FHQWHU >.DP @
x7H[WERRN WLWOHG 33UDFWLFDO *UDSK 0LQLQJ ZLWK 5′ ZULWWHQ
HQWLUHO\ E\ VWXGHQWV WR EH SXEOLVKHG E\ &KDSPDQ +DOO &5&
3UHVV XQGHU WKH 'DWD 0LQLQJ DQG .QRZOHGJH 'LVFRYHU\ 6HULHV
ZDV GHOLYHUHG LQ -DQXDU\ DQG ZLOO EH SXEOLVKHG LQ
KWWS ZZZ FVF QFVX HGX QHZV 7KH OHDG HGLWRU DQG FR
HGLWRUV RI WKH ERRN DUH PHPEHUV RI WKH 6'0 FHQWHU
x+LJK SURGXFWLYLW\ LQ WKH 6'0 FHQWHU D ODUJH QXPEHU RI SDSHUV
SXEOLVKHG 'XULQJ WKH ODVW \HDUV -DQXDU\ ± -XO\
PHPEHUV RI WKH 6'0 FHQWHU SXEOLVKHG SDSHUV VHH SXEOLFDWLRQ OLVW DQG RUJDQL]HG DQG SUHVHQWHG QXPHURXV WXWRULDOV LQYLWHG WDONV RU LQYLWHG VHVVLRQV 3DSHUV SXEOLVKHG SULRU WR FDQ EH IRXQG LQ WKH 6'0 FHQWHU ZHE VLWH KWWS VGPFHQWHU OEO JRY
x3DUDOOHO 1HW&') VXFFHVVIXOO\ XVHG LQ SURGXFWLRQ 3DUDOOHO 1HW&') 3QHW&') GHVLJQHG EXLOW DQG VXSSRUWHG E\ 6'0 FHQWHU PHPEHUV LV QRZ XVHG LQ VHYHUDO SURGXFWLRQ FRGHV ,W KDV EHHQ VXFFHVVIXOO\ XVHG E\ WKH ODUJH VFDOH 1&$5 &RPPXQLW\ $WPRVSKHUH 0RGHO &$0 DQG *OREDO &ORXG 5HVROYLQJ 0RGHO *&50 $FFRUGLQJ WR x , 2 LPSURYHPHQW DFKLHYHG IRU FROOHFWLYH , 2 SDWWHUPV RQ WKH /XVWUH SDUDOOHO ILOH V\VWHP :H HQKDQFHG , 2 HIILFLHQF\ IRU WKH /XVWUH ILOH V\VWHP E\ DV PXFK DV E\ LQWURGXFLQJ D QRYHO WHFKQLTXH FDOOHG SDUWLWLRQHG FROOHFWLYH ,2 3DU&ROO 3DU&ROO SDUWLWLRQV SDUDOOHO SURFHVVHV LQWR VXEJURXSV HDFK FDUU\LQJ RXW VPDOOHU \HW DJJUHJDWHG ,2 RSHUDWLRQV 7KLV WHFKQLTXH LV LPSRUWDQW LQ SDUW EHFDXVH LW GRHV QRW UHTXLUH D FKDQJH LQ ILOH IRUPDW 6HH GHWDLOV LQ 6HFWLRQ x1HZ KLJK SHUIRUPDQFH GULYHU IRU /XVWUH GHYHORSHG DQG LQWHJUDWHG LQWR SRSXODU SDFNDJHV :H KDYH GHYHORSHG D /XVWUH GULYHU IRU 03, ,2 WKDW HQDEOHV KLJKHU SHUIRUPDQFH E\ EHWWHU WXQLQJ DFFHVV WR DYRLG SHUIRUPDQFH SLWIDOOV ZLWK /XVWUH ILOH V\VWHP RQ WKH &UD\ PDFKLQHV 7KLV GULYHU KDV EHHQ LQWHJUDWHG LQWR 09$3,&+ YHUVLRQ ZKLFK LV D SRSXODU 03, LPSOHPHQWDWLRQ IRU ,QILQL%DQG FOXVWHUV DQG 03,&+ YHUVLRQ ZKLFK VHUYHV DV WKH EDVLV RI PRVW YHQGRU LPSOHPHQWDWLRQV 7KURXJK WKHVH GLVWULEXWLRQV WKLV WHFKQRORJ\ ZLOO HQKDQFH , 2 SHUIRUPDQFH IRU D YDULHW\ RI DSSOLFDWLRQV XVLQJ 03, ,2 GLUHFWO\ RU WKURXJK VXFK KLJK OHYHO , 2 OLEUDULHV DV +') DQG 3QHW&') 6HH GHWDLOV LQ 6HFWLRQ x$',26 VSHHGV XS , 2 RQ &UD\ ;7 ,QILQL%DQG FOXVWHUV DQG ,%0 %OXH *HQH 3 :H GHYHORSHG WKH $GDSWDEOH , 2 V\VWHP $',26 D FRPSRQHQWL]DWLRQ RI WKH , 2 OD\HU ,W SURYLGHV DQ HDV\ WR XVH SURJUDPPLQJ LQWHUIDFH WKDW DEVWUDFWV WKH , 2 PHWDGDWD LQIRUPDWLRQ DQG GDWD VWUXFWXUHV IURP WKH VRXUFH FRGH LQWR DQ H[WHUQDO ;0/ ILOH $',26 DOORZV XVHUV WR RQO\ FKDQJH WKH GHFODUDWLRQ RI WKH WUDQVSRUW PHWKRGV LQ WKH ;0/ ILOH ZLWKRXW DQ\ VRXUFH FRGH PRGLILFDWLRQ $',26 FRQWDLQV D QHZ ILOH IRUPDW ES ELQDU\ SDFNHG WKDW LV KLJKO\ RSWLPL]HG IRU , 2 IRU FKHFNSRLQW RSHUDWLRQV $',26 KDV EHHQ LQWHJUDWHG LQWR WKH &KLPHUD *7& *76 ;*& DQG 6 ' FRGHV DQG FDQ DFKLHYH XS WR *% V RQ D *% V ILOH V\VWHP $',26 KDV LPSURYHG , 2 IRU WKH ;*& FRGH RQ . SURFHVVRUV E\ [ ,Q RQH FDVH FRPSDULQJ &KLPHUD ZLWK SDUDOOHO +') ZLWK $',26 XVLQJ ES WKH LPSURYHPHQW JDLQ ZDV RYHU [ RQ FRUHV 6HH PRUH GHWDLOV LQ VHFWLRQV DQG x)DVW%LW LQGH[LQJ WHFKQRORJ\ UHFHLYHG 5 ' DZDUG )DVW%LW LV D YHU\ HIILFLHQW LQGH[LQJ WHFKQRORJ\ IRU DFFHOHUDWLQJ GDWDEDVH TXHULHV RQ PDVVLYH GDWDVHWV )DVW%LW KDV EHHQ SURYHQ WR EH WKHRUHWLFDOO\ RSWLPDO LW SHUIRUPV WLPHV IDVWHU WKDQ DQ\ NQRZQ LQGH[LQJ PHWKRG EDVHG RQ LWV XVH RI RXU SDWHQWHG FRPSUHVVLRQ PHWKRG ,W FDQ VHDUFK RYHU PXOWL YDULDEOH VFLHQWLILF GDWD ZKHUH DWWULEXWHV KDYH KLJK FDUGLQDOLW\ QXPEHU RI SRVVLEOH YDOXHV 7KHVH XQLTXH FKDUDFWHULVWLFV PDGH LW XVHIXO LQ D YDULHW\ RI VFLHQWLILF DSSOLFDWLRQV 2XU LPSOHPHQWDWLRQ ZDV SDFNDJHG LQ DQG UHOHDVHG XQGHU DQ RSHQ VRXUFH OLFHQVH DQG KDV DWWUDFWHG D ORW RI LQWHUHVW LQ PXOWLSOH VFLHQWLILF DSSOLFDWLRQV DV ZHOO DV QHZ DUHDV VXFK DV QHWZRUN WUDIILF GDWD DQDO\VLV DQG TXHU\ EDVHG YLVXDOL]DWLRQ ,W UHFHLYHG WKH SUHVWLJLRXV 5 ' DZDUG LQ 6HH GHWDLOV LQ 6HFWLRQ x$ KXJH IROG VSHHGXS RI SDUWLFOH VHDUFK IRU WKH /DVHU :DNHILHOG 3DUWLFOH $FFHOHUDWRU SURMHFW DWWDLQHG ZLWK )DVW%LW :H XVHG )DVW%LW WR VSHHG XS WKH RSHUDWLRQV RI VHDUFKLQJ DQG WUDFNLQJ SDUWLFOHV LQ /DVHU :DNHILHOG 3DUWLFOH $FFHOHUDWRU /:3$ SURMHFW MRLQW ZLWK 9$&(7 FHQWHU %\ UHSODFLQJ DQ H[LVWLQJ ,'/ EDVHG DQDO\VLV SURJUDP ZLWK D )DVW%LW EDVHG SURJUDP ZH REVHUYHG D WKUHH RUGHUV RI PDJQLWXGH VSHHGXS IURP VHFRQGV WR VHFRQGV LQ WKH ILUVW WHVW UXQ ,Q DQRWKHU DSSOLFDWLRQ )DVW%LW RSHQ VRXUFH WHFKQRORJ\ ZDV XVHG ZLWKRXW LQYROYLQJ WKH )DVW%LW GHYHORSHUV LQ D VRIWZDUH FDOOHG 7UL[; %0, ,W KDV EHHQ UHSRUWHG WKDW )DVW%LW HQDEOHG VFUHHQLQJ OLEUDULHV RI OLJDQGV WLPHV IDVWHU WKDQ WKH VWDWH RI DUW VFUHHQLQJ WRROV 6HH GHWDLOV LQ 6HFWLRQ x$QRWKHU IROG VSHHGXS ZDV DOVR DFKLHYHG ZLWK VSHFLDOL]HG )DVW%LW VWUXFWXUHV IRU *\URNLQHWLF )XVLRQ UHJLRQ LGHQWLILFDWLRQ :H GHYHORSHG D VSHFLDOL]HG ELWPDS LQGH[ )DVW%LW VWUXFWXUH E\ GLUHFWO\ XWLOL]LQJ WKH PHVK VWUXFWXUH RI *\URNLQHWLF 7UDQVSRUW &RGH *7& IRU VLPXODWLQJ WKH PDJQHWLFDOO\ FRQILQHG IXVLRQ SODVPD &RQVHTXHQWO\ ZH ZHUH DEOH WR LPSURYH WKH VSHHG RI LGHQWLI\LQJ UHJLRQV RI LQWHUHVW E\ QHDUO\ IROG VHH GHWDLOV LQ VHFWLRQ x$Q DFFXUDWH WRRO IRU FODVVLILFDWLRQ RI RUELWV LQ 3RLQFDUH SORWV GHYHORSHG DQG GHSOR\HG 6RIWZDUH IRU WKH DXWRPDWLF FODVVLILFDWLRQ RI RUELWV LQ 3RLQFDUH SORWV ZDV GHYHORSHG DQG GHSOR\HG IRU XVH E\ 333/ DQG RWKHU IXVLRQ VFLHQWLVWV VROYLQJ D ORQJ VWDQGLQJ SUREOHP IRU VFLHQWLVWV LQ WKLV GRPDLQ ,W WDNHV DV LQSXW WKH FRRUGLQDWHV RI WKH SRLQWV LQ DQ RUELW DQG DVVLJQV WR LW RQH RI IRXU FODVV ODEHOV EDVHG RQ WKH VKDSH WUDFHG RXW E\ WKH SRLQWV .H\ FKDOOHQJHV WR VROYLQJ WKLV SUREOHP ZHUH WKH H[WUDFWLRQ RI UREXVW IHDWXUHV UHSUHVHQWLQJ WKH RUELWV DQG WKH FUHDWLRQ RI D KLJK TXDOLW\ WUDLQLQJ VHW 7KH FURVV YDOLGDWLRQ HUURU UDWH XVLQJ HQVHPEOHV RI GHFLVLRQ WUHHV LV OHVV WKDQ DQG WKH FRGH ZRUNV TXLWH ZHOO LQ WKH ZRUGV RI D SK\VLFLVW XVLQJ LW ZKR KDV UHFRPPHQGHG LW WR FROOHDJXHV 7KLV VRIWZDUH ZRXOG UHSODFH WHGLRXV PDQXDO ODEHOLQJ RI WKH RUELWV ZKLFK LV RIWHQ HUURU SURQH DQG VXEMHFWLYH 6HH GHWDLOV LQ VHFWLRQ x3DUDOOHO 5 S5 IRU KLJK SHUIRUPDQFH VWDWLVWLFDO FRPSXWLQJ GHOLYHUHG VXSHU OLQHDU VFDOLQJ :H KDYH GHYHORSHG SDUDOOHO 5 S5 PLGGOHZDUH IRU DQ HDV\ WR XVH DOPRVW ]HUR RYHUKHDG SOXJ LQ RI SDUDOOHO DQDO\VLV IXQFWLRQV ZULWWHQ LQ FRPSLOHG ODQJXDJHV LQWR D ZLGHO\ XVHG RSHQ VRXUFH 5 VWDWLVWLFDO HQYLURQPHQW S5 GHOLYHUHG D VXSHU OLQHDU VFDODELOLW\ LQ WHUPV RI WKH QXPEHU RI SURFHVVRUV DQG LPSURYHG WKH SHUIRUPDQFH RI WKH VWDWH RI WKH DUW WHFKQRORJ\ E\ DQ DYHUDJH IDFWRU RI ,WV 56FD/$3$&. OLEUDU\ LV GLVWULEXWHG DV DQ 530 SDFNDJH DFURVV GLIIHUHQW /LQX[ GLVWULEXWLRQV DQG LQ PRUH WKDQ FRXQWULHV ZRUOG ZLGH WKURXJK WKH 5?V &5$1 GLVWULEXWLRQ VLWH 3DUDOOHO 5 IRUPV D VHUYHU VLGH DQDO\VLV HQJLQH ZLWK D VHOHFW VHW RI DQDO\VLV URXWLQHV LQ WKH 'DVKERDUG ZHE DSSOLFDWLRQ 7KH LQLWLDO VHW RI URXWLQHV ZDV LGHQWLILHG EDVHG RQ WKHLU IUHTXHQW XVDJH E\ FOLPDWH DQG IXVLRQV FRPPXQLWLHV S5 LV GLVFXVVHG IXUWKHU LQ 6HFWLRQ x$ QHZ WRRO IRU DXWRPDWLF GLVFRYHU\ RI IURQW GHWHFWLRQ GHYHORSHG :H FUHDWHG DQ DQDO\WLFDO PHWKRGRORJ\ IRU DXWRPDWLF GLVFRYHU\ RI WXUEXOHQW SDWWHUQV QDPHO\ IURQW GHWHFWLRQ DQG WUDFNLQJ ERWK LQ VSDFH DQG WLPH LQ WKH HOHFWULFDO SRWHQWLDO IOXFWXDWLRQ E\ SODVPD WXUEXOHQFH GDWD IURP WKH ;*& IXVLRQ VLPXODWLRQV 7KH WRRO XVHV $XWRPDWLF 3DUDOOHOL]DWLRQ RI 'DWD 3DUDOOHO 6WDWLVWLFDO &RPSXWLQJ &RGHV VHH GHWDLOV LQ VHFWLRQ x2UGHUV RI PDJQLWXGH IROG VSHHGXS DFKLHYHG IRU $OO 3DLUV 6LPLODULW\ 6HDUFK $366 7KH VFDODEOH DOJRULWKP ZDV GHYHORSHG ZLWK VSHFLDOL]HG LQGLFHV DQG KHXULVWLF RSWLPL]DWLRQ RYHU GDWD VHWV ZLWK PLOOLRQV RI UHFRUGV LQ KLJK GLPHQVLRQDO VSDFHV :H GHYHORSHG DQ RSHQ VRXUFH OLEUDU\ RI DOJRULWKPV IRU IDVW LQFUHPHQWDO DQG VFDODEOH DOO SDLUV VLPLODULW\ VHDUFKHV VHH GHWDLOV LQ VHFWLRQ x'HHSHU LQVLJKWV LQWR IXVLRQ GDWD DFKLHYHG 7KH LQLWLDO DQDO\VLV E\ WKH 6'0 FHQWHU RI FRKHUHQW VWUXFWXUHV LQ *\URNLQHWLF 6LPXODWLRQ RI (QHUJHWLF 3DUWLFOH *6(3 6FL'$& FHQWHU?V IOXLG GDWD LV SURYLGLQJ SUHYLRXVO\ XQH[SORUHG LQVLJKWV LQWR WKH VWDWLVWLFV RI WKH VWUXFWXUHV LQ WKH LRQ KHDW IOX[ YDULDEOH ,W GLVFRYHUHG WKDW WKHUH DUH VRPH VPDOO VWUXFWXUHV ZLWK QHJDWLYH LRQ KHDW IOX[ WKDW QHHG IXUWKHU LQYHVWLJDWLRQ WR GHWHUPLQH LI WKH\ DUH GXH WR QRLVH RU SK\VLFV VHH GHWDLOV LQ VHFWLRQ x3UR5DWD HQDEOHG V\VWHPV ELRORJ\ VWXGLHV LQ YDULRXV '2( HQHUJ\ DQG HQYLURQPHQW DSSOLFDWLRQV :H KDYH EURXJKW WR SURGXFWLRQ RXU RSHQ VRXUFH 3UR5DWD UREXVW VWDWLVWLFDO VRIWZDUH ZLWK WKH *8, IRU KLJK WKURXJKSXW TXDQWLWDWLYH VKRWJXQ SURWHRPLFV 3UR5DWD KDV EHHQ GRZQORDGHG PRUH WKDQ WLPHV DQG KDV EHHQ XVHG E\ WKH '2( %LRHQHUJ\ FHQWHUV DQG *HQRPLFV *7/ SURMHFWV WR SUHGLFW WKH FRPSRVLWLRQ RI WKH FHOOXORVRPDO FRPSOH[ IRU ELRPDVV GHJUDGDWLRQ DQG WR SHUIRUP JHQRPH VFDOH IXQFWLRQDO DQQRWDWLRQ RI HWKDQRO SURGXFLQJ EDFWHULD WR LQIHU PHWDEROLF DURPDWLF FRPSRXQG GHJUDGDWLRQ SDWKZD\V E\ K\GURJHQ SURGXFLQJ EDFWHULD DQG WR XQGHUVWDQG WKH FRPSRVLWLRQ RI FRPSOH[ PLFURELDO FRPPXQLWLHV IURP HQYLURQPHQWDOO\ KD]DUGRXV VLWHV 3UR5DWD LV GLVFXVVHG LQ 6HFWLRQ x7KH .HSOHU GHYHORSHUV KRVWHG DQG FROODERUDWHG ZLWK WKH ,7(5 WHDP 7KH 6'0 &HQWHU FROODERUDWHG ZLWK WKH ,7(5 (XURSHDQ ,QWHJUDWHG 7RNDPDN 0RGHOOLQJ SURMHFW WHDP DW WKH ,QVWLWXWH RI )XVLRQ 5HVHDUFK )UDQFH 7KLV JURXS KDV VHOHFWHG .HSOHU WKH ZRUNIORZ V\VWHP GHYHORSHG E\ WKH 6'0 FHQWHU IRU WKHLU ZRUNIORZ GHYHORSPHQW DQG YLVLWHG WZLFH WR FRRUGLQDWH WKHLU ZRUN ZLWK WKH .HSOHU GHYHORSPHQW WHDP 'XULQJ WKH YLVLWV E\ ,7(5 SURMHFW WHDP PHPEHUV LQ DQG ZH VKDUHG RXU GHYHORSPHQWV RI YDULRXV FRPSRQHQWV RI LQWHUHVW WR WKH ,7(5 WHDPV .HSOHU GHYHORSPHQW LV GLVFXVVHG LQ 6HFWLRQ x1HZ ZRUNIORZ UHOLDELOLW\ DQG IDXOW WROHUDQFH WRROV ZHUH GHYHORSHG LQ .HSOHU 'HYHORSPHQW RI ZRUNIORZ UHOLDELOLW\ DQG IDXOW WROHUDQFH LQFOXGHG GHYHORSPHQW RI D QHZ PRGHO DQG RI WKH VSHFLILFDWLRQV DQG SLORW YHUVLRQV RI D .HSOHU EDVHG DOWHUQDWLYH DFWRU IRU ZRUNIORZ UHFRYHU\ 6HH VHFWLRQ x$XWRPDWLF SURYHQDQFH FDSWXUH IUDPHZRUN GHYHORSHG 5XQ WLPH SURYHQDQFH FDSWXUH VFULSWV DQG DXWRPDWLF GDWD EDVH IHHG KDYH EHHQ GHYHORSHG WR EH XVHG ZLWK WKH .HSOHU ZRUNIORZ V\VWHP 7KLV LQFOXGHV V\VWHP SURYHQDQFH RQ WKH VHWXS RI VLPXODWLRQ SURJUDPV WKH ZRUNIORZ SURYHQDQFH DQG GDWD SURYHQDQFH WKDW FDSWXUHV WKH KLVWRU\ RI HDFK GDWD ILOH 7KH GDWD SURYHQDQFH LV QRZ XVHG WR ILQG ILOHV RI LQWHUHVW DQG PRYH WKHP WR WKH XVHU PDFKLQH GLUHFWO\ IURP WKH GDVKERDUG ZKLFK LV XVHG WR PRQLWRU VLPXODWLRQV DQG VXSSRUW UHPRWH DQDO\VLV RI VLPXODWLRQ GDWD 6HH GHWDLOV LQ 6HFWLRQ DQG 6HFWLRQ x,QWHJUDWHG IUDPHZRUN IRU UHDO WLPH PRQLWRULQJ RI ODUJH VFDOH VLPXODWLRQV HOLPLQDWHG XQQHFHVVDU\ FRPSXWDWLRQV 7KH 6'0 FHQWHU KDV GHYHORSHG DQ LQWHJUDWHG IUDPHZRUN FXUUHQWO\ EHLQJ XVHG LQ SURGXFWLRQ UXQV E\ WKH &HQWHU IRU 3ODVPD )XVLRQ (GJH 6LPXODWLRQ &3(6 VFLHQWLVWV 7KH WHFKQRORJLHV SURYLGHG E\ WKH FHQWHU LQFOXGH WKH .HSOHU ZRUNIORZ V\VWHP D GDVKERDUG SURYHQDQFH WUDFNLQJ DQG UHFRUGLQJ SDUDOOHO DQDO\VLV FDSDELOLWLHV DQG 650 EDVHG GDWD PRYHPHQW 7KH 'DVKERDUG QRZ KDV IDVW YLVXDOL]DWLRQ ZLWK :HE DFFHVV DQG RWKHU IHDWXUHV LQFOXGLQJ WKH DELOLW\ WR FRPSDUH LPDJHV IURP PXOWLSOH WLPH VWHSV VKRWV DQG GLVSOD\ PRYLHV FRPSRVHG IURP PXOWLSOH LPDJHV E\ WKH ZRUNIORZ V\VWHP 7KLV LQWHJUDWHG V\VWHP KDV EHHQ XVHG WR SHUIRUP VLPXODWLRQ PRQLWRULQJ LQ UHDO WLPH DV ZHOO DV FRPSOH[ FRGH FRXSOLQJ WDVNV 0RQLWRULQJ LQFOXGHV G\QDPLF JHQHUDWLRQ RI JUDSKV DQG LPDJHV SRVWHG RQ WKH 'DVKERDUG ,Q D UHFHQW UHYLHZ RI WKH &3(6 SURMHFW DOO WKH UHYLHZHUV JDYH WKH SURMHFW WKH KLJKHVW SRVVLEOH JUDGH RI 3H[FHOOHQW ′ 6HH GHWDLOV LQ 6HFWLRQ x7KH ),(67$ IUDPHZRUN DSSOLHG WR D QHZ FRGH IRU UHDO WLPH FRGH PRQLWRULQJ 0XOWLSOH IXVLRQ FRGHV DV ZHOO DV FRGHV LQ RWKHU GRPDLQV KDYH EHHQ XVLQJ ZRUNIORZ WHFKQRORJ\ XVLQJ RXU ),(67$ IUDPHZRUN 7KH ZRUNIORZ WLHV WKH UHVXOWV RI WKH FRGH UXQV LQWR WKH 'DVKERDUG WKURXJK WKH 9LV,7 FRPPDQG OLQH LQWHUIDFH PDNLQJ WKH UHVXOWV TXLFNO\ DYDLODEOH WR VFLHQWLVWV VHH GHWDLOV LQ VHFWLRQ 1. Storage Efficient Access (SEA) 7KH FRUH , 2 IXQFWLRQDOLW\ RQ WRGD\?V KLJK SHUIRUPDQFH FRPSXWLQJ +3& V\VWHPV FRQVLVWV RI D FROOHFWLRQ RI , 2 VRIWZDUH WKDW SURYLGHV D FRQYHQLHQW DQG HIILFLHQW LQWHUIDFH WR WKH DYDLODEOH , 2 KDUGZDUH 7KH SURMHFWV LQ WKLV OD\HU IRFXV RQ WKLV FRUH , 2 IXQFWLRQDOLW\ DQG WKH\ KDYH WZR FRPSOHPHQWDU\ JRDOV )LUVW ZH GHYHORS DQG VXSSRUW D FROOHFWLRQ RI KLJKO\ VFDODEOH DQG IUHHO\ DYDLODEOH , 2 VRIWZDUH FRPSRQHQWV WKDW DUH XVHG LQ SURGXFWLRQ DSSOLFDWLRQV E\ VFLHQWLVWV DQG ZH DFWLYHO\ HQJDJH WKH FRPPXQLW\ WR KHOS DSSOLFDWLRQ VFLHQWLVWV EHWWHU XQGHUVWDQG KRZ WR XVH WKHVH WRROV 6HFRQG WKURXJK RXU LQWHUDFWLRQV ZLWK WKH FRPPXQLW\ ZH LGHQWLI\ VSHFLILF GHILFLHQFLHV LQ IXQFWLRQDOLW\ SHUIRUPDQFH DQG XVDELOLW\ WKDW ZH WKHQ ZRUN WR DGGUHVV 6XFFHVVIXO LPSURYHPHQWV DUH VXEVHTXHQWO\ LQWHJUDWHG LQWR SURGXFWLRQ UHOHDVHV HQVXULQJ WKDW WKHVH EHQHILWV DUH PDGH ZLGHO\ DYDLODEOH 2YHUDOO RXU ZRUN FDQ EH SODFHG LQ IRXU FDWHJRULHV GLVFXVVHG LQ WKH IROORZLQJ VHFWLRQV )LOH V\VWHP EHQFKPDUNLQJ DQG DSSOLFDWLRQ , 2 EHKDYLRU 3DUDOOHO , 2 LQIUDVWUXFWXUH HYROXWLRQ $SSOLFDWLRQ LQWHUIDFHV DQG GDWD PRGHOV 1H[W JHQHUDWLRQ , 2 VRIWZDUH WHFKQRORJLHV )LOH 6\VWHP %HQFKPDUNLQJ DQG $SSOLFDWLRQ , 2 %HKDYLRU 7KH KLJK SHDN UDWHV RI +3& , 2 V\VWHPV VLPSO\ GR QRW WUDQVODWH LQWR DGHTXDWH VXVWDLQHG SHUIRUPDQFH IRU FRPSXWDWLRQDO VFLHQFH DSSOLFDWLRQV 7KH URRW FDXVH RI WKLV SHUIRUPDQFH JDS LV WKH PLVPDWFK EHWZHHQ WKH UHTXLUHPHQWV RI WKH V\VWHP?V DSSOLFDWLRQV DQG WKH FDSDELOLWLHV RI , 2 KDUGZDUH DQG VRIWZDUH 6\VWHPDWLF HYDOXDWLRQ RI ERWK , 2 V\VWHP FDSDELOLWLHV DQG DSSOLFDWLRQ UHTXLUHPHQWV SURYLGHV PXFK QHHGHG LQVLJKW LQWR WKH HIILFLHQW XVH RI H[LVWLQJ V\VWHPV DQG KHOS JXLGH WKH GHVLJQ RI QH[W JHQHUDWLRQ , 2 V\VWHPV 7KH REMHFWLYH RI WKLV ZRUN LV WR VWXG\ ILOH V\VWHP FKDUDFWHULVWLFV WKDW KDYH VLJQLILFDQW LPSDFWV WR WKH SDUDOOHO , 2 RSHUDWLRQV DQG HYDOXDWH WKH UHODWLYH SHUIRUPDQFH RI WKH ILOH V\VWHPV DYDLODEOH WR LPSRUWDQW 6FL'$& DSSOLFDWLRQV RQ '2( FRPSXWH SODWIRUPV 7KH SHUIRUPDQFH IXQFWLRQDOLW\ DQG VFDODELOLW\ RI 03, ,2 SDUDOOHO QHW&') DQG +') DUH FULWLFDO IRU PDQ\ DSSOLFDWLRQV $FFRPSOLVKPHQWV :H LQVWUXPHQWHG WKH /XVWUH ILOH V\VWHP WR ORJ , 2 RSHUDWLRQV IRU WKH SXUSRVH RI XQGHUVWDQGLQJ , 2 SDWWHUQV DQG WUDFNLQJ GRZQ , 2 SHUIRUPDQFH GHJUDGDWLRQ XVLQJ FRPPRQ EHQFKPDUNV 7KHVH ORJV DUH XVHG WR WUDFN , 2 DFFHVVHV WR LQGLYLGXDO /XVWUH VHUYHUV DQG REVHUYH SHUIRUPDQFH RQ D ILQHU JUDQXODULW\ WKDQ SRVVLEOH ZLWK VLPSOH 03, ,2 ORJV 8VLQJ WKLV FDSDELOLW\ ZH KDYH H[WHQVLYHO\ FKDUDFWHUL]HG WKH SDUDOOHO , 2 SHUIRUPDQFH RQ WKH -DJXDU VXSHUFRPSXWHU ><29 @ :H KDYH H[DPLQHG WKH EHVW VWULSH VL]HV RYHU -DJXDU DQG VKRZHG WKDW WKH ILOH GLVWULEXWLRQ SDWWHUQ DFURVV WKH 'DWD 'LUHFW 1HWZRUNV ''1 VWRUDJH FRXSOHWV FDQ GUDPDWLFDOO\ LPSDFW SHUIRUPDQFH UHVXOWLQJ LQ D IDFWRU RI WZR GLIIHUHQFH IRU FHUWDLQ RSHUDWLRQV ><2& @ ,Q DGGLWLRQ ZH KDYH DOVR H[DPLQHG WKH VFDODELOLW\ RI PHWDGDWD DQG GDWD LQWHQVLYH RSHUDWLRQV 2XU UHVXOWV KDYH GHPRQVWUDWHG WKDW IRU SDUDOOHO ILOH RSHQ WKH VKDUHG ILOH PRGH KDV WKH EHVW VFDODELOLW\ FRPSDUHG WR WKH VHSDUDWH ILOH PRGH 0RUHRYHU ZH KDYH LQYHVWLJDWHG WKH SHUIRUPDQFH LPSDFWV RI SDUDOOHO , 2 WHFKQLTXHV IRU KDQGOLQJ VPDOO DQG QRQ FRQWLJXRXV , 2 LQFOXGLQJ GDWD VLHYLQJ DQG FROOHFWLYH , 2 :H KDYH DOVR GRFXPHQWHG WKDW ZLWKRXW VSHFLILF RSWLPL]DWLRQV WKH SHUIRUPDQFH RI ZULWHV LV KLQGHUHG E\ LQWHUQDO /XVWUH ORFN FRQWHQWLRQ ><92 @ :H DOVR REWDLQHG WKH 6 ' DSSOLFDWLRQ , 2 NHUQHO IURP RXU FROODERUDWRUV -DFTXHOLQH &KHQ DW 6DQGLD 1DWLRQDO /DERUDWRULHV 5DPDQDQ 6DQNDUDQ DQG 6FRWW .ODVN\ DW 251/ 6 ' LV D SDUDOOHO WXUEXOHQW FRPEXVWLRQ DSSOLFDWLRQ XVLQJ D GLUHFW QXPHULFDO VLPXODWLRQ VROYHU GHYHORSHG DW 6DQGLD 1DWLRQDO /DERUDWRULHV 7KH 6 '?V , 2 LV RULJLQDOO\ SURJUDPPHG LQ )RUWUDQ , 2 IXQFWLRQV DQG HDFK SURFHVV ZULWHV DOO LWV VXE DUUD\V WR D VHSDUDWH ILOH DW HDFK FKHFNSRLQW 7KLV DSSURDFK FDQ UHVXOW LQ WKRXVDQGV WR PLOOLRQV RI ILOHV IURP D VLQJOH SURGXFWLRQ UXQ :H LPSOHPHQWHG WKUHH , 2 PHWKRGV LQFOXGLQJ 03, , 2 SDUDOOHO QHW&') DQG SDUDOOHO +') $OO WKUHH PHWKRGV ZULWH WKH DUUD\V LQWR D VKDUHG ILOH LQ WKHLU JOREDOO\ FDQRQLFDO RUGHU 7KLV DSSURDFK UHGXFHV WKH QXPEHU RI ILOHV WR RQH SHU FKHFNSRLQW :H WHVWHG RQ -DJXDU XVLQJ XS WR SURFHVV FRUHV WKH WRWDO QXPEHU RI FRUHV RQ -DJXDU ZDV WKHQ DQG EDVHG RQ RXU DQDO\VLV ZH GHPRQVWUDWHG WKDW LW LV SRVVLEOH WR LPSURYH WKH VFDODELOLW\ RI D UHSUHVHQWDWLYH DSSOLFDWLRQ 6 ' E\ RSWLPL]LQJ LWV , 2 DFFHVV SDWWHUQ 7KH DJJUHJDWHG , 2 EDQGZLGWK RI 6 ' FDQ EH VXVWDLQHG WR YHU\ ODUJH VFDOH )RU H[DPSOH ZH GHPRQVWUDWHG WKDW WKHUH LV D EDQGZLGWK LPSURYHPHQW E\ FRQWUROOLQJ WKH ILOH GLVWULEXWLRQ SDWWHUQ LQ WKH RULJLQDO 6 ' SURJUDP ,Q DGGLWLRQ UHSODFLQJ WKH RULJLQDO ILOH SHU SURFHVV LPSOHPHQWDWLRQ ZLWK D VKDUHG ILOH FDQ DYRLG D EDQGZLGWK GURS DFURVV SURFHVVHV EHFDXVH RI WKH UHGXFWLRQ RI WLPH VSHQW LQ SDUDOOHO ILOH FUHDWLRQ ><92 @ :H KDYH FRQVWUXFWHG WZR EHQFKPDUNV +3,2 Array >&$/ @ DQG 6 D6LP >&)/ @ IRU HYDOXDWLQJ WKH VFDODELOLW\ RI SDUDOOHO , 2 ZLWK RYHUODSSLQJ QRQ RYHUODSSLQJ DQG FRQWLJXRXV QRQ FRQWLJXRXV SDWWHUQV 8VLQJ WKHVH EHQFKPDUNV DV ZHOO DV %7,2 1$6$ EHQFKPDUN 6 ' , 2 FRPEXVWLRQ FRGH DQG )/$6+ , 2 DVWURSK\VLFV FRGH ZH HYDOXDWHG VKDUHG ILOH , 2 SHUIRUPDQFH RQ WKH /XVWUH DQG *3)6 ILOH V\VWHPV $V RQ WKH -DJXDU V\VWHP ZH REVHUYHG VLJQLILFDQW SHUIRUPDQFH GHJUDGDWLRQ GXH WR WKH ILOH V\VWHP ORFN FRQWHQWLRQ 2XU H[SHULPHQWV VKRZ WKDW LI WKH , 2 UHTXHVWV DUH FDUHIXOO\ DOLJQHG ZLWK ILOH V\VWHP ORFN ERXQGDULHV SHUIRUPDQFH FDQ EH LPSURYHG LQ VRPH FDVHV E\ DQ RUGHU RI PDJQLWXGH RU PRUH 6HH JUDSK LQ )LJXUH :H KDYH ZRUNHG ZLWK WKH $/&) WHDP WR GHSOR\ WKH 'DUVKDQ , 2 FKDUDFWHUL]DWLRQ WRRO GHYHORSHG XQGHU EDVH IXQGLQJ ):3 RQ RXU %OXH *HQH V\VWHP 'DUVKDQ KDV EHHQ LQWHJUDWHG LQWR WKH VRIWZDUH VWDFN VXFK WKDW WKH , 2 RI PRVW DSSOLFDWLRQV LV DXWRPDWLFDOO\ DQG WUDQVSDUHQWO\ FKDUDFWHUL]HG $IWHU PRQWKV RI VXFFHVVIXO XVH ZH VHOHFWHG D WZR PRQWK SHULRG DQG SHUIRUPHG D GHWDLOHG DQDO\VLV RI WKH ZRUNORDGV VHHQ GXULQJ WKLV WLPH :H IXUWKHU XVHG WKH GDWD WR LGHQWLI\ DQ , 2 LQWHQVLYH DSSOLFDWLRQ LQ QHHG RI LPSURYHPHQW WKH +6&' FRPEXVWLRQ SK\VLFV FRGH DQG UDSLGO\ LPSOHPHQWHG , 2 RSWLPL]DWLRQV IRU WKH FRGH 7KLV ZRUN ZRQ WKH EHVW SDSHU DZDUG DW 0667 >&+$ @ 3DUDOOHO , 2 ,QIUDVWUXFWXUH (YROXWLRQ 0XOWLSOH SDUDOOHO ILOH V\VWHP RSWLRQV DUH QRZ DYDLODEOH DQG PRVW +3& V\VWHPV QRZ LQFOXGH D UXGLPHQWDU\ , 2 VRIWZDUH VWDFN +RZHYHU WKH SHUIRUPDQFH RI WKH , 2 VWDFN RQ PDQ\ V\VWHPV LV PXFK ORZHU WKDQ SRVVLEOH JLYHQ WKH KDUGZDUH DYDLODEOH $V +3& V\VWHPV VFDOH DQG DSSOLFDWLRQ FRPSOH[LW\ LQFUHDVHV H[WUDFWLQJ WKH KLJKHVW SRVVLEOH SHUIRUPDQFH IURP WKH , 2 KDUGZDUH LV FULWLFDO WR WKH RYHUDOO HIIHFWLYHQHVV RI WKH V\VWHP 7KH REMHFWLYH RI WKLV ZRUN LV WR LPSURYH WKH VWDWH RI SDUDOOHO , 2 VXSSRUW IRU +3& 7KH 3DUDOOHO 9LUWXDO )LOH 6\VWHP 39)6 DQG 520,2 03, ,2 LPSOHPHQWDWLRQV DUH LQ ZLGH XVH DQG SURYLGH NH\ SDUDOOHO , 2 IXQFWLRQDOLW\ 7KLV ZRUN EXLOGV RQ WKHVH WZR FRPSRQHQWV E\ HQKDQFLQJ WKHP LQ RUGHU WR HQVXUH WKHVH FDSDELOLWLHV FRQWLQXH WR EH DYDLODEOH DV V\VWHPV FRQWLQXH WR VFDOH ,Q DGGLWLRQ WR LPSURYHPHQWV WR WKHVH WRROV VSHFLDO DWWHQWLRQ LV SDLG WR &UD\ V\VWHPV XVLQJ WKH /XVWUH SDUDOOHO ILOH V\VWHP $FFRPSOLVKPHQWV 0DQ\ DGYDQFHV LQ WKH 39)6 SDUDOOHO ILOH V\VWHP >&/5 @ SURMHFW ZHUH IDFLOLWDWHG E\ 6'0 VXSSRUW 7KUHH 39)6 UHOHDVHV ZHUH PDGH EHWZHHQ DQG LQFOXGLQJ PDQ\ EXJ IL[HV 0\ULFRP 0; DQG 3RUWDOV FRPPXQLFDWLRQ GULYHUV DQG D QHZ ILOH GLVWULEXWLRQ PHFKDQLVP )XQFWLRQDOLW\ ZDV DOVR DGGHG WR 39)6 WR DOORZ FRQWURO RI OD\RXW RI ILOHV IDFLOLWDWLQJ UHVHDUFK EHLQJ SHUIRUPHG LQ DFWLYH VWRUDJH DW 311/ 7KLV IXQFWLRQDOLW\ ZDV DOVR UROOHG LQWR D UHOHDVH 2Q WKH UHVHDUFK VLGH ZH KDYH EHHQ LPSOHPHQWLQJ WUDFLQJ LQ 39)6 DQG 520,2 DORQJ ZLWK FROODERUDWRUV $URRQ 1DWDUDM DQG $O 0DORQ\ 8 RI 2UHJRQ )$6726 ([WUHPH 26 3URMHFW DQG .ZDQ /LX 0D 8& 'DYLV 6FL'$& 8OWUDYLV ,QVWLWXWH ZLWK WKH JRDO RI SHUIRUPDQFH YLVXDOL]DWLRQ RI WKH HQWLUH , 2 VRIWZDUH VWDFN RQ WKH %OXH *HQH 3 V\VWHP ,QLWLDO YLVXDOL]DWLRQV KDYH EHHQ JHQHUDWHG %HFDXVH RI WKH FHQWUDO UROH &UD\ V\VWHPV SOD\ LQ '2( FRPSXWH LQIUDVWUXFWXUH ZH KDYH SODFHG H[WUD HPSKDVLV RQ XQGHUVWDQGLQJ DQG LPSURYLQJ , 2 SHUIRUPDQFH RQ WKH &UD\ ;7 V\VWHPV LQ SDUWLFXODU ZKHQ XVLQJ WKH /XVWUH ILOH V\VWHP 7R EHWWHU XQGHUVWDQG , 2 RQ WKH V\VWHP ZH EXLOW DQ DOWHUQDWLYH 03, ,2 SDFNDJH IRU WKH &UD\ ;7 DQG /XVWUH VWDUWLQJ ZLWK WKH 520,2 LPSOHPHQWDWLRQ FDOOHG WKH 2SSRUWXQLVWLF DQG $GDSWLYH 03, ,2 /LEUDU\ RYHU /XVWUH 23$/ ><9& <9& @ 8VLQJ 23$/ ZH KDYH SURILOHG WKH LQWHUQDO SURFHVVLQJ RI FROOHFWLYH , 2 RSHUDWLRQV RQ WKH &UD\ ;7 DQG XQFRYHUHG RSSRUWXQLWLHV IRU LPSURYHPHQWV :H GHYHORSHG D WHFKQLTXH FDOOHG SDUWLWLRQHG FROOHFWLYH ,2 3DU&ROO WKDW SDUWLWLRQV SDUDOOHO SURFHVVHV LQWR VXEJURXSV HDFK FDUU\LQJ RXW VPDOOHU \HW DJJUHJDWHG , 2 RSHUDWLRQV ,Q GRLQJ VR WKLV SDUWLWLRQLQJ UHGXFHV WKH FRVWV RI FROOHFWLYH RSHUDWLRQV DQG LPSURYHV WKH VFDODELOLW\ RI FROOHFWLYH ,2 E\ DV PXFK DV RQ SURFHVVHV ><9 @ 0RUHRYHU H[SHULPHQWV RQ /LQX[ FOXVWHUV VXJJHVW WKDW WKH WHFKQLTXH FDQ EH DSSOLHG WR D ZLGH UDQJH RI SODWIRUPV :H KDYH FROODERUDWHG ZLWK WKH 1DWLRQDO &HQWHU RI &RPSXWDWLRQDO 6FLHQFHV DQG KDYH GHSOR\HG RXU &UD\ /XVWUH RSWLPL]HG OLEUDU\ RQ -DJXDU DV D FRQWULEXWHG DOWHUQDWLYH SDFNDJH DW 2DN 5LGJH 1DWLRQDO /DERUDWRU\ DQG ZH KDYH ZRUNHG ZLWK WKH 09$3,&+ WHDP WR KDYH RXU VSHFLILF LPSURYHPHQWV LQWHJUDWHG LQWR WKLV SRSXODU GLVWULEXWLRQ 7KURXJK FROODERUDWLRQ ZLWK WKH $UJRQQH /HDGHUVKLS &RPSXWLQJ )DFLOLW\ $/&) ZH KDYH HQVXUHG WKDW WKH , 2 V\VWHP RQ WKH %OXH *HQH 3 V\VWHP ZLOO PHHW SHUIRUPDQFH DQG UHOLDELOLW\ JRDOV 7KLV LQFOXGHV DLGLQJ LQ WKH VSHFLILFDWLRQ RI WKH VWRUDJH KDUGZDUH SRUWLQJ DQG GHSOR\PHQW RI 39)6 DW ODUJH VFDOH DQG ZRUNLQJ ZLWK ,%0 WR VROYH D VLJQLILFDQW IXQFWLRQDOLW\ SUREOHP LQ HDUO\ YHUVLRQV RI WKHLU 03, ,2 VRIWZDUH IRU WKH V\VWHP :H LPSOHPHQWHG D ORFN IUHH GULYHU IRU WKH %OXH *HQH WKDW HQDEOHV 39)6 XVH LPSURYHG WKH VFDODELOLW\ RI VRPH PHWDGDWD RSHUDWLRQV DQG LQWHJUDWHG ,%0?V FKDQJHV EDFN LQWR WKH 520,2 VRXUFH WUHH :H LQFRUSRUDWHG VXFFHVVIXO UHVHDUFK HIIRUWV LQWR WKH SURGXFWLRQ 520,2 03, ,2 OLEUDU\ >7*/ @ LQFOXGLQJ /XVWUH VSHFLILF LPSURYHPHQWV ILOH GRPDLQ DQG VWULGHG , 2 RSWLPL]DWLRQV :H KDYH DOVR GHYHORSHG DGGLWLRQDO WHVW FDVHV WR H[HUFLVH QHZ FRGH SDWKV LQ WKH ,%0 %OXH *HQH 3 03, ,2 LPSOHPHQWDWLRQ UHVXOWLQJ IURP WKHLU ZRUN WR IL[ OLPLWDWLRQV GXH WR D ELW SRLQWHU RQ WKH V\VWHP :H KDYH DOVR FRPSOHWHG SURWRW\SHV RI 03, ,2 DWRPLF PRGH DQG VKDUHG ILOH SRLQWHU RSHUDWLRQV XVLQJ 03, 50$ DQG SRLQW WR SRLQW FRPPXQLFDWLRQ 7KHVH WHFKQLTXHV SURYLGH DQ RSWLRQ IRU DYRLGLQJ ILOH V\VWHP ORFNLQJ RYHUKHDG RQFH 03, IHDWXUHV EHFRPH DYDLODEOH RQ OHDGHUVKLS FRPSXWLQJ SODWIRUPV >/57 @ 7R IXUWKHU DGGUHVV WKH GUDZEDFN RI XVLQJ EORFN EDVHG ILOH ORFN SURWRFRO ZH KDYH LPSOHPHQWHG D VFDODEOH GLVWULEXWHG ORFN PDQDJHPHQW PHWKRG WKDW SURYLGHV WUXH E\WH UDQJH ORFNLQJ JUDQXODULW\ :H XVHG 6 ' , 2 DQG 6 D6LP EHQFKPDUNV WR HYDOXDWH VHYHUDO ORFN VWUDWHJLHV DQG REVHUYHG WKH LPSURYHPHQW RI ORFNLQJ WKURXJKSXW XS WR EHWZHHQ RQH WR WZR RUGHUV RI PDJQLWXGH LQ SHUIRUPDQFH :H DOVR GHYHORSHG D K\EULG WZR SKDVH ORFNLQJ SURWRFRO WR LPSURYH WKH QRQ FRQWLJXRXV , 2 SHUIRUPDQFH >&5/ @ $SSOLFDWLRQ ,QWHUIDFHV DQG 'DWD 0RGHOV ,Q RUGHU WR PDNH DSSOLFDWLRQV PRUH QLPEOH ZLWK UHVSHFW WR WKHLU , 2 EHKDYLRU PRUH HIIRUW PXVW EH VSHQW RQ WKH DSSOLFDWLRQV DQG WKH LQWHUIDFHV WKDW WKH\ XVH WR LQWHUDFW ZLWK WKH , 2 V\VWHP 7KH REMHFWLYH RI WKLV ZRUN LV WR LPSURYH WKH XVDELOLW\ DQG REVHUYHG )LJXUH 3DUDOOHO QHW&') , 2 WKURXJKSXW IRU DSSOLFDWLRQV XVLQJ SDUDOOHO , 2 E\ HQKDQFHPHQWV WR RU UHSODFHPHQWV IRU SRSXODU DSSOLFDWLRQ LQWHUIDFHV WR SDUDOOHO , 2 UHVRXUFHV 7KLV WDVN ZDV DGGHG LQ UHVSRQVH WR D SHUFHLYHG QHHG IRU LPSURYHG SHUIRUPDQFH DW WKLV OD\HU %HFDXVH RI LWV SRSXODULW\ LQ WKH VFLHQWLILF FRPPXQLW\ ZH KDYH IRFXVHG RQ WKH 1HW&') LQWHUIDFHV DQG LQ SDUWLFXODU RQ D SDUDOOHO LQWHUIDFH WR 1HW&') ILOHV $FFRPSOLVKPHQWV 6LJQLILFDQW ZRUN KDV JRQH LQWR PDNLQJ WKH 3DUDOOHO QHW&') 3QHW&') >//& @ VRIWZDUH UHDG\ IRU SURGXFWLRQ 7KH RULJLQDO LGHD RI 3QHW&') LV LOOXVWUDWHG LQ WKH )LJXUH ,QVWHDG RI KDYLQJ DOO SURFHVVRUV FRPPXQLFDWH ZLWK WKH SDUDOOHO ILOH V\VWHP WKRXJK D VLQJOH SURFHVVRU 3QHW&') OHWV HDFK SURFHVVRU FRPPXQLFDWH GLUHFWO\ ZLWK SDUDOOHO ILOH V\VWHP :H PRYHG WR XVLQJ 691 DQG 7UDF WR PDQDJH WKH 3DUDOOHO QHW&') VRXUFH WUHH IDFLOLWDWLQJ JUHDWHU FRPPXQLW\ LQYROYHPHQW :H DOVR LPSURYHG WKH VRIWZDUH WR EHWWHU RSHUDWH RQ %OXH *HQH / %OXH *HQH 3 DQG 6L&RUWH[ V\VWHPV 6XSSRUW IRU LQFUHDVLQJO\ ODUJH GDWDVHWV KDV EHFRPH D FULWLFDO LVVXH IRU 3QHW&') 7KH RULJLQDO 8&$5 QHW&') IRUPDW VXSSRUWV YDULDEOHV XS WR *%\WHV GXH WR ELW OLPLWDWLRQV ZLWKRXW VSHFLDO ZRUN EXW RXU XVHUV DUH EHJLQQLQJ WR VXUSDVV WKLV OLPLW :H KDYH GHYHORSHG DQ H[WHQVLRQ WKDW XVHV ELWV IRU VL]HV QDPHG WKH 3&') ′ IRUPDW WKDW DOORZV XV WR VWRUH YDULDEOHV RI HIIHFWLYHO\ XQOLPLWHG VL]H :H KDYH V\QFKURQL]HG WKHVH FKDQJHV ZLWK WKH VHULDO QHW&') WHDP VR WKDW VHULDO WRROV FDQ LQWHURSHUDWH 3QHW&') LV QRZ XVHG LQ VHYHUDO SURGXFWLRQ FRGHV 5HFHQWO\ LW KDV EHHQ VXFFHVVIXOO\ XVHG E\ WKH ODUJH VFDOH *OREDO &ORXG 5HVROYH 0RGHO *&50 DQG WKH FRPPXQLW\ FOLPDWH V\VWHP PRGHO &&60 *&50 LV WKH JOREDO DWPRVSKHULF FLUFXODWLRQ DQG FOLPDWH VLPXODWLRQ DSSOLFDWLRQ GHYHORSHG DW 3DFLILF 1RUWKZHVW 1DWLRQDO /DERUDWRU\ DQG WKH &RORUDGR 6WDWH 8QLYHUVLW\ &&60 LV D FOLPDWH VLPXODWLRQ IUDPHZRUN WKDW FRQVLVWV RI FRPSRQHQW PRGHOV VXFK DV DWPRVSKHUH ODQG RFHDQ DQG VHD LFH GHYHORSHG DW WKH 8QLYHUVLW\ &RUSRUDWLRQ IRU $WPRVSKHULF 5HVHDUFK 8&$5 5HFHQWO\ DQ RSWLPL]DWLRQ LV GHYHORSHG WR HQDEOH GDWD DJJUHJDWLRQ IRU PXOWLSOH VPDOO VL]HG UHTXHVWV WKDW FDQ EHWWHU XWLOL]H , 2 EDQGZLGWK RQ PRGHUQ SDUDOOHO FRPSXWHUV >*/& @ ,Q FROODERUDWLRQ ZLWK 311/ ZH DOVR GHYHORSHG D QHZ , 2 PHWKRG EDVHG RQ WKLV QHZ RSWLPL]DWLRQ LQWR *&50 DQG FRQGXFWHG D SHUIRUPDQFH HYDOXDWLRQ RQ WKH &UD\ ;7 SDUDOOHO FRPSXWHU DW 1(56& $ VLJQLILFDQW SHUIRUPDQFH LPSURYHPHQW LV REVHUYHG RYHU WKH FXUUHQW EHVW , 2 PHWKRG :LWK WKH WRWDO GDWD DPRXQWV RI DQG *% IRU DQG SURFHVVHV UHVSHFWLYHO\ )LJXUH VKRZV XS WR LPSURYHPHQW LQ WHUP RI ZULWH EDQGZLGWKV 7KH ILOH V\VWHP SHDN SHUIRUPDQFH DW WKH WLPH WKHVH H[SHULPHQWV ZHUH FDUULHG RXW KDG *% SHU VHFRQG ZULWH EDQGZLGWK 7KHVH UHVXOWV ZHUH SUHVHQWHG LQ WKH :RUNVKRS RQ +LJK 5HVROXWLRQ &OLPDWH 0RGHOLQJ >36$ @ $ SURWRW\SH RI GDWD DQDO\VLV FDSDELOLWLHV LQ 3QHW&') KDV EHHQ LPSOHPHQWHG :H DUH LPSOHPHQWLQJ WKLV QHZ GDWD DQDO\VLV $3, RQ WRS RI WKH H[LWLQJ 3QHW&') GDWD DFFHVV $3, WR SURYLGH XVHUV ZLWK UDQJH TXHU\ VWDWLVWLFDO DQG RWKHU FRQWHQW EDVHG GDWD DQDO\VLV IXQFWLRQV RQ QHW&') GDWDVHWV ZLWKRXW DQ\ GHWDLOHG XQGHUVWDQGLQJ RI XQGHUO\LQJ GDWD RUJDQL]DWLRQ IURP HQG XVHUV :H FRQWLQXHG RXU FROODERUDWLRQ ZLWK 9 3DVFXFFL 8 8WDK RQ SDUDOOHO , 2 LQ WKHLU ,'; PXOWL UHVROXWLRQ GDWD IRUPDW H[WHQGLQJ WKH VXSSRUW DQG SHUIRUPLQJ D VWXG\ ZLWK - &KHQ 61/ WHDP?V 6 ' FRGH 7KLV GDWD IRUPDW IDFLOLWDWHV LQWHUDFWLYH YLVXDOL]DWLRQ RI YHU\ ODUJH VFDOH GDWD VHWV 7KLV ZRUN LV EHLQJ FRPSOHWHG ZLWK D SRWHQWLDO SDSHU WDUJHW RI &OXVWHU $ SDSHU GHWDLOLQJ HDUO\ UHVXOWV ZDV SUHVHQWHG DW WKH 3DUDOOHO 'DWD 6WRUDJH :RUNVKRS 36': KHOG LQ FRQMXQFWLRQ ZLWK 6& >.39 @ )LJXUH (YDOXDWLRQ RI *&50 , 2 SHUIRUPDQFH ZLWK 3DUDOOHO 1HW&') GDWD DJJUHJDWLRQ 7KH $GDSWDEOH , 2 6\VWHP $',26 2YHU WKH ODVW GHFDGH WKHUH KDV EHHQ D VXEVWDQWLDO LQFUHDVH LQ WKH FDSDELOLW\ RI OHDGHUVKLS FRPSXWLQJ V\VWHPV ZLWK WKH 7RS PDFKLQHV VHHLQJ D PDQ\ IROG LQFUHDVH IURP D IHZ 7)ORSV WR PRUH WKDQ D 3)ORS IRU WKH FXUUHQW JHQHUDWLRQ RI +3& V\VWHPV 7KLV LQFUHDVH LQ FRPSXWH SRZHU KDV QRW EHHQ PLUURUHG LQ , 2 SHUIRUPDQFH ZKLFK LQVWHDG KDV VXIIHUHG E\ FRPSDULVRQ DQG HYHQ GHFUHDVHG LQ DEVROXWH WHUPV 7KLV LPEDODQFH FRXSOHG ZLWK WKH LQFUHDVH LQ GDWD YROXPHV SURGXFHG E\ DSSOLFDWLRQV KDV UHVXOWHG LQ D QHZ ERWWOHQHFN LQ WKH SHUIRUPDQFH RI WKHVH V\VWHPV 7KH $GDSWDEOH , 2 6\VWHP $',26 ZDV GHYHORSHG DV D JUDVV URRWV HIIRUW E\ WKH FRPSXWDWLRQDO VFLHQFH FRPPXQLW\ WR DGGUHVV WKH LVVXH RI PDNLQJ , 2 3HDV\ WR XVH′ DQG DOORZ DSSOLFDWLRQ VFLHQWLVWV WR ZULWH H[WUHPH DPRXQWV RI GDWD 3%?V GD\ ZLWK YHU\ OLWWOH LPSDFW RQ WKHLU FDOFXODWLRQV $',26 UHVHDUFK LV GULYHQ PRVWO\ E\ WKH SHUIRUPDQFH UHTXLUHPHQWV RI DSSOLFDWLRQ VFLHQWLVWV ZKHUH WKH DGRSWLRQ RI QHZ WHFKQRORJLHV LV XVXDOO\ FRQVWUDLQHG GXH WR WKH ODFN RI XVDELOLW\ LQ DFKLHYLQJ WKH GHVLUHG SHUIRUPDQFH LQ DQ HDV\ WR XVH V\VWHP $',26 SURYLGHV DQ HDV\ WR XVH SURJUDPPLQJ LQWHUIDFH ZKLFK LV DV VLPSOH DV )RUWUDQ ILOH , 2 VWDWHPHQWV 7KH FRQFHSWXDO DUFKLWHFWXUH RI $',26 DQG LWV DEVWUDFWLRQ OD\HU LV VKRZQ LQ )LJXUH $EVWUDFWLQJ WKH , 2 PHWDGDWD LQIRUPDWLRQ DQG GDWD VWUXFWXUHV IURP WKH VRXUFH FRGH LQWR DQ H[WHUQDO ;0/ ILOH FDQ UHGXFH FRGH SROOXWLRQ DQG FUHDWH WKH FRQQHFWLRQ EHWZHHQ KLJK OHYHO $3,V DQG WKH XQGHUO\LQJ , 2 LPSOHPHQWDWLRQ GHWDLOV DV ZHOO DV RWKHU WHFKQLFDO GHVFULSWLRQV VXFK DV EXIIHULQJ DQG VFKHGXOLQJ DQG DV\QFKURQRXV GDWD PRYHPHQW %HFDXVH RI WKLV VHSDUDWLRQ GLIIHUHQW UHVHDUFK JURXSV FDQ ZULWH , 2 PHWKRGV LQ $',26 WR DOORZ IRU H[WUHPH SHUIRUPDQFH DV VKRZQ LQ )LJXUH )RU H[DPSOH WKH SUHYLRXV , 2 LPSOHPHQWDWLRQV LQ WKH 6 ' VLPXODWLRQ DQG WKH 6&(& FRGH UXQQLQJ RQ . FRUHV DQG . FRUHV RQ WKH ;7 DW 251/ SURGXFH RYHU ; SHUIRUPDQFH LQFUHDVHV FRPSDUHG WR FROOHFWLYH 03, ,2 $',26 DOVR DOORZV WKH RXWSXW WR EH LQ WKH $',26 %3 IRUPDW DV ZHOO DV +') DQG SDUDOOHO 1HWFGI DQG $6&,, 7KH $',26 %3 IRUPDW ZDV FUHDWHG IRU H[WUHPH VFDOH FRPSXWLQJ DQG DOORZV IRU H[WUHPH SHUIRUPDQFH LQ UHDGLQJ GDWD RQ +3& V\VWHPV ,Q UHFHQW UHVXOWV LW RIWHQ DFKLHYHV ± ; SHUIRUPDQFH LPSURYHPHQWV LQ UHDGLQJ SHUIRUPDQFH 1H[W *HQHUDWLRQ , 2 6RIWZDUH 7HFKQRORJLHV 6RPH FKDOOHQJHV IRU IXWXUH , 2 V\VWHPV FDOO IRU WKH GHYHORSPHQW RI DOWRJHWKHU QHZ VRIWZDUH WHFKQRORJLHV 2QH IRFXV IRU VRIWZDUH GHYHORSPHQW LV RQ WKH FUHDWLRQ RI D FROODERUDWLYH ILOH FDFKLQJ V\VWHP IRU XVH LQ +3& HQYLURQPHQWV 6XFK D V\VWHP ZRXOG WDNH DGYDQWDJH RI VPDOO SRUWLRQV RI PHPRU\ RQ D FROOHFWLRQ RI PDFKLQHV WR JHQHUDWH D FDFKH RI VXIILFLHQW VL]H WR HQDEOH DJJUHJDWLRQ DQG UHRUJDQL]DWLRQ RI , 2 RSHUDWLRQV IURP +3& DSSOLFDWLRQV $ VHFRQG IRFXV RI RXU ZRUN LV LQ LPSURYLQJ WKH DQDO\VLV FDSDELOLWLHV RI +3& V\VWHPV $ SURPLVLQJ WHFKQRORJ\ IRU LPSURYLQJ DQDO\VLV LV DFWLYH VWRUDJH ZKLFK SURYLGHV WKH DELOLW\ WR SHUIRUP GDWD SURFHVVLQJ RQ WKH VWRUDJH QRGHV RI PRGHUQ ILOH V\VWHPV :H ZLOO GLVFXVV RXU VXFFHVVHV LQ ERWK RI WKHVH DUHDV LQ WKLV VHFWLRQ )LJXUH 7KH $',26 DEVWUDFWLRQ OD\HU )LJXUH $',26 0HWKRGV GHPRQVWUDWH RSWLPL]HG SHUIRUPDQFH DFURVV PXOWLSOH DSSOLFDWLRQV $FFRPSOLVKPHQWV $ GLVWULEXWHG FROODERUDWLYH , 2 FDFKLQJ V\VWHP KDV EHHQ LPSOHPHQWHG >/&& /&& E@ $ JOREDO GLVWULEXWHG PHWDGDWD FDFKH PDLQWDLQV VWDWH RQ ZKHUH GDWD LV ORFDWHG LQ WKH FDFKH DQG D OHDVW UHFHQWO\ XVHG /58 SROLF\ LV DGRSWHG IRU FDFKH HYLFWLRQ 7R HQIRUFH WKH DWRPLFLW\ RI DOO ILOH V\VWHP UHDG DQG ZULWH FDOOV ZH DGRSWHG D WZR SKDVH PXOWLSOH UHDGHU VLQJOH ZULWHU ORFNLQJ SROLF\ /RFNV DUH PDQDJHG E\ WKH HQWLW\ UHVSRQVLEOH IRU PHWDGDWD RQ WKDW UHJLRQ ,Q VRPH VSHFLILF FDVHV ZKHUH LW LV VHPDQWLFDOO\ FRUUHFW ZH HQDEOH PXOWLSOH ZULWHUV WR LPSURYH FRQFXUUHQF\ )RU WHVWLQJ DQG HYDOXDWLRQ SXUSRVHV ZH KDYH KRRNHG WKH FDFKLQJ V\VWHP LQWR 520,2 IRU ERWK WKH 8)6 DQG 39)6 GULYHUV HQDEOLQJ WHVWLQJ RQ D ZLGH YDULHW\ RI SDUDOOHO ILOH V\VWHPV 7ZR PHFKDQLVPV IRU FROODERUDWLYH FDFKLQJ KDYH EHHQ LPSOHPHQWHG 7KH ILUVW UHOLHV RQ , 2 WKUHDGV WR GULYH WKH FDFKLQJ V\VWHP ,Q WKLV PRGH DGGLWLRQDO WKUHDGV VSDZQHG RQ FRPSXWH QRGHV DUH UHVSRQVLEOH IRU FDFKLQJ DQG , 2 7KH VHFRQG PHFKDQLVP UHOLHV RQ 03, SURFHVV VSDZQLQJ FDOOV 7KLV DSSURDFK FDQ EH XVHG WR VHW XS D GLVWLQFW VHW RI FDFKLQJ SURFHVVHV FDOOHG 3FDFKH VHUYHUV′ ZKHUH WKLV 03, IXQFWLRQDOLW\ LV DYDLODEOH %DVHG RQ WKH REVHUYDWLRQ WKDW WKH PDMRULW\ RI WKH , 2 SDWWHUQV SUHVHQWHG LQ VFLHQWLILF DSSOLFDWLRQV DUH ZULWH RQO\ DQG GR QRW UHTXLUH DQ\ VRIWZDUH FRQWURO IRU SDUDOOHO GDWD FRQVLVWHQF\ RU FRQFXUUHQW DFFHVV DWRPLFLW\ ZH GHYHORSHG D WZR VWDJH ZULWH EHKLQG PHWKRG IRU LPSURYLQJ WKH SHUIRUPDQFH RI SDUDOOHO ZULWH RQO\ RSHUDWLRQV >/&& D@ 7KLV DSSURDFK UHGLVWULEXWHV DQG FRPELQHV QRQ FRQWLJXRXV ILOH UHTXHVWV EHWZHHQ SURFHVVHV WR JHQHUDWH ODUJH FRQWLJXRXV , 2 UHTXHVWV 6LPLODUO\ ZH KDYH H[SHULPHQWHG ZLWK XVLQJ 326,; DV\QFKURQRXV , 2 URXWLQHV DV D PHFKDQLVP IRU IOXVKLQJ GLUW\ FDFKH GDWD GXULQJ LGOH SHULRGV 7HVWLQJ DQG GHYHORSPHQW ZHUH SHUIRUPHG RQ VHYHUDO SDUDOOHO PDFKLQHV 7XQJVWHQ UXQQLQJ /XVWUH ILOH V\VWHP DQG WKH ,%0 FOXVWHU UXQQLQJ *3)6 ILOH V\VWHP DW 1&6$ -D]] UXQQLQJ 39)6 ILOH V\VWHP DW $1/ DQG (ZRN UXQQLQJ /XVWUH DW 251/ :H XVH VHYHUDO , 2 EHQFKPDUNV IRU SHUIRUPDQFH HYDOXDWLRQ WKH 1$6$ %7,2 EHQFKPDUN WKH )/$6+ DSSOLFDWLRQ , 2 NHUQHO DQG 6 ' DSSOLFDWLRQ , 2 NHUQHO 2XU H[SHULPHQWV VKRZ WKDW ZLWK RI FRPSXWH QRGHV DOORFDWHG DV , 2 GHOHJDWHV , 2 EDQGZLGWK LPSURYHPHQWV UDQJH IURP WR :KHQ WKH QXPEHU RI , 2 GHOHJDWHV LV RI WKH DSSOLFDWLRQ QRGHV ZH REVHUYHG XS WR LPSURYHPHQW >1/& @ 6WDUWLQJ ZLWK DQ LQLWLDO NHUQHO OHYHO DFWLYH VWRUDJH SURWRW\SH >))* @ ZH LGHQWLILHG VHYHUDO IDFWRUV WKDW GHJUDGH SHUIRUPDQFH DQG LQYHVWLJDWHG V\QFKURQL]DWLRQ RSWLRQV EHWZHHQ UHPRWH FOLHQW DQG DFWLYH VWRUDJH SURFHVVLQJ FRPSRQHQW 7R DGGUHVV VRPH RI WKHVH SHUIRUPDQFH LVVXHV ZLWK WKH NHUQHO VSDFH LPSOHPHQWDWLRQ ZH GHYHORSHG D QHZ XVHU VSDFH LPSOHPHQWDWLRQ RI DFWLYH VWRUDJH %DVHG RQ SUHOLPLQDU\ H[SHULPHQWV DQG H[SHULHQFH LW DSSHDUV WR EH IDVWHU DQG PRUH IOH[LEOH WKDQ WKH RULJLQDO YHUVLRQ DQG LW LV DOVR PRUH SRUWDEOH LW RSHUDWHV RQ ERWK /XVWUH DQG 39)6 :H KDYH GHFLGHG WR IRFXV RXU HIIRUWV RQ WKH QHZ XVHU VSDFH DSSURDFK :H KDYH HYDOXDWHG WKLV SURWRW\SH >31 @ IRFXVLQJ RQ LWV XVH LQ RXW RI FRUH DSSOLFDWLRQV :H DUH ZRUNLQJ RQ SHUIRUPDQFH PRGHO IRU DFWLYH VWRUDJH 7KH SXUSRVH RI GHYHORSLQJ VXFK DQ DQDO\WLFDO PRGHO IRU DFWLYH VWRUDJH LV WR HQDEOH SUHGLFWLRQ RI H[SHFWHG SHUIRUPDQFH EHQHILWV EHIRUH GHSOR\LQJ LQ D XVHU DSSOLFDWLRQ 7KH SHUIRUPDQFH PRGHO ZH DUH GHYHORSLQJ LQFOXGHV V\VWHP SDUDPHWHUV QHWZRUN EDQGZLGWK QXPEHU RI VHUYHU DQG FRPSXWH QRGHV GLVN EDQGZLGWK DQG LQIRUPDWLRQ DERXW WKH FRPSXWH QRGHV XVHU SDUDOOHO MRE ZRXOG UXQ RQ 2XU UHFHQW ZRUN KDV IRFXVHG RQ GHYHORSLQJ VXSSRUW IRU VWULSHG ILOHV >31 @ 2XU SUHYLRXV LPSOHPHQWDWLRQV KDYH EHHQ XQDEOH WR GHDO ZLWK VWULSHG ILOHV L H ILOHV ZKRVH GDWD LV VSUHDG DFURVV VHYHUDO QRGHV 6WULSLQJ ILOHV LV W\SLFDOO\ SHUIRUPHG LQ SDUDOOHO ILOH V\VWHPV WR LPSURYH DJJUHJDWH , 2 EDQGZLGWK ,Q WKH FXUUHQW GHVLJQ ZH ODXQFK D SURFHVVLQJ FRPSRQHQW SHU VWRUDJH QRGH XVHG E\ WKH PDWFKLQJ ILOH DQG PDNH HYHU\ SURFHVVLQJ FRPSRQHQW SURFHVV RQO\ WKH ILOH FKXQNV VWRUHG LQ LWV RZQ QRGH ,I WKH SURFHVVLQJ FRPSRQHQWV ZULWH WR DQ RXWSXW ILOH WKH RXWSXW ILOH ZLOO KDYH WR EH FUHDWHG ZLWK WKH VDPH VWULSLQJ SDWWHUQ DV WKH LQSXW ILOH DQG HYHU\ SURFHVVLQJ FRPSRQHQW ZLOO KDYH WR ZULWH WR RQO\ WKH ILOH FKXQNV VWRUHG LQ LWV QRGH 7KH H[SHULPHQWDO UHVXOWV RQ D /XVWUH ILOH V\VWHP VKRZ WKDW RXU LPSOHPHQWDWLRQ IRU VWULSHG ILOHV FDQ UHGXFH WKH QHWZRUN WUDIILF WR QHDU ]HUR ,Q DGGLWLRQ ZH KDYH EHHQ ZRUNLQJ RQ DGRSWLQJ $FWLYH 6WRUDJH WR GHDO ZLWK GDWD ILOHV ZLWK VSHFLILF FRPSOH[ IRUPDWV VXFK DV QHW&') RU +') WKDW DUH YHU\ FRPPRQ IRU GDWD H[FKDQJH LQ VRPH VFLHQWLILF DSSOLFDWLRQV ,Q SDUWLFXODU LQ FROODERUDWLRQ ZLWK WKH 6FL'$& *&50 JOREDO FORXG UHVROYLQJ PRGHO SURMHFW ZH KDYH EHHQ ZRUNLQJ ZLWK QHW&') FOLPDWH GDWD 2XWUHDFK :H WDNH RXWUHDFK YHU\ VHULRXVO\ :H KDYH SUHVHQWHG WXWRULDOV RQ WRSLFV UHODWHG WR VWRUDJH DQG SDUDOOHO , 2 LQ WKH ODVW WKUHH \HDUV LQFOXGLQJ VL[ IXOO GD\ WXWRULDOV DW WKH 6& FRQIHUHQFH VHULHV :H RUJDQL]HG DQG KHOG D V\PSRVLXP DW $1/ IRU 39)6 UHVHDUFKHUV DQG GHYHORSHUV KHOSLQJ HYHU\RQH FDWFK XS ZLWK HDFK RWKHU?V ZRUN DQG FRRUGLQDWH IXWXUH HIIRUWV :H DFWLYHO\ SDUWLFLSDWH LQ '2( ([DVFDOH ZRUNVKRSV DQG RWKHU DSSOLFDWLRQ RULHQWHG PHHWLQJV WR KHOS HGXFDWH WKH FRPPXQLW\ RQ , 2 EHVW SUDFWLFHV :H KDYH ZRUNHG FORVHO\ ZLWK *DUWK *LEVRQ RI &DUQHJLH 0HOORQ 8QLYHUVLW\ DQG WKH 6FL'$& 3HWDVFDOH 'DWD 6WRUDJH ,QVWLWXWH WR KHOS LQ XVLQJ 39)6 DV WKH IRXQGDWLRQ IRU FODVV SURMHFWV LQ SDUDOOHO ILOH V\VWHPV 6R IDU 39)6 KDV EHHQ XVHG LQ WZR FRXUVHV DV WKH EDVLV IRU ZRUN LQ GLVWULEXWHG GLUHFWRU\ VWRUDJH LQ KLJK SHUIRUPDQFH ILOH V\VWHPV DQG LQ DOWHUQDWLYH GDWD RUJDQL]DWLRQV RQ ORFDO VWRUDJH %RWK HIIRUWV KDYH UHVXOWHG LQ VWXGHQW SXEOLFDWLRQV >3*/ 367 @ :H SUHVHQWHG WKH NH\QRWH WDON 36WRUDJH LQ DQ ([DVFDOH :RUOG ′ DW 61$3, DQG SUHVHQWHG 30DNLQJ WKH 0RVW RI WKH 6RIWZDUH 6WDFN′ DW WKH 7$&& ([WUHPH 6FDOH , 2 DQG 'DWD $QDO\VLV :RUNVKRS :H DOVR VWDUWHG WKH ,QWHUIDFHV DQG $EVWUDFWLRQV IRU 6FLHQWLILF 'DWD 6WRUDJH ZRUNVKRS VHULHV LQ ZLWK WKH WKLUG ZRUNVKRS WR EH KHOG ODWHU WKLV \HDU 2. Scientific Data Mining and Analysis (DMA) 7KH 'DWD 0LQLQJ DQG $QDO\VLV '0$ OD\HU SURYLGHV WKH GDWD XQGHUVWDQGLQJ WHFKQRORJLHV QHFHVVDU\ IRU HIILFLHQW DQG HIIHFWLYH DQDO\WLFV RI FRPSOH[ VFLHQWLILF GDWD 7KLV LV DFFRPSOLVKHG WKURXJK WKH GHYHORSPHQW DQG GHSOR\PHQW RI WKH WKUHH FRUH WHFKQRORJLHV x+LJK SHUIRUPDQFH SDUDOOHO VWDWLVWLFDO FRPSXWLQJ x(IILFLHQW VHDUFKLQJ DQG ILOWHULQJ LQ GDWD LQWHQVLYH VFLHQWLILF DSSOLFDWLRQV x)HDWXUH H[WUDFWLRQ DQG WUDFNLQJ IRU VFLHQWLILF DSSOLFDWLRQV +LJK SHUIRUPDQFH SDUDOOHO VWDWLVWLFDO FRPSXWLQJ 2XU RYHUDUFKLQJ JRDO ZDV WR VFDOH XS WKH H[LVWLQJ GDWD DQDO\VLV WRROV ERWK LQ FDSDELOLW\ DQG FDSDFLW\ WR EULGJH WKH JDS EHWZHHQ VFLHQWLILF GDWD SURGXFWLRQ DQG GDWD SURFHVVLQJ UDWHV2D ZHOO UHFRJQL]HG QHHG E\ WKH '2( VFLHQWLILF DSSOLFDWLRQV >0RX @ 7KH FKDOOHQJH LV WKDW many of the current data analysis routines are written in non-parallel languages such as R, Matlab, and IDL and do not scale to massive data sets 7R DGGUHVV WKLV FKDOOHQJH RXU DSSURDFK ZDV EDVHG RQ XVLQJ D VWDWLVWLFDO SDFNDJH FDOOHG R>,* @ :H SURYLGHG D PLGGOHZDUH EHWZHHQ 5?V KLJK OHYHO GDWD DQDO\WLFV ODQJXDJH DQG D JHQHULF SDUDOOHO RSWLPL]HG SRUWDEOH FRPSXWDWLRQDO DQDO\WLFV HQJLQH :H IXUWKHU XVHG WKRVH WRROV WR SURYLGH DSSOLFDWLRQ VFLHQWLVWV ZLWK WKH WDUJHWHG NQRZOHGJH GLVFRYHU\ FDSDELOLWLHV IRU FRPSOH[ VFLHQWLILF GDWD VXFK DV x Analysis of fluctuation energy distribution±WKH REVHUYHG FRPSRVLWH HQHUJ\ VLJQDO LV GLVWULEXWHG LQ VSDFH DQG WLPH :H IRFXVHG RQ GLVFRYHU\ RI WXUEXOHQW SDWWHUQV LQ WKH dphi2 ;*& HQHUJ\ GDWD ZKHUH dphi2 LV WKH VTXDUH RI HOHFWULFDO SRWHQWLDO IOXFWXDWLRQ E\ WXUEXOHQFH x Mapping and tracking the evolution of turbulence in space and time±ZH GHYHORSHG D PHWKRGRORJ\ IRU multi-resolution DFURVV VSDFH DQG WLPH DQDO\VLV RI WXUEXOHQFH HVSHFLDOO\ WKURXJK IURQW WUDFNLQJ WR HVWDEOLVK VXFK G\QDPLFV $FFRPSOLVKPHQWV Parallel R (pR) for high performance statistical computing delivered super-linear scaling 7KH JRDO RI WKLV DFWLYLW\ ZDV WR SURYLGH DQ HDVLO\ H[WHQVLEOH PHFKDQLVP WR DGG WKLUG SDUW\ SDUDOOHO DQDO\VLV FDSDELOLWLHV WR R :H DGYDQFHG RXU SURWRW\SH Parallel R (pR) V\VWHP RQ VHYHUDO SHUIRUPDQFH PHWULFV 7KH RScaLAPACK OLEUDU\ ><6% @ ZDV PDWXUHG WR D SURGXFWLRQ OHYHO DQG GLVWULEXWHG DV DQ 530 SDFNDJH RI YDULRXV /LQX[ GLVWULEXWLRQV >530 @ DV ZHOO DFURVV ! PLUURU VLWHV WKURXJK 5?V &5$1 QHWZRUN ><6% @ RScaLAPACK UHSODFHG PDQ\ GDWD DQDO\VLV URXWLQHV LQ R WKDW DUH EDVHG RQ /$3$&. >$%' @ OLQHDU DOJHEUD VROYHUV ZLWK WKH FRUUHVSRQGLQJ SDUDOOHO RSWLPL]HG DQG SRUWDEOH 6FD/$3$&. >%&& @ URXWLQHV :H H[WHQGHG RScaLAPACK WR SURYLGH D OLJKWZHLJKW HDV\ WR XVH pR PLGGOHZDUH LQWHUIDFH >6$& 326 @ WKDW EULGJHG WKH 5 VWDWLVWLFDO HQYLURQPHQW ZLWK SDUDOOHO WKLUG SDUW\ GDWD PLQLQJ DQG DQDO\VLV OLEUDULHV ZULWWHQ LQ FRPSLOHG ODQJXDJHV VXFK DV 03, & & )RUWUDQ 6LPLODU WR 56FD/$3$&. WKH pR LQWHUIDFH WR SDUDOOHO DQDO\VLV IXQFWLRQV PLPLFV WKH VHULDO IXQFWLRQV LQWHUIDFHV ZKLFK LV D VLQJOH IXQFWLRQ FDOO ,Q FRQWUDVW WR RScaLAPACK ZKLFK ZDV GHVLJQHG VSHFLILFDOO\ IRU ScaLAPACK WKH pR DUFKLWHFWXUH ZDV DEVWUDFWHG WR DOORZ SDUDOOHO WKLUG SDUW\ DQDO\VLV IXQFWLRQV WR LQWHJUDWH ZLWK WKH R HQYLURQPHQW ZLWKRXW UHTXLULQJ PDMRU PRGLILFDWLRQV WR HLWKHU pR RU WR WKH H[WHUQDO WKLUG SDUW\ OLEUDULHV ,Q DGGLWLRQ pR RIIHUHG WKH HIILFLHQF\ DQG VFDODELOLW\ RI WKH XQGHUO\LQJ SDUDOOHO WKLUG SDUW\ DQDO\VLV FRGHV ZLWK D IHZ PLFURVHFRQGV RYHUKHDG LQGXFHG E\ pR PLGGOHZDUH ,Q FRQWUDVW WR R LWV SHUIRUPDQFH RQ D VLQJOH SURFHVVRU KDV EHHQ LPSURYHG E\ D IDFWRU RI DQG LW KDV VKRZQ D VXSHU OLQHDU VFDODELOLW\ FRPSDUHG WR D K\SRWKHWLFDO LGHDO VFDOLQJ ZLWK WKH QXPEHU RI SURFHVVRUV LI R ZHUH UXQ LQ SDUDOOHO 8QOLNH FRPSHWLQJ WHFKQRORJLHV VXFK DV Rmpi > :H DOVR H[WHQGHG WKH FDSDELOLW\ RI 56FD/$3$&. OLEUDU\ WR VXSSRUW RSHQ03, EDFN HQG LQ UHVSRQVH WR PXOWLSOH XVHUV? UHTXHVWV HDVHG 56FD/$3$&.?V LQVWDOODWLRQ YLD LPSURYHG DXWRFRQI SURYLGHG SURFHVVRU JULG PDQLSXODWLRQ URXWLQHV SURYLGHG ERWK VWDWLF DQG G\QDPLF 03, OLEUDU\ VXSSRUW HWF 7KH XSGDWHG YHUVLRQ ZDV UHOHDVHG RQ WKH 5?V &5$1 ZHE VLWH ,Q DGGLWLRQ WR GDWD SDUDOOHOLVP HQDEOHG E\ pR ZH H[SORUHG WDVN SDUDOOHOLVP VXSSRUW LQ 5 %\ OHYHUDJLQJ RXU taskPR >6%< @ ZH SXUVXHG WZR FRPSOHPHQWDU\ GLUHFWLRQV D HPEDUUDVVLQJO\ SDUDOOHO H[HFXWLRQ PRGH DQG E DXWRPDWLF RXW RI RUGHU H[HFXWLRQ PRGH E\ DSSO\LQJ FRPSLOHU SDUDOOHOL]DWLRQ DSSURDFKHV WR DXWRPDWLFDOO\ SDUDOOHOL]H R VFULSWLQJ FRGHV ZLWKRXW UHTXLULQJ HQG XVHU PRGLILFDWLRQV >0/6 /0< /0< @ :H DOVR PDGH SDUDOOHO DQDO\VLV FDSDELOLWLHV DYDLODEOH DV SDUW RI PRUH FRPSOH[ VFLHQWLILF ZRUNIORZV VHH 6HFWLRQ :H SXUVXHG WKLV HIIRUW DORQJ WKH WKUHH FRPSOHPHQWDU\ GLUHFWLRQV D WR HQDEOH DQDO\VLV FDSDELOLWLHV ZLWK WKH 'DKVERDUG :HE DSSOLFDWLRQ VHH 6HFWLRQ E WR SURYLGH ZHE VHUYLFHV EDVHG SDUDOOHO VWDWLVWLFDO FRPSXWLQJ FDSDELOLW\ ZLWKLQ .HSOHU ZRUNIORZ HQJLQH DQG F WR HQDEOH VRFLDO QHWZRUN EDVHG NQRZOHGJH DQQRWDWLRQ >%*.6 @ 7KHVH HIIRUWV ZHUH LQ FROODERUDWLRQ ZLWK WKH 63$ WHDP /HYHUDJLQJ pR?V DQDO\VLV URXWLQHV WKH 'DVKERDUG ZHE DSSOLFDWLRQ VHH 6HFWLRQ ZDV XSJUDGHG WR LQFRUSRUDWH D GDWD DQDO\VLV OD\HU :H HYDOXDWHG WKH DQDO\VLV IXQFWLRQV XVDJH E\ 6FL'$& VFLHQWLVWV LQFOXGLQJ IXVLRQ DQG FOLPDWH VFLHQWLVWV DQG GHVLJQHG D OLEUDU\ RI DQDO\WLFDO URXWLQHV WR H[SRVH RQ WKH 'DVKERDUG :HE 6HUYLFHV KDYH EHFRPH D FULWLFDO SDUW RI WRGD\ VFLHQWLILF UHVHDUFK GRPDLQ 7KH\ VXSSRUW ORRVHO\ FRXSOHG VHUYLFH RULHQWHG DUFKLWHFWXUH WR SURYLGH VFDODEOH LQWHURSHUDEOH V\VWHPV 6LQFH .HSOHU KDV EHHQ D YHU\ HIIHFWLYH WRRO IRU PDQDJLQJ VFLHQWLILF ZRUNIORZV ZH H[WHQGHG GDWD DQDO\VLV FDSDELOLWLHV RI .HSOHU WR ZHE VHUYLFHV HQDEOHG GDWD DQDO\VLV 7KLV QHZ IHDWXUH RI DFFHVVLQJ GDWD DQDO\WLFV URXWLQHV WKURXJK D ZHE VHUYLFH HQDEOHG XVHUV WR EHQHILW IURP WKH IXQFWLRQDOLW\ DQG KLJK SHUIRUPDQFH RI SDUDOOHO R DV SDUW RI D PRUH FRPSOH[ VFLHQWLILF ZRUNIORZ :H FR WDXJKW WKH 63$?V 6XSHUFRPSXWLQJ WXWRULDO DQG SUHVHQWHG D PRGXOH RQ WKH UHPRWH R HQDEOHG .HSOHU DFWRU $ WKLUG DFWLYLW\ ZDV WR GHSOR\ WKLV SDUDOOHO DQDO\VLV LQIUDVWUXFWXUH DFURVV YDULRXV '2( VFLHQWLILF DSSOLFDWLRQV :RUNLQJ ZLWK D QXPEHU RI 6FL'$& DSSOLFDWLRQ VFLHQWLVWV ZH REVHUYHG WKDW RQH RI WKH NH\ REVWDFOHV LV WKH ODFN RI LQIUDVWUXFWXUH WKDW FRXSOHV GDWD DQDO\WLFV FDSDELOLWLHV ZLWK GDWD PDQDJHPHQW LQIUDVWUXFWXUH )RU H[DPSOH WKH JURZLQJ DPRXQW RI FOLPDWH GDWD LV FXUUHQWO\ PDGH DYDLODEOH IRU GRZQORDG WKURXJK WKH '2( (DUWK 6FLHQFH *ULG (6* SRUWDO \HW WKH DQDO\VLV RI WKLV GDWD LV W\SLFDOO\ GRQH RQ HDFK LQVWLWXWLRQDO LQIUDVWUXFWXUH 0RUHRYHU WKH FXUUHQW (6* SRUWDO SURYLGHV D FRQYHQLHQW LQWHUIDFH IRU XVHUV WR GRZQORDG VPDOO VXEVHWV RI GDWD EXW LV QRW GHVLJQHG IRU ODUJH VFDOH GDWD WUDQVIHUV :H DGGUHVVHG WKLV QHHG WKURXJK WKH (6* 'RZQORDG 3URMHFW ('3 ('3 SURYLGHV D FRQYHQLHQW PHFKDQLVP IRU (6* FXVWRPHUV WR JHQHUDWH UHVXOWV OHYHUDJLQJ PDVVLYH DPRXQWV RI SUHYLRXVO\ LQDFFHVVLEOH GDWD :H ZRUNHG ZLWK 0DUFLD %UDQVWHWWHU 251/ IURP WKH '2( &OLPDWH 0RGHOLQJ 6FL'$& DQG WKH (6* JURXS DQG DSSOLHG WKH 'DWD 0RYHU /LWH '0/ WRRO XVHG LQ (6* 7KH 6'0 FHQWHU SURGXFHG D VLPSOH FRPPDQG OLQH LQWHUIDFH WKDW LQYRNHG WKH XVH RI WKH (6* SRUWDO PHWD GDWDEDVH DQG WKH '0/ WRRO WR HQDEOH ODUJH VFDOH GDWD WUDQVIHUV RI WKH FOLPDWH VLPXODWLRQ GDWD IURP WKH (DUWK 6\VWHP *ULG (6* SRUWDO WR 251/ V\VWHPV Automatic Spatio-Temporal Turbulent Front Detection and Evolution in Fusion Plasma )HZ ZRXOG DUJXH WKDW IXVLRQ HQHUJ\ KDV EHHQ WKH +RO\ *UDLO RI UHQHZDEOH HQHUJ\ HIIRUWV 7KH JUDQG FKDOOHQJH LV WR SURGXFH PRUH HQHUJ\ WKURXJK D IXVLRQ UHDFWLRQ WKDQ WKDW UHTXLUHG WR LQLWLDWH WKH SURFHVV LQ D UHDFWRU $ NH\ ERWWOHQHFN LV WKH WXUEXOHQFH RU XQVWDEOH PRWLRQ RI WKH IXVLRQ SODVPD 7XUEXOHQFH LQIOXHQFHV WKH GHJUHH RI HQHUJ\ ORVW E\ SODVPD GXULQJ WKH IXVLRQ SURFHVV WKHUHIRUH FRQWUROOLQJ WKH WXUEXOHQFH LV FULWLFDO WR YLDEOH HQHUJ\ SURGXFWLRQ 'LVFRYHU\ RI G\QDPLF WXUEXOHQW SDWWHUQV DQG WUHQGV IURP WKH GDWD SURGXFHG E\ D FRPSXWHU VLPXODWHG IXVLRQ UHDFWLRQ RIIHUV D SRWHQWLDO WR UHYHDO ZD\V WR FRQWURO WKH WXUEXOHQFH 2QH VWUDWHJ\ ZDV WKH RQH RI reduced, yet informative, data representation IRU WKH WDUJHW GDWD DQDO\VLV WDVN )RU D IL[HG WLPH VWHS ZH DSSUR[LPDWHG ZLWK OLQH VHJPHQWV LQ D VSDWLDO UHJLRQ DURXQG WKH SRLQW RI LQWHUHVW 7KH SRLQWV FRUUHVSRQGLQJ WR WKH IURQWV DUH WKH SRLQWV ZKHUH WKH OLQH VHJPHQWV FKDQJH WKHLU VORSH IURP WKH GLUHFWLRQ DOPRVW SDUDOOHO WR WKH x D[LV JUHHQ WR WKH GLUHFWLRQ DOPRVW SDUDOOHO WR WKH y D[LV EOXH DQG UHG VHH )LJ D 6XFK DQ DSSURDFK RQO\ UHTXLUHG WKH VORSH DQG LQWHUFHSW RI WKH DSSUR[LPDWLQJ OLQH VHJPHQWV IRU D IHZ VHTXHQWLDO VOLGLQJ ZLQGRZV 7KH RWKHU VWHSV RI WKH HQG WR HQG IURQW GHWHFWLRQ DQG WUDFNLQJ SURFHVV VHH (UURU 5HIHUHQFH VRXUFH QRW IRXQG E KDYH EHHQ ORFDO E\ QDWXUH DQG KDYH XWLOL]HG pRapply PHWKRG IRU D PXOWL QRGH PXOWL FRUH SDUDOOHO H[HFXWLRQ ZLWK DQ LGHDO VSHHG XS DV GHVFULEHG QH[W t r Fig. 1. Front detection and tracking in fusion simulation data. Fig. 2. A multi-step turbulent front detection and tracking process Automatic Parallelization of Data-Parallel Statistical Computing Codes with pRapply in Hybrid Multi-Node Multi-Core HPC Environments :H H[WHQGHG WKH pR PLGGOHZDUH >%+ @ ZLWK pRapply IXQFWLRQ IRU WKH R RSHQ VRXUFH VWDWLVWLFDO HQYLURQPHQW WR VXSSRUW DXWRPDWLF SDUDOOHOL]DWLRQ RI GDWD SDUDOOHO WDVNV LQ PXOWL QRGH PXOWL FRUH DQG K\EULG HQYLURQPHQWV >%. 6.% @ pR UHTXLUHV IHZ RU QR FKDQJHV WR H[LVWLQJ VHULDO FRGHV RIIHUV D OLQHDU VSHHG XS ZLWK WKH LQFUHDVLQJ QXPEHU RI SURFHVVRUV DQG \LHOGV RYHU HQG WR HQG H[HFXWLRQ WLPH LPSURYHPHQWV LQ RXU WHVWV FRPSDUHG WR WKH FRPPRQO\ XVHG snow R SDFNDJH :H UHOHDVHG pRapply DV RSHQ VRXUFH VRIWZDUH Fast, Incremental, and Scalable All Pairs Similarity Search 6HDUFKLQJ SDLUV RI VLPLODU GDWD UHFRUGV LV DQ RSHUDWLRQ UHTXLUHG IRU PDQ\ GDWD PLQLQJ WHFKQLTXHV OLNH FOXVWHULQJ DQG FROODERUDWLYH ILOWHULQJ $V WKH VFDOH RI WKH GDWD KDV EHHQ LQFUHDVLQJ WR VHYHUDO PLOOLRQV RU ELOOLRQV RI UHFRUGV LQ D KLJK GLPHQVLRQDO VSDFH HQDEOLQJ IDVW DQG LQFUHPHQWDO VLPLODULW\ VHDUFK RYHU VXFK GDWD VHWV KDV EHFRPH D IRUPLGDEOH WDVN 7R DGGUHVV WKLV FKDOOHQJH ZH GHYHORSHG DQ RSHQ VRXUFH OLEUDU\ RI DOJRULWKPV IRU IDVW LQFUHPHQWDO DQG VFDODEOH DOO SDLUV VLPLODULW\ VHDUFKHV WKURXJK LPSURYHG LQGH[LQJ V\VWHPDWLF KHXULVWLF RSWLPL]DWLRQV DQG SDUDOOHOL]DWLRQ )LUVW ZH GHVLJQHG D VHTXHQWLDO DOJRULWKP IRU DOO SDLUV VLPLODULW\ VHDUFK $366 WKDW LQYROYHV ILQGLQJ DOO SDLUV RI UHFRUGV KDYLQJ VLPLODULW\ DERYH D VSHFLILHG WKUHVKROG 2XU SURSRVHG IDVW PDWFKLQJ WHFKQLTXH VSHHGV XS $366 FRPSXWDWLRQ E\ XVLQJ QRYHO WLJKWHU ERXQGV IRU VLPLODULW\ FRPSXWDWLRQ DQG LQGH[LQJ GDWD VWUXFWXUH ,W RIIHUV WKH IDVWHVW VROXWLRQ NQRZQ WR GDWH ZLWK XS WR ; VSHHG XS RYHU WKH VWDWH RI WKH DUW H[LVWLQJ $366 DOJRULWKP :H IXUWKHU DGGUHVVHG WKH LQFUHPHQWDO IRUPXODWLRQ RI $366 SUREOHP ZKHUH $366 LV SHUIRUPHG PXOWLSOH WLPHV RYHU D JLYHQ GDWD VHW ZKLOH YDU\LQJ WKH VLPLODULW\ WKUHVKROG 7KH JRDO LV WR DYRLG UHGXQGDQW FRPSXWDWLRQV DFURVV PXOWLSOH LQYRFDWLRQV RI $366 E\ VWRULQJ KLVWRU\ RI FRPSXWDWLRQ GXULQJ HDFK $366 'HSHQGLQJ RQ WKH VLPLODULW\ WKUHVKROG YDULDWLRQ RXU SURSRVHG KLVWRU\ ELQQLQJ DQG LQGH[ VSOLWWLQJ WHFKQLTXHV DFKLHYH VSHHG XSV IURP ; WR RYHU ; RYHU WKH VWDWH RI WKH DUW $366 DOJRULWKP 7R WKH EHVW RI RXU NQRZOHGJH WKLV LV WKH ILUVW ZRUN WKDW DGGUHVVHV WKLV SUREOHP )LQDOO\ ZH GHVLJQHG VFDODEOH SDUDOOHO DOJRULWKPV IRU $366 WKDW WDNH DGYDQWDJH RI PRGHUQ PXOWL SURFHVVRU PXOWL FRUH DUFKLWHFWXUHV WR IXUWKHU VFDOH XS WKH $366 FRPSXWDWLRQ 2XU SURSRVHG LQGH[ VKDULQJ WHFKQLTXH GLYLGHV WKH $366 FRPSXWDWLRQ LQWR LQGHSHQGHQW WDVNV DQG DFKLHYHV LGHDO VWURQJ VFDOLQJ EHKDYLRU RQ VKDUHG PHPRU\ DUFKLWHFWXUHV :H DOVR SURSRVH D FRPSOHPHQWDU\ LQFUHPHQWDO LQGH[ VKDULQJ WHFKQLTXH ZKLFK SURYLGHV D PHPRU\ HIILFLHQW SDUDOOHO $366 VROXWLRQ ZKLOH PDLQWDLQLQJ DOPRVW OLQHDU VSHHG XS 3HUIRUPDQFH RI RXU SDUDOOHO $366 DOJRULWKPV UHPDLQV FRQVLVWHQW IRU GDWDVHWV RI YDULRXV VL]HV 7R WKH EHVW RI RXU NQRZOHGJH WKLV LV WKH ILUVW ZRUN WKDW H[SORUHV SDUDOOHOL]DWLRQ IRU $366 :H GHPRQVWUDWH WKH HIIHFWLYHQHVV RI RXU WHFKQLTXHV XVLQJ UHDO ZRUOG PLOOLRQ UHFRUG GDWD VHWV (IILFLHQW VHDUFKLQJ DQG ILOWHULQJ LQ GDWD LQWHQVLYH VFLHQWLILF DSSOLFDWLRQV ,Q WKH '2( GDWD PDQDJHPHQW ZRUNVKRS UHSRUW >0RX @ PRVW DSSOLFDWLRQV DUWLFXODWHG FULWLFDO QHHGV WR HIILFLHQWO\ ILQG LPSRUWDQW GDWD EDVHG RQ VHDUFK FRQGLWLRQV RYHU GDWD YDOXHV 7KLV VHFWLRQ DGGUHVVHV WKH WHFKQRORJ\ IRU HIILFLHQW VHDUFKLQJ DQG ILOWHULQJ RI ODUJH VFDOH VFLHQWLILF PXOWLYDULDWH GDWDVHWV ZLWK KXQGUHGV RI VHDUFKDEOH DWWULEXWHV WR GHOLYHU WKH PRVW UHOHYDQW GDWD WR WKH DSSURSULDWH DQDO\VLV WRROV VXFK DV WKRVH LQ 6HFWLRQV 0RUH VSHFLILFDOO\ RXU JRDO LV WR GHYHORS D VFDODEOH VHDUFKLQJ WRRO IRU VFLHQWLILF GDWD DQG WR LQWHJUDWH WKLV WRRO ZLWK GDWD DQDO\VLV DQG GDWD VWRUDJH HQYLURQPHQWV $FFRPSOLVKPHQWV :H KDYH GHYHORSHG DQ H[WUHPHO\ HIILFLHQW LQGH[LQJ WHFKQRORJ\ IRU DFFHOHUDWLQJ GDWDEDVH TXHULHV RQ PDVVLYH GDWDVHWV FDOOHG )DVW%LW 7KH)DVW%LW LQGH[LQJ VRIWZDUH JUHZ RXW RI \HDUV RI EDVLF UHVHDUFK LQ VFLHQWLILF GDWD PDQDJHPHQW 7KH FRUH RI WKH VRIWZDUH LV D VHW RI QRYHO ELWPDS LQGH[HV WKDW DUH RSWLPL]HG IRU VFLHQWLILF GDWD )RU WKH PDMRULW\ RI VFLHQWLILF GDWD WKH H[LVWLQJ GDWD LV QRW PRGLILHG FastBit LQGH[HV WDNH DGYDQWDJH RI WKLV IDFW DQG RSWLPL]H ELWPDS LQGH[HV XVLQJ D FRPELQDWLRQ RI WHFKQLTXHV GHVFULEHG EULHIO\ EHORZ :H KDYH VKRZQ WKDW FastBit LV WKHRUHWLFDOO\ RSWLPDO DQG FDQ DQVZHU TXHULHV ± WLPHV IDVWHU WKDQ WKH PRVW SRSXODU FRPPHUFLDO LQGH[LQJ PHWKRGV 7KHUH DUH WKUHH VHWV RI RUWKRJRQDO WHFKQLTXHV LQ FastBit ELQQLQJ HQFRGLQJ DQG FRPSUHVVLRQ 7KH ELQQLQJ WHFKQLTXHV DOORZ XV WR ZRUN ZLWK VFLHQWLILF GDWD WKDW FRQWDLQV D YHU\ ODUJH QXPEHU RI GLIIHUHQW YDOXHV 3HUIRUPDQFHV RI RWKHU ELWPDS LQGH[HV GHWHULRUDWH VLJQLILFDQWO\ DV WKH QXPEHU RI GLVWLQFW YDOXHV LQFUHDVHV ZH KDYH GHYHORSHG D ELQ EDVHG FOXVWHULQJ WHFKQLTXH WR RYHUFRPH WKLV VKRUWFRPLQJ >:66 @ 7KLV DOORZHG XV WR GHDO ZLWK VFLHQWLILF GDWD YDOXHV WKDW DUH W\SLFDOO\ LQWHJHUV RU IORDWLQJ SRLQW QXPEHUV $ VLJQLILFDQW SDUW RI WKH SHUIRUPDQFH DGYDQWDJH VWHPV IURP D QHZ FRPSUHVVLRQ WHFKQLTXH ZH KDYH LQYHQWHG DQG SDWHQWHG FDOOHG :RUG $OLJQHG +\EULG :$+ FRPSUHVVLRQ WKDW SURYLGHV D IDFWRU RI ; SHUIRUPDQFH LPSURYHPHQW RYHU DQ\ NQRZQ ELWPDS LQGH[LQJ PHWKRG )XUWKHU LPSURYHPHQWV ZHUH DFKLHYHG E\ D QXPEHU RI DGYDQFHG ELWPDS HQFRGLQJ WHFKQLTXHV WKDW GHWHUPLQH KRZ WKH GDWD LV UHSUHVHQWHG LQ WKH ELWPDSV :H KDYH H[SORUHG DQG GHYHORSHG D FRPELQDWLRQ RI WZR OHYHO HQFRGLQJ PHWKRGV WKDW OHG WR DQ DGGLWLRQDO ILYH IROG LPSURYHPHQW >:66 @ 7KH FRPELQHG HIIHFW RI WKHVH PHWKRGV KDV SURGXFHG XQSUHFHGHQWHG VSHHGXS LQ VHDUFKLQJ YHU\ ODUJH GDWDVHWV SHUPLWWLQJ FHUWDLQ DSSOLFDWLRQV VXFK DV YLVXDO DQDO\WLFV WR SHUIRUP VHDUFKHV LQWHUDFWLYHO\ LQ UHDO WLPH :H KDYH FRPSDUHG )DVW%LW ZLWK RWKHU GHVLJQ FKRLFHV RI LPSOHPHQWLQJ ELWPDSV DQG KDYH VKRZQ LWV VXSHULRULW\ WR RWKHU PHWKRGV >22: @ ,Q WKH )DVW%LW VRIWZDUH ZDV UHFRJQL]HG ZLWK DQ 5 ' $ZDUG IRU LWV LQQRYDWLYH WHFKQRORJLHV DQG LWV FRQWULEXWLRQ WR WKH ZLGHU FRPPXQLW\ )DVW%LW KDV EHHQ HQKDQFHG VR WKDW YDULRXV RSHUDWLRQV LPSRUWDQW WR H[SORUDWLRQ RI VFLHQWLILF GDWD FDQ EH SHUIRUPHG HIILFLHQWO\ RQ WKH ELWPDSV GLUHFWO\ ,Q DGGLWLRQ WR FHOO LGHQWLILFDWLRQ EDVHG RQ VSHFLILHG FRQGLWLRQV RQ WKH YDULDEOHV )DVW%LW FDQ SHUIRUP 5HJLRQ *URZLQJ L H FRQQHFWLQJ QHLJKERULQJ FHOOV LQWR UHJLRQV DQG 5HJLRQ 7UDFNLQJ L H WUDFNLQJ WKH HYROXWLRQ RI UHJLRQV RYHU WLPH $Q H[DPSOH RI DSSO\LQJ WKLV WHFKQLTXH WR FRPEXVWLRQ GDWD LV VKRZQ LQ )LJXUH 7KH WRS OHIW RI WKH ILJXUH VKRZV WKH SURJUHVV RI 3IODPH IURQWV′ LQ D FRPEXVWLRQ VLPXODWLRQ )LJXUH 8VLQJ )DVW%LW LQGH[HV WR LGHQWLI\ IODPH IURQWV LQ D FRPEXVWLRQ VLPXODWLRQ %\ DSSO\LQJ &HOO ,GHQWLILFDWLRQ FRQGLWLRQV WKDW LGHQWLI\ WKH IODPH IURQW UHJLRQ JURZLQJ DQG ILQGLQJ RYHUODSV RI UHJLRQV WR LGHQWLI\ UHJLRQ JURZLQJ LW ZDV SRVVLEOH WR FRQVWUXFW TXLFNO\ WKH JUDSK EHORZ 7KLV JUDSK VKRZV KRZ FRPEXVWLRQ NHUQHOV \HOORZ GRWV RQ WKH WRS FRPELQH RYHU WLPH JRLQJ GRZQ LQ WKH GLDJUDP WR IRUP IODPH IURQWV FastBit KDV EHHQ XVHG LQ D QXPEHU RI DSSOLFDWLRQV 7KH UHOHDVH RI )DVW%LW LQ DV RSHQ VRXUFH DQG WKH QXUWXULQJ DQG VXSSRUW RI XVHUV? JURXS KDV IDFLOLWDWHG RXWUHDFK WR YDULRXV DSSOLFDWLRQ GRPDLQV :H GHVFULEH EHORZ D FRXSOH RI UHSUHVHQWDWLYH DSSOLFDWLRQV WKDW XVH WKH )DVW%LW WHFKQRORJ\ x Query Driven Visualization 2QH RI WKH PRVW LPSRUWDQW DSSOLFDWLRQV RI )DVW%LW LV WKH 4XHU\ 'ULYHQ 9LVXDOL]DWLRQ 4'9 GHYHORSHG LQ FROODERUDWLRQ ZLWK WKH 6FL'$& 9LVXDOL]DWLRQ DQG $QDO\WLFV &HQWHU RI (QDEOLQJ 7HFKQRORJLHV 9$&(7 ,Q D QXPEHU RI RQ JRLQJ SURMHFWV )DVW%LW LV XVHG DV WKH VRIWZDUH WR VSHHG XS WKHLU GDWD DFFHVVHV $Q H[DPSOH ZKHUH )DVW%LW LV XVHG LV LQ WKH WUDFNLQJ RI SDUWLFOHV LQ WKH VLPXODWLRQV RI WKH /DVHU :DNHILHOG 3DUWLFOH $FFHOHUDWRU >53: @ ,Q WKLV SDUWLFXODU WHVW WKH GDWD VHW FRQWDLQV DERXW PLOOLRQ SDUWLFOHV 8VLQJ )DVW%LW WR WUDFN DV PDQ\ DV PLOOLRQ SDUWLFOHV LV PXFK IDVWHU WKDQ XVLQJ WKH FXVWRP FRGH WKDW GLUHFWO\ ZRUNV ZLWK WKH UDZ GDWD 7\SLFDOO\ PXFK IHZHU SDUWLFOHV DUH WUDFNHG LQ DQ DQDO\VLV $V WKH QXPEHU RI WUDFNHG SDUWLFOHV GHFUHDVHV WKH UHODWLYH DGYDQWDJH RI )DVW%LW LQFUHDVHV EHFDXVH D VPDOOHU SDUW RI WKH ELWPDS LQGH[HV LV QHHGHG LQ RUGHU WR ORFDWH WKH SDUWLFOHV ,Q WKLV WHVW XVLQJ )DVW%LW LV DV PXFK DV WLPHV IDVWHU ZKHQ WUDFNLQJ D IHZ KXQGUHG SDUWLFOHV D XVH FDVH FRPPRQ LQ PRVW UHDOLVWLF GDWD DQDO\VLV VFHQDULRV (YHQ LI WKH QXPEHU RI SDUWLFOHV WUDFNHG LV D IHZ PLOOLRQV )DVW%LW SHUIRUPV WLPHV IDVWHU WKDQ VFDQQLQJ WKH GDWD 7KLV NLQG RI SHUIRUPDQFH LV HVVHQWLDO IRU UHDO WLPH LQWHUDFWLRQ ZKLFK FDQQRW EH DFKLHYHG E\ XVLQJ DQDO\WLFDO VRIWZDUH SDFNDJHV VXFK DV ,'/ )LJXUH LV D VFKHPDWLF GLDJUDP VKRZLQJ WKH XVH RI )DVW%LW LQ DQ LQWHUDFWLYH PDQQHU WR VHOHFW VXEVHWV RI SDUWLFOHV E\ VSHFLI\LQJ FRQGLWLRQ WR FRPSXWH KLVWRJUDPV DQG WR WUDFN SDUWLFOHV RYHU WLPH VWHSV
相关主题
文本预览