IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART A SYSTEMS AND HUMANS 1 3-D Virtual

IEEETRANSACTIONSONSYSTEMS,MAN,ANDCYBERNETICS—PARTA:SYSTEMSANDHUMANS1

3-DVirtualStudioforNaturalInter-“Acting”

NamgyuKim,WoontackWoo,Member,IEEE,GerardJ.Kim,Member,IEEE,andChan-MoPark

Abstract—Virtualstudioshavelongbeenusedincommercialbroadcasting.However,mostvirtualstudiosarebasedon“bluescreen”technology,anditstwo-dimensional(2-D)naturerestrictstheuserfrommakingnaturalthree-dimensional(3-D)interac-tions.Actorshavetofollowprewrittenscriptsandpretendasifdirectlyinteractingwiththesyntheticobjects.Thisoftencreatesanunnaturalandseeminglyuncoordinatedoutput.Inthispaper,weintroduceanimprovedvirtual-studioframeworktoenableactors/userstointeractin3-Dmorenaturallywiththesyntheticenvironmentandobjects.Theproposedsystemusesastereocam-eratofirstconstructa3-Denvironment(fortheactortoactin),amultiviewcameratoextracttheimageand3-Dinformationabouttheactor,andareal-timeregistrationandrenderingsoft-wareforgeneratingthefinaloutput.Synthetic3-Dobjectscanbeeasilyinsertedandrendered,inrealtime,togetherwiththe3-Denvironmentandvideoactorfornatural3-Dinteraction.Theenablingofnatural3-Dinteractionwouldmakemorecinematictechniquespossibleincludingliveandspontaneousacting.Theproposedsystemisnotlimitedtobroadcastproduction,butcanalsobeusedforcreatingvirtual/augmented-realityenvironmentsfortrainingandentertainment.

IndexTerms—Augmentedreality,calibration,imageseg-mentation,multiviewandstereocameras,registration,three-dimensional(3-D)interaction,three-dimensional(3-D)modelreconstruction,virtualenvironment(VE),virtualstudio,z-keying.

I.INTRODUCTION

VIRTUALstudioshavelongbeeninuseforcommercial

broadcastingandmotionpictures.Mostvirtualstudios

arebasedon“bluescreen”technology,anditstwo-dimensional

(2-D)naturerestrictstheuserfrommakingnaturalthree-

dimensional(3-D)interactions.Actorshavetofollowprewrit-

tenscriptsandmotionpaths,andpretendasifdirectly

interactingwiththesyntheticobjects.Thisoftencreatesan

unnaturalandseeminglyuncoordinatedoutput.Thisisquite

apparentwhenthebroadcastisliveandtheprocessingtocorrect

suchout-of-synchoutputisnearlyimpossibleonthefly.For

instance,itiscommontonotice,inalivebroadcastofweather-

ManuscriptreceivedFebruary10,2004;revisedJuly27,2004,November14,2004,andDecember23,2004.TheworkdescribedinthispaperwassupportedinpartandcarriedoutatAdvancedTelecommunicationsResearch(ATR)MediaIntegrationandCommunications(MIC)Laboratory,Japan,from2000to2001,andalsobytheKoreanMinistryofEducation’sBrainKorea(BK)21programandtheKoreanMinistryoftheInformationandCommunication’sInformationTechnologyResearchCenter(ITRC)program,andcontinuedatthePohangUniversityofScienceandTechnology(POSTECH)VirtualReality(VR)Laboratory.ThispaperwasrecommendedbyAssociateEditorI.Gu.N.Kim,G.J.Kim,andC.-M.ParkarewiththeDepartmentofCom-puterScienceandEngineering,PohangUniversityofScienceandTech-nology,Pohang,Korea(e-mail:ngkim@postech.ac.kr;gkim@postech.ac.kr;parkcm@postech.ac.kr).W.WooiswiththeDepartmentofInformationandCommunications,GwangjuInstituteofScienceandTechnology(GIST),Gwangju,Korea(e-mail:wwoo@gist.ac.kr).DigitalObjectIdentifier10.1109/TSMCA.2005.855752forecastnews,anincorrecttimingonthepartoftheannouncer

(e.g.,theweathermapbeingpulledoutafterapause).

Anotherlimitationofthecurrentvirtualstudioisinthecre-

ationofthebackdrop.Intheexampleoftheweatherforecast,

thebackdropwasquitesimplewith2-Ddrawingsofweather

mapsandtemperaturecharts.2-Dimagesorvideoscanbe

usedasabackdropaswell(tomaketheuserseembeing

somewhereelse),butthecurrenttechnologydoesnotallowfor

theactortointeractwiththeenvironment.Tosituatetheactorin

amorecomplexenvironmentandenable3-Dinteractionwith

it,theoperatingenvironmentmustbemadein3-D.Ingeneral,

however,themodelingofrealistic3-Dvirtualenvironments

(VE)isdifficultandlaborintensive.Inaddition,therendering

oftheVEonthefly(plusmixingitwiththeimageoftheactor)

requiresexpensivespecial-purposerenderinghardware.

Inthispaper,weintroduceanimprovedframeworkfor

virtualstudiosthatenablesactors/userstointeract,in3-D,

morenaturallywiththesyntheticenvironmentandobjects.The

proposedframeworkconsistsoffourmainparts:1)image-

based3-D-environmentgenerationusingaportablecamera

withstereoscopicadapter;2)real-timevideo-actorgeneration

usingamultiviewcamera;3)mixinggraphicalobjectsinto

the3-Dscene;and4)supportfor“natural”interactionsuch

ascollisiondetection,first-personview,andtactilefeedback.

Theproposedsystemisahybridapproachofaugmented-

andvirtual-realitytechniques,andisarelativelyinexpensive

methodforgeneratinga3-Dimage-basedinteractiveenviron-

mentinrealtime,withoutspecialrenderinghardwareanda

blue-screen-studiosetup.

Thispaperisorganizedasfollows.InSectionII,wereview

priorresearchesrelatedtothisworksuchasvirtualstudios,

3-Dinformation-processingtechniques,objectregistration,

image-basedrendering(IBR),and3-Dinteractiontechniques.

SectionIIIwillpresentanoverviewoftheproposedinteractive

virtualstudio,anddetailedtechnicaldetailsaboutthefourmain

componentsofthesystemfollowinthesubsequentsections.

Inthefinalsections,weillustratetheimplementedsystemand

evaluateitsmeritsandlimitationsinthecontextofvirtualstu-

dios.Finally,weconcludethepaperwithanexecutivesummary

andadiscussionofnewpossibleapplicationsandfuturework.

II.RELATEDWORK

Ingeneral,virtual-studiosetsrequire“blue-screen”(chro-

makeying)technology,high-endgraphicworkstations,camera-

trackingtechnologyandsignalcompositingforhighrealism,

andexactmixingresults[5].The2-Dnatureofcurrentvirtual

studiosasdevelopedandusedinthecurrentbroadcastindustry

limitsitsusetosituationswherethecameraangleisfixedand

thereisminimaluserinteraction[6].

1083-4427/$20.00©2006IEEE