Multimodal menu presentation and selection in immersive virtual environments

Multimodal Menu Presentation and Selection in Immersive VirtualEnvironmentsNamgyu Kim,Gerard Jounghyun Kim,Chan-Mo Park,Inseok Lee and Sung H.LimDept.of Computer Science and EngineeringandDept.of Industrial EngineeringPohang University of Science and Technology(POSTECH)San31,Hyoja-dong,Pohang,Kyungbuk,Koreangkim@postech.ac.krAbstractUsability has become one of the key ingredients in mak-ing virtual reality(VR)systems work,and a big part of a usable VR system is in the design of effective inter-face/interaction schemes.During the past few years,sev-eral empirical studies have been conducted to evaluate the usability of generic interaction techniques in the context of virtual environment design.As a continued effort in this line of research,we investigate in the usability of vari-ous menu presentation and multimodal selection schemes in immersive virtual environments.It is often required to make commands in a virtual environment,and when there are many number of available commands with a complex structure,a hierarchical menu has been one of the most popular choices.While the task of menu selection may be viewed as a composite task of manipulation and selection, and therefore,we can certainly build on the prior research in generic interaction tasks,there are also enough idiosyn-crasies that warrant a more in depth look as a generic task of its own.We have identified5major menu display and13 possible menu selection methods;among them we havefin-ished the usability testing for thefirst two(tracking/button and gesture/gesture)interaction methods(across the5dis-play methods).In this paper,we explain how we came up with the classification,how we set up and ran the usabil-ity testing,and report on the results we have obtained from testing the two interaction methods.1.IntroductionUsability has become one of the most important aspect of making virtual reality(VR)systems succeed for a given application,and a big part of a usable VR system is in the design of effective interface/interaction schemes.During the past few years,several empirical studies have been con-ducted to evaluate the usability of generic interaction tech-niques such as navigation,selection,and manipulation in the context of virtual environment design[4,6,14].While these results are valuable for designing interaction meth-ods,such a collective effort still falls short of forming a mature interaction/interface design guideline for VR.It is not clear how these research results apply to designing in-terfaces for particular task,application,or even a composite generic task.The reason may be that there are simply not enough studies yet to conclude on a definite set of guide-lines due the many factors that must be considered including the types of interaction or display devices,types of domain tasks,display resolution,frequency of the primitive tasks, number of hands used,etc.As a continued effort in this line of research,we in-vestigate in the usability of various menu presentation and multimodal selection schemes in immersive virtual environ-ments.It is often required to make commands in a virtual environment,and when there are many number of available commands with a complex structure,a hierarchical menu has been one of the most popular choices both in2D and 3D environments.A typical example is a computer aided design system in which many functionalities are available with a large number of commands with up to three to four hierarchical levels.On top of this,the task of design re-quires the user to continually use the menu system.Auto-CAD,for instance,has about few hundred available com-mands with typical depths of two to three,and an average task(e.g.making a block with a hole)might require three to four commands[2].The menu system is also the most fa-miliar2D computer interface that we know of,and it would be most beneficial for the computer users to have it extended in the immersive3D environments.Therefore,while thetask of menu selection may be viewed as a composite task of manipulation and selection(or as a simple2D naviga-tion)and thus,we can certainly build on the prior research in generic interaction tasks,it should be studied in more depth as one generic VE task for its aforementioned impor-tance,frequent usage,and other idiosyncrasies(e.g.virtual space visibility,showing of the command path).In our study,wefirst reclassified various2D and3D menu presentation styles in the context of immersive VE and identified5major menu display methods appropriate for the usability testing.We believe that menu display oc-cupies a significant portion of the visual display,and thus is an important element in the overall interaction closed loop (i.e.see,select,see,select...).Then,by viewing the menu selection task as a composite task of“positioning”(manip-ulation)and“making a command”,and assigning different interaction modalities for each subtask,wefinally identified 13possible menu selection methods after eliminating some of the“tivially deemed infeasible or insignificant”ones. Among them,we havefinished the usability testing for the first two(tracking/button and gesture/gesture combination) interaction methods across the5display methods.The main objective of the usability test was to collect quantitative data such as the task completion time(for positioning and mak-ing a command),and accuracy(error rate)under the differ-ent menu interaction schemes,and by analyzing them derive a principle for immersive menu system design for VR sys-tem.Qualitative factor data were also collected by menus of a survey that addressed user preference and interaction naturalness.This paper is organized as follows.In Section2,we review some of the research results relevant to this work. In Section3and4,we explain how we classified different menu display and multimodal selection methods,and gen-erated thefinal candidates for the usability testing.Section 5describes the experiment set-up and discusses on the re-sults.Finally,in Section6,we the paper with a summary.2.Related WorkEffective selection styles and structure designs of2D desktop menus have been investigated extensively in the Human Computer Interaction(HCI)areas[9,15].Many HCI studies attest the convenience of the menu systems in the2D desktop environment.But,Whether these mer-its of2D menus carry over to the3D VE is not clear be-cause methods of displaying and interacting with the virtual menus in3D are quite different.Nevertheless,3D virtual menus have been implemented and used by many employ-ing various methods of menu invocation,locating,selection, and highlighting[12,8,1].Jacoby et al.designed a hierarchical menu structure with pop-up and pull-down the3D virtual menus and employed four hand gestures for selecting the menu items[12].A use of a3D widget,a virtual instrument with menu items,were proposed by Ferneau et al.for menu selection method[8]. The device used for menu selection was a3D mouse,and the buttons of the3D mouse had to correspond to the items of the physical virtual instrument.This approach assumed that the abilities of the physical device match those of the virtual.A technique of embedding a general2D interaction metaphor in the3D VE considered by Angus et al.[1]. They focused on the implementation problems of importing flat screen applications to VR environments without having to modify the application source code.But,their approach could result in a performance degradation by using large textures and by inconsistent texture resolution at different view positions.Feiner et al.built2D windows for a3D augmented reality environment which provided a transpar-ent window display relative to user’s head and body[7]. This system provided three kinds of windows:surround-fixed,display-fixed,and world-fixed.The surround-fixed is displayed at afixed position in the world,and display-fixed is positioned at afixed location relative to user’s head ori-entation.World-fixed isfixed locations or objects which are invokable by the user.Our classification for menu display is much influenced by this work.While,most commonly,a virtual menu is implemented as a2Dflat object,Liang et al.introduced the3D object menu concept in their interactive3D modeling system,JD-CAD[13].The3D Ring menu which used3D objects in stead of2Dflat items was easy to use with minimum depth ambiguity through its daisy mechanism.Problems exhib-ited in3D object selection are also manifested in3D menu selection,an analogy suggested and found by[6].In this study,the effect of different placements of the menu bar, font type and size,the menu availability,the use of high-lighting and the depth and breadth of the menu were all considered,and he several design guidelines were derived the use of maximum8,18point sized menu items with highlighting on selected items,and the discretional use of context sensitive menus.Bowman et ed virtual menus in their applications-“Virtual Venue”and“VR Gorilla Exhibit”-using the“pen and tablet”metaphor[3].Bowman reported a result that highlights the efficiency of using2D menus in3D environ-ments instead of employing direct actions upon world ob-jects through multiple metaphors.Most work related to the virtual menu did not carry out usability tests.On the other hand,selection,manipulation and navigation techniques in VE have been investigated by many[4,11,14].These work give testing guidelines and methods for interaction techniques testing.However,menu selection is different from the general selection task because of menu’s2D appearance and the predetermined selectionFigure1.Variants of VE menu display by location.mechanism.Our study considers menu selection as concep-tually generic task composed of the two subtasks of“posi-tioning”and“making a command”.3.Menu Presentation3.1.Location and View DirectionIn3D environments,unlike in the2D,we mustfirst care-fully consider where to locate the menu system within the world,which in turn,will determine the user’s viewing di-rection to the menu.There are three possibilities we have considered.Figure1illustrates the three variants.World Fixed(WF):The menu system resides at afixed location in a“strategic”world location.View Fixed(VF):The menu system is attached at and viewed from afixed offset from the user(thus,it moves with the“head tracked”user).Object Fixed(OF):The menu system is attached at one or more“strategic”virtual objects(which may move in time).WF allows a relatively comprehensive display of the overall menu structure and menu selection history(because it is located at a strategic location away from where the task is being carried out),while with VF and OF,a more com-pact menu display must be used not to block the task area. This is especially true in an immersive environment where head mounted displays are used,as the most HMD’s suffer from low resolutions and narrowfields of view.The next section addresses this design issue.3.2.HMD Resolution and Display ItemsThis study aims at studying the effectiveness of various menu systems in an immersive head tracked virtual envi-ronment and assumes the use of an HMD.Although the situation is fast changing,it will still be some time before even HMD’s with the relatively low resolutions(e.g.640 x480)and wide FOV’s become widely available and af-fordable.With such a limitation,it is difficult to display large amounts of information at once in one display frame (e.g.letters on the menu are not recognizable if displayed in small scales),a potential problem for a hierarchical menu system with many available commands and levels.The fol-lowings are variants of menu display methods ranging from the usual to the compact ones to address this problem.Fig-ure2pictorially depicts these alternatives.Pull-down(PD):The usual pull-down menu that dis-plays the highest level menu items,and show its branches only during the selection task.Pop-up(PU):The usual pop-up menu that disappears once the selection is made.The menu structure associ-ated with the particular menu selection path is shown only on user’s invocation.Stack Menu:A menu system that persistently displays the selection path either at the top portion of the pop-up menu(disappears once the selection is done),thus called the Fixed Stack(FS),or at a separate location(e.g.at the corner of the screen),thus called the Bas-ket Stack(BS).Only the menu options selectable at a given level is shownObject-Specific Menu:This menu system is a con-glomerate of the pull-down/pop-up and object-fixedmenu schemes.Each object contains a specific pull-down(OPD)or pop-up(OPU)menu applicable to that object class,thus the menu structure is distributed among the virtual objects in the scene.Oblique/Layered(OL):This is aflat menu presentation displayed in an oblique fashion,or its structure orga-nized and displayed bylayers.Figure2.Variants of VE menu display by items 3.3.Final Candidates for ExperimentSome of the menu display schemes identified according to the respective criterion go hand in hand for one another.For instance,as already indicated,OPD/OPU are only pos-sible under OF.Table1summarizes the two main criteria we have used in classifying the various menu display methods and reasons of why some of the combinations were elimi-nated from thefinal candidates for the experiment.Location and DirectionWF VF OFPU R2R3 HMD PD R2R3Resolution FS R1R1 and BS R1R1Display OPU R3R2R3Items OPD R3R2R3OL R1R1 Table1.A classification of menu display meth-ods for VE.The combination marked with the“”in the table are the combinations that are selected for the usability testing, while the others were excluded for the following reasons.Reason1(R1):In WF or OF,there is less constraint on limiting the number of display items to maximize the visibility of the task area,because in the WF case, the menu location is away from the task area,and in the OF case,the user is quite aware of the task(and the task to be done)as a specific object is selected.Reason2(R2):This reason is complimentary to R1.It is also the very reason behind the design of Oblique/Layered menu system.Reason3(R3):In terms of the selection mechanism, we believe that there is little difference between the WF and the OF;the only difference is that in OF,the menus might move and the menu items may differ slightly from object to object.Since our focus is in the interaction(although display method is part of it), we decided to effectively merge WF and OF into one group.Thefivefinal candidates for menu display methods to be tested in conjunction with the interaction methods are(See Figure3):WF-PD:World Fixed,Pull-downWF-PU:World Fixed,Pop-upVF-FS:View Fixed,Fixed StackVF-BS:View Fixed,Basket StackVF-OL:View Fixed,Oblique/LayeredWF-PDWF-PUVF-FSVF-BSVF-OLFigure3.Menu display methods seen by testsubjects4.Multimodal Menu Interface:Positioningand Making a CommandWe already mentioned earlier that the menu selectiontask is composed of two primitive subtasks,positioning andmaking a command.In this section,we describe the classi-fication of menu selection methods,according to three dif-ferent modal input methods for these two subtasks respec-tively.For positioning,we considered tracking,gesture andvoice,and for making a command,we considered buttoninput,gesture,and voice to signal thefinal yes/no decision.Tracking simply tracks to user-hand or a metaphorical ob-ject to designate a desired menu item and is considered acontinuous event-driven modality.On the other hand,voiceand gesture recognition,that allows users to directly speakof the menu item(e.g.start,enter,escape)or make posi-tioning or commands(e.g.next,previous)has a discreteevent-driven modality.Table2shows the overall interac-tion scheme classification.The“X”marks the infeasiblecombinations for the selection task.For instance,trackingand voice combination under the column of zero hand issimply impossible(i.e.one hand used).We make a note ofthat the“”merely represent combinations that might bepossible;further investigations might eliminate some of thecombinations.For instance,tracking and gesture with onehand might not be an appropriate combination,as the posi-tioning task will be affected by the gesture making task.5.The Experiment5.1.Experimental DesignThe menu system usability test is set in an immersiveenvironment for virtual ers can create,arrage andmodify pipes in a virtual building,through the menu systemand other auxiliary functionalities such as sound feedbackfor collision and task completion.Figure8shows the menustructure used in the experiment.As a start,we selectedTB1and GG1among many combinations shown in Table2.For TB1,users used a tracker(Polhemus Fastrak)and a3D mouse(Logitech3D mouse)and for GG1,used a glove(5DT glove).Four mouse buttons were used for menu invo-cation(pedestal button),ray invocation(left button),enter(right button),and cancel(middle button).The glove sys-tem hadfive gestures for menu invocation,enter,cancel,previous and next.Each gesture is a pinch action-betweenthe thumb and palm,thumb and ring,thumb and pinkie,thumb and index,thumb and middle.The gestures weremade as simple as possible,and a limit switch was used onthe thumb to ease the recognition of the hand action.Figure4are examples of scenes seen by the user in the testVE.Figure4.Two interaction methods(TB and GG)for menu selectionThe users were asked to perform the following two tasks1.(T1)make a box and view help information regardinga menu-this task consists of two independent com-mands1.2.(T2)make a T-shaped pipe,select the pipe,andthen change its color to blue-this consists of threesequence-dependent commands.1The term“command”is used to describe a distinct function of thevirtual piping system that may require several steps of primitive interactiontasksPositioningContinuous DiscreteEvent-driven Event-drivenModality ModalityTracking Gesture Voice Num.of hands012012012Button X TB1TB2X GB1GB2X VB1XMaking Discretea Event-Gesture X TG1TG2X GG1GG2X VG1X Command DrivenModalityV oice X TV1X X GV1X VV0X X Table2.A classification of menu selection methods for VE.Each task is repeated5times across the respective5dis-play methods(Figure3).We gathered four types of test data:Completion time-from when users invocate an initial menu to when users complete the given task.Positioning time-from when users invocate the menu to when users locate to the right menu item.Command time-from the time of correct positioning, till making a command(the menu item chosen right beforefinal selection).Error-the frequency of selection of wrong menu items.Erroneous menu selection data are excluded for figures of the above three measures.We also surveyed the user,asking their opinions on the relative convenience of searching the menu items among5 display methods,and among the two modality combinations (TB1and GG1).5.2.SubjectsA total of16users participated in the experiment-8 for TB1and8for GG1.Half of each group were experi-enced in using3D devices.Before the experiment,users werefirst trained to make menu selection in a particular modality(TB1or GG1).For the TB1test,users learned the mouse manipulation,and for the GG1test,users learned the gesture commands.In the training environment,users were shown a box and a command text as shown in ers were trained by making a gesture or a mouse button push corresponding to the command text.Figures5 shows the“start”command text:TB1requires a pedestal button push and GG1requires a thumb-and-palm gesture in response.We repeated the training task and when the task completion time for e ach task came down to about20sec-onds,we accepted the trainee as experimentsubjects.Figure 5.Learning basic interaction skills for menu selection in two different modalities.5.3.Results and DiscussionA factorial analysis of variance(ANOV A)was performed on the completion time,positioning time,command time and error.Table3shows the means and standard deviations for each task result.In general,it was found that with GG1, it took less time to make a command than to home in on the right menu item.In contrast,with the TB1,the pattern was the other way around.This shows the effectiveness of gestures for positioning in menu selection.Between the two tasks(T1and T2),we did notfind any significant statistical differences.Interestingly,the user survey found the gesture input method as more convenient and natural.Figure6shows the completion time,positioning time and command time across5display methods.In case for T1,we found a significant difference between VF-FS and WF-PD,while for T2,we could notfind this property.Time Completion Positioning Command (Sec.)Time Time Time GG1T1TB1GG1T2TB1Table3.Means and standard deviations of per-formances in T1and T2.We performed the Tukeys’multiple comparison analysis, and as a result,three similar performance groups surfaced; WF-PD/WF-PU,WF-PD/VF-BS/VF-OL,and VF-FS/VF-BS/VF-OL.There existed a statistically significant perfor-mance differences among these three groups,explaining the worst task performance under the WF-PD in both TB1and GG1.Figure7shows the relationship between the display methods and input modalities.The marked data is the ra-tio of positioning time to the completion time.With TB1, WF-PD takes more positioning time than that of other dis-play methods in both T1and T2.These results are explained by the amount of“dis-tance”that must be traversed to home in on the right item. Structure-wise,WF-PD generally requires the most amount of travel distance.This is further seen by the differences between the two tasks,T1and T2.In T1,the menu items needed are separated relatively farther than those needed by T2due to the menu structure that tends to group dependent menu items together.To summarize,we found that the gesture input method for the positioning task was a faster and more convenient method compared to the continuous tracking,due to the amount of“travel”that has to be carried out with the con-tinuous tracking.This was even more apparent in menu se-lection tasks in which menu items were separated farther in their locations.6.Conclusion and Future WorkIn this paper,we have made a classification for various menu selection systems for VE according to their3D lo-cation,viewing direction,display items,display resolution, and input modalities.Among the many possibilities,as a start,we have conducted a usability test for thecombina-Figure6.Relative task performances across thefive display methods.tion of continuous tracking/button selection,and gesture positioning/gesture selection across5menu display meth-ods.We have found that the gesture input method for the positioning task was a faster and more convenient method compared to the continuous tracking,due to the amount of “travel”that has to be carried out with the continuous track-ing.This was even more apparent in menu selection tasks in which menu items were separated farther in their locations. We plan to continue our usability testing for the remaining input modality combinations,andfinally derive a compre-hensive design guidelines for menu systems in an immer-sive VE.References[1]I.G.Angus and H.A.Sowizral.Embedding the2d interac-tion metaphor in a real3d virtual environment.Proceedings of SPIE,Stereoscopic Displays and Virtual Reality Systems, 2409:282–293,1995.[2]AutoDesk.AutoCAD Realease12:User Guide.AutoDesk,Inc.,1993.[3] D.Bowman,L.F.Hodges,and J.Bolter.The virtual venue:User-computer interaction in information-rich virtual envi-ronments.Presence:Teleoperation and Virtual Environ-ments,7(5):478–493,October1998.Figure7.Relationship between display methods and input modalities.[4] D.A.Bowman and L.F.Hodges.An evaluation of tech-niques for grabbing and manipulation remote objects in im-mersive virtual environments.Proceedings of the Sympo-sium on Interactive3D Graphics,pages138–188,1997. [5]R.Darken.Hands-off interaction with menus in virtualspaces.Proceedings of SPIE,Stereo Displays and Virtual Reality Systems,2177:365–371,1994.[6]R.P.Darken.Navigating in large virtual worlds.The Inter-national Journal of Human-Computer Interaction,8(1):49–72,1996.[7]S.Feiner,B.MacIntyre,M..Haupt,and E.Solomon.Win-dows on the world:2d windows for3d augmented reality.Proceedings of UIST’93,pages145–155,1993.[8]M.Ferneau and J.Humphries.A gloveless interface forinteraction in scientific visualization virtual environments.Proceedings of SPIE,Stereoscopic Displays and Virtual Re-aliy Systems,2409:268–274,1995.[9]J.D.Foley,V.L.Wallace,and P.Chan.The human factorsof computer graphics interaction techniqus.IEEE Computer Graphics and Applications,pages13–48,November1984.[10] D.J.Gillan,K.Holden,S.Adam,M.˜Rudisill,andL.Magee.How shouldfitts’law be applied to human-computer interaction.Interacting with Computers, 4(3):291–313,1992.[11] D.Hix,E.J.Swan,J.L.Gabbard,J.˜Durbin,and T.King.User-centered design and evaluation of a real-time battle-field visulization virtual environment.Proceedings of IEEE Virtual Reality,pages96–103,1999.Figure8.The virtual CAD menu structure usedin the experiment.[12]R.H.Jacoby and ing virtual menus in virtualenvironment.Proceedings of SPIE,Visual Data Interpreta-tion,1668:39–48,1992.[13]J.Liang and M.Green.Jdcad:a highly interface3d modelingputers and Graphics,18(4):499–506,1994. [14]M.R.Mine.Virtual environment interaction techniques.UNC Chapel Hill Computer CS Tech.Report,TR95-018, 1995.[15]J.Preece,Y.Rogers,H.Sharp,D..Benyon,S.Holland,andT.Carey.Human-Computer Interaction.Addison-Wesley, 1995.[16]Sense8.WorldToolKit Reference Manual.Sense8ProductLine,April1999.。