User Performance in Relation to 3D Input Device Design

Computer Graphics 32(4), November 1998. pp50-54. (c) ACM User Performance in Relation to 3D Input Device Design

Shumin Zhai IBM Almaden Research Center 650 Harry Road, San Jose, CA 95123, USA +1 408 9271112 zhai@almaden.ibm.com

ABSTRACT
Based mainly on a series of studies the author conducted at the University of Toronto, this article reviews the usability of various 6 degrees of freedom (6 DOF) input devices for 3D user interfaces. The following issues are covered in the article: the multiple aspects of input device usability (performance measures), mouse based 6 DOF interaction, mouse modifications for 3D interfaces, free-moving isotonic 6 DOF devices, desktop isometric and elastic 6 DOF devices,armature-based 6 DOF devices, position vs. rate control, and the form factors of 6 DOF control handle. These issues are treated at an introductory and practical level, with pointers to more technical and theoretical references.

INTRODUCTION

As three-dimensional (3D) graphics moves to the core of many mainstream computer systems and applications, the search for usable input devices for 3D object manipulation becomes both an academic inquiry and a practical concern. In the case of the 2D Graphical User Interface (GUI), the computer mouse established itself very early and quickly replaced the light pen as the de facto standard input device (See [Perry, 1989] for a review of mouse history). In the case of 3D interfaces, however, there is still not an obvious winner suitable for all applications. Primarily based on the author's own research, this article offers a few perspectives on the usability of various input devices for 3D interface. This article does not intend to present a comprehensive literature review or a series of experimental studies in a methodical manner. Rather, it intends to be introductory and practical. Interested readers are encouraged to examine more technical details in the papers referenced.

In order to be able to manipulate 3D objects, one generally needs at least six degrees of freedom (6 DOF), three for X, Y, and Z translation and three for 3D rotation. The difficulty in establishing a standard 6 DOF device is twofold. First, there are engineering challenges in terms of sensor technologies, manufacturing cost, and designer's creativity. It is highly likely that the most elegant 6 DOF device has still not been designed. Second, and perhaps more importantly, even if we could easily make any device we like to, there is only a very limited knowledge about what properties a good 6 DOF controller should have. Given the long history of human factors study on input control devices, dated back to World War II [Orlansky, 1949], "One would expect the relationship of the hand to the controlled element, being at the one time both an input and output, to be a fruitful area for research", but the reality is that little is well understood [Burrows, 1965]. Burrows pointed out that the reluctance to conduct research in this area is understandable in view of the immensity of the possible interactions among the many dimensions of control feel.

This is not to say that there isn't any intellectual guidance to 6 DOF input device design. Motivated by the manual control problems in vehicles, air crafts and other dynamic complex machines, the topic of "manual control and tracking" has been extensively studied (see [Poulton, 1974] for a summary). However, system dynamics resulted from mass, spring, viscosity, transmission delay etc in these systems, soon dominated the area. The study of input control device properties (e.g. [Bahrick, Bennett, & Fitts, 1955] ) quickly gave way to mathematical control theory modeling of man-machine systems. The more general body of knowledge on human motor control and learning (see [Schmidt, 1988] for example), while offers many insights, rarely provide direct design guidelines. One recent review of the scattered literature related to input device design is provided in [Zhai, 1995].

PERFORMANCE MEASURES OF 6 DOF INPUT DEVICES

There are many choices in designing or selecting a 6 DOF input device. The choice on every design dimension may have implications on users' performance. Aside from application specific requirements, there are at least the following six aspects to the usability of a 6 DOF input device:

Speed
Accuracy
Ease of learning
Fatigue
Coordination
Device persistence and acquisition

The first four aspects are common to all input devices and their meaning are obvious. The fifth aspect, coordination, is unique to multiple degrees of freedom input control. There are many ways of measuring the degree of coordination. One effective way of quantifying it is based on the ratio between the length of actual trajectory and that of the most efficient trajectory in the coordination spaces, including translation space, rotation space, and the 2D space between translation and rotation [Zhai, 1998]. By such a measure, in order to produce the most coordinated path, one has to simultaneously move all degrees of freedom involved at the same pace towards their respective goal states.

The sixth aspect of input device usability is the ease of device acquisition. This is often an overlooked aspect of input device usability. Although a mouse is less dexterous than a pen like input device (a stylus), the fact that a mouse can be more easily acquired is one important reason that made it the dominant 2 DOF input device. Many factors, such as the distance to the computer keyboard home row (ASDFGHJKL keys), contribute to the ease of device acquisition. One of them is the device location persistence when released. With a mouse or a trackball, when released by the hand (in order to type something, for example) it stays in position. This is not true with a stylus.

With these measures in mind, the remainder of the paper examines a few common classes of 6DOF input devices.

MOUSE BASED 6 DOF INPUT

Mouse mapping

The simplest implementation of 6 DOF manipulation can be 6 graphical sliders on the computer screen. Each can be dragged with a standard 2 DOF input device, such as a mouse. There are two fundamental usability problems to such a 6 DOF interaction technique. First, people can not mentally decompose orientation into separate rotational axes [Parsons, 1995]. Second, since one has to time-multiplex between the separate degrees of freedom, it is not possible to form a coordinated movement in the 6 DOF space.

Researchers soon moved away from the 6 slider implementation to more complex mapping techniques. One such technique is enclosing the manipulated 3D object with a virtual sphere [Chen, Mountford, & Sellen, 1988]. Some of these techniques in fact have be widely used in 3D graphics software such as VRML browsers and CAD packages. A recent study [Hinckley, 1997] found the mouse mapping techniques still inferior to integrated 6 DOF devices.

Mice Modified for 3D Operation

Efforts have been made to add more physical degrees of freedom to the mouse for 3D interfaces. One example is the roller mouse that had an additional degree of freedom by means of a roller ([Venolia, 1993], Figure 1). A user could rotate the roller to move a 3D cursor in the depth dimension (translation only). Another example is the Rockin' mouse with two additional degrees of freedom by means of rocking motions on a tablet [Balakrishnan, 1997] (Figure 2). A mouse with a roller for 3D input

Figure 1. A mouse with a roller for 3D input [Venolia, 1993]

Since dedicated physical degrees of freedom are provided for the third dimension, these modified mice should outperform a conventional mouse that operates by means of simple mode switching. (One common mouse switching technique, for example, is that pressing and holding a mouse button down switches vertical mouse cursor motion to motion in the depth dimension). It was shown that in a 3D positioning task, the rockin' mouse was 30% faster in comparison to a standard mouse tablet [Balakrishnan, 1997]. On the other hand, since the depth dimension is operated by a behavior and muscle groups different from those of the x-y mouse motion, it can be difficult to produce simultaneous, coordinated motion with either the roller mouse or the Rockin' mouse. A mouse with two additional degrees of freedom

Figure 2. A mouse with two additional degrees of freedom [Balakrishnan, 1997]

"FLYING" MICE

Given the success of the mouse in 2D interfaces, it is natural to attempt to make a flying mouse - a mouse that can be moved and rotated in the air for 3D object manipulation. Indeed many such devices have been made. Theoretically, a mouse is a free-moving, i.e. isotonic device. When using such a device, the displacement of the device is typically mapped to a cursor displacement. Such type of mapping (transfer function) is also called position control. Figure 3 shows a few examples of 6 DOF isotonic position control devices. Most of these devices are instrumented with a magnetic tracker for 6 DOF sensing. A sample of free moving 6 DOF devices

Figure 3. A sample of free moving 6 DOF devices - "flying mice". (a) The "Bat", designed by C. Ware [Ware, 1990] (b) The Cricket(tm), manufactured by Digital Image Design Inc., New York, NY, USA (c) The MITS Glove, designed by the author, consists of a Bird(tm) tracker and a clutch.

The advantages of these "flying mice" devices are:

Easy to learn, because of the natural, direct mapping.
Relatively fast speed. Studies have shown that 6 DOF isotonic position control devices tend to outperform other type of devices in speed. This was particularly true for novice users [Zhai, 1995; Zhai, 1998; Zhai & Milgram, 1993b] (Figure 4).

Figure 4. Trial completion time in a 6 DOF docking task. The Free moving device was faster, particularly in early stage (From [Zhai, 1998])

However, there are many disadvantages to this class of devices:

Limited movement range. Since it is position control, hand movement can be mapped to only a limited range of the display space. In the case of standard 2 DOF mouse, this problem can be solved by making it a "relative" device. One can lift the mouse up and put it down at a new location of the mouse pad. Similarly, one can also use a clutch (Figure 3 c) in a 6DOF isotonic position control device.
Lack of coordination. In position control object movement is directly proportional to hand/finger movement and hence constrained to anatomical limitations: joints can only rotate to certain angle. These limitations prohibit well-coordinated movement, particularly when the user has to re-clutch or reposition the hand (finger) relative to the input device surface (Figure 6). See [Zhai, 1998] for detailed discussion on coordination (Figure 5).
Fatigue. This is a significant problem with free moving 6 DOF devices because the user's arm has to be suspended in the air without support. The mouse did not have such a problem because it stays on the desktop. One can rest the arm and hand on the desktop surface when operating a mouse, but not with the flying mice.
Difficulty of device acquisition. The flying mice lack persistence in position when released. With a mouse or trackball, when released by the hand, it stays in position. This is not true with a stylus, nor with any of the flying mice. This property of not "staying put" inhibits some classes of interaction. Such devices may have to be put on desk and picked up for the next transaction. In the case of a glove, this is even more problematic.

Figure 5. Coordination as measured by inefficiency in free moving position control vs. in elastic rate control. Significantly longer movement was "wasted" with the free moving device. See [Zhai, 1998] for details.

FORM FACTOR MATTERS

Of course, there are many variations within each class of devices. One such variation is the form factor of the control handle (shape and size). As shown in Figure 3, there has been a variety of shape and size used in constructing free moving 6 DOF input devices. Some of these devices, such as the glove implementation (Figure 3 c) require manipulation with wrist and arms, but exclude the figures. Figure 6 shows an alternative design, based on the very same sensor with a different shape and size that do allow the participation of the fingers that have higher dexterity. It was shown that significant performance advantage could be gained with such a design [Zhai, 1996].

Figure 6. The Fingerball. This device uses the same sensor as in the glove in Figure 3 (C). However, the different form factor made it possible to manipulate it with fingers, in addition to wrist and arm, leading to better performance [Zhai, 1996]
Figure 7. Performance difference between the Fingerball and the Glove in a 6 DOF docking task [Zhai, 1996]

"DESKTOP" DEVICES

The alternative to free moving input device, are devices that are mounted on stationary surface. We may call these desktop devices. Figure 8 shows some examples of 6DOF "joystick" devices that fall into this class. Common to such devices is a self-centering mechanism. They are either isometric devices that do not move by a significantly perceptible magnitude or elastic devices that are spring-loaded. When tension is released from the handle, the handle returns to a null position. Typically these devices work in rate control mode, i.e. the input variable, either force or displacement, is mapped onto the velocity of the cursor. In other words, the cursor position is the integration of input variable over time. Hence the term first order control. In contrast, in the case of the free moving (isotonic) devices, input variable (device displacement) is scaled to position (location and orientation) itself. Hence the term zero-order control. A sample of desktop 6 DOF input devices

Figure 8. A sample of desktop 6 DOF input devices. A sample of input devices for 6 DOF manipulation;. (a) The Spaceball(tm) is an isometric device manufactured by Spaceball Technologies Inc., Boston, MA, USA. (b) The SpaceMaster(tm) is an elastic device with a small range of movement (5 mm in translation and 15° in rotation), manufactured by BASYS GmbH, Erlangen, Germany. (c) The Space Mouse(tm) is an elastic device with slight movement (5 mm in translation and 4° in rotation). It is initially designed by DLR, the German aerospace research establishment, manufactured by Space Control Company, Malching, Germany and marketed by Logitech, Fremont, CA, USA.

Compatibility between resistance and transfer function

With proper clutching mechanism, it is conceivable to implement an isometric device in position control mode or an isotonic device in rate control mode (Figure 9) Interestingly, these two combinations tend to produce poor user performance (Figure 10). The reason is quite simple: the self-centering mechanism in an isometric device facilitates the "start, speed-up, maintain speed, slow-down and stop" cycle in rate control. The later half of the cycle is somewhat automatic with the self-centering mechanism in isometric devices. With a free moving device, one has to deliberately return to the null position. Two input device design dimensions: transfer function
vs. controller resistance

Two input device design dimensions: transfer function
vs. controller resistance

Figure 9. Two input device design dimensions: transfer function vs. controller resistance Figure 10. Isotonic rate control and Isometric position control tend to produce poor performance [Zhai & Milgram, 1993a] [Zhai, 1995]

The pros and cons of desktop 6 DOF devices

When used in rate control, an isometric device offers the following advantages:

Reduced fatigue, since the user's arm can be rested on the desktop.
Increased coordination. The integral transformation in rate control makes the actual cursor movement a step removed from the hand anatomy, resulting in greater coordination (Figure 11) [Zhai, 1998].
Smoother and more steady cursor movement. The rate control mechanism (integration) is a low pass filter, reducing high frequency noises.
Device persistence and faster acquisition. Since these devices stay stationary on the desktop, they can be acquired more easily.

Figure 11. Coordination as quantified by movement inefficiency. The Free moving position control device "wasted" more movement. Adpated from Figure 10 in [Zhai, 1998]

On the other hand, isometric rate control devices may have the following disadvantages:

Rate control is an acquired skill. A user typically takes tens of minutes, a significant duration for learning computer interaction tasks, to gain controllability of isometric rate control devices. It may take hours of practice to approach the level of isotonic position speed [Zhai, 1995] [Zhai & Milgram, 1993b].
Lack of control feel. Since an isometric device feels completely rigid, insufficient feedback is provided to the user at the kinesthetic channel. Kinesthetic (or proprioceptive) feedback can be critical to user's control performance.

A prototype of a 6 DOF Elastic General-purpose
Grip (EGG)

Figure 12. A prototype of a 6 DOF Elastic General-purpose Grip (EGG) [Zhai, 1995] [Zhai, 1993]

ELASTIC DEVICES

An elastic device moves proportionally to the force applied to it and yet maintains the self-centering mechanism needed for rate control. Figure 12 shows a prototype of a 6 DOF elastic rate control input device, dubbed EGG (elastic general-purpose grip). A similar suspension structure had been used for different purposes [Galyean & Hughes, 1991]. Since a user can feel both the pressure and the displacement with an elastic device, a stronger feedback is provided (see [Zhai, 1995] Chapter 3 for detailed neurophysiological analysis). However, the use of spring loading is not without tradeoff. The more displacement for the same force is provided, the stronger the kinesthetic feedback is provided, but the weaker the self-centering mechanism is for rate control compatibility (See [Zhai, 1995] Chapter 3 for this two-factor analysis of elastic devices).

When optimized, the appropriate amount of elasticity does improve a user's performance. This is particularly true at the early learning stages (Figures 13 & 14).

Time performance difference between an elastic
and an isometric rate controller in a 6 DOF docking task

Figure 13. Time performance difference between an elastic and an isometric rate controller in a 6 DOF docking task [Zhai, 1995] [Zhai, 1993]

Tracking error with an elastic vs. isometric
6 DOF controller task

Figure 14. Tracking error with an elastic vs. isometric 6 DOF controller task [Zhai, 1995] [Zhai & Milgram, 1993a]

MULTI-DOF ARMATURES

Another class of multi DOF input devices are mechanical armatures. Some of these are specialized and geometrically compatible with the graphical object being controlled so as to enable the posing of computer graphics characters more-or-less using puppeteering techniques. This is a technique that has been used by ILM, for exaple [Knep, 1995]. Digital Image Design Incorporated's "Monkey" [DID] as seen in the cover of this special issue and the armatures from Puppet Works [Puppetworks] are commercially available examples of this class of technology.

While strong in their intended, specialized purpose, their use is limited. They can do an excellent job for some types of character posing, but their applicability to interaction in general is limited, at least in this "puppet" type configuration.

One option that can bring armatures much closer to the generality seen with the other devices discussed occurs if they are configured as a single arm. This can be done with either the DID or Puppet Works devices, and is the standard configuration for other armatures, such as Immersion Corp.'s MicroScribe illustrated in Figure 15 [Immersion_corp].

A single arm multi-DOF armature input device

Figure 15. A single arm multi-DOF armature input device [Immersion Corp, ]

In this configuration, the armature is actually a hybrid between a flying-mouse type of device and a desktop device. It is like a desktop device in that it is typically mounted on the desktop, and consumes a small footprint. On the other hand, it is like a flying mouse in that manipulating the end point of the arm in space provides the desired 6DOF position data. Conceptually, these are near isotonic - with exceptional singularity positions - position control device like a flying mouse and thus share many of the pros and cons of isotonic position control discussed earlier. In addition, this approach has the following particular advantages:

Not susceptible to interference; most flying mice, based on magnetic tracker, have susceptibility to electromagnetic interference as well as to metal objects.
Less delay: response is usually better than most flying mouse technology, whose built-in electromagnetic noise reduction filters introduce significant time delay.
Can be configured to "stay put", when friction on joints is adjusted and therefore better for device acquisition.
Some, like the MicroScribe have potential to serve double duty as 3D digitizer (which is what it was designed for) as well as general input devic

On the other hand, they also have some particular drawbacks:

Device acquisition (especially if they do not "stay put").
Fatigue: as with flying mouse.
Constrained operation. The user has to carry the mechanical arm to operate, which is more cumbersome than most flying mice tethered by a cable. At certain singular points, position/orientation is awkward.

This class of devices can also be equipped with force feedback, as with SensAble Technology's Phantom Device [SenSable]. While showing some promise and novel user experiences, force feedback devices have yet to demonstrate their real benefit in the lab or in practice for mainstream applications. It is, of course, very interesting as a research area.

CONCLUSIONS

From this brief and incomplete guided tour, one can conclude that the complexity of 6 DOF input is far from being solved. None of the existing devices fulfills all aspects of usability requirement for 3D manipulation. A 6 DOF device that suits all, or the majority of, 3D applications is yet to be developed. However, the research conducted so far has offered implications for potential improvement and many insights into the characteristics and pros and cons of various designs. These insights should help the selection of various types of devices for different tasks. For example, when speed and short learning is a primary concern (such as in video games), free moving devices are most suitable. When fatigue, control trajectory quality and coordination are more important, such as in tele-operation, isometric or elastic rate control devices should be selected. The article has only covered a few dimensions of 6 DOF input device design. Many other dimensions, such as the cooperation of two hands in 3D input [Hinckley, Pausch, Goble, & Kassell, 1994] and those covered by other articles in this issue, are equally important. Further research, both developmental and experimental, is certainly needed.

ACKNOWLEDGEMENTS

I thank Paul Milgram and Bill Buxton for their longstanding mentorship. Some of the studies cited in this paper were conducted at the University of Toronto with funding support from ITRC, NSERC and DCIEM. Bill Buxton reviewed the article and co-authored the Multi-DOF Armatures section.

REFERENCES

Bahrick, H. P., Bennett, W. F., & Fitts, P. M. (1955). Accuracy of positioning responses as a function of spring loading in a control. Journal of Experimental Psychology, 49( 6), 437-444.

Balakrishnan, R., Baudel, T., Kurtenbach, G., and Fitzmaurice, G. (1997). The Rockin'Mouse: Integral 3D manipulation on a plane, In. Proceedings of CHI '97 Conference on Human Factors in Computing Systems.

Burrows, A. A. (1965). Control feel and the dependent variable. Human Factors, 7(5), 413-421.

Chen, M., Mountford, S. J., & Sellen, A. (1988). A study in interactive 3-D rotation using 2-D control devices. Proceedings of ACM Siggraph'88.

DID. Monkey . http://www.didi.com/www/areas/ products/monkey2/index.html.

Galyean, T. A., & Hughes, J. F. (1991). Sculpting: An interactive volumetric modeling technique. Computer Graphics, 25(4), 267-274.

Hinckley, K., Tulio, J., Pausch, R., Proffitt, D, Kassell, N.,. (1997). Usability Analysis of 3D Rotation Techniques. Proceedings of ACM Symp. User Interface Software and Technology.

Hinckley, K., Pausch, R., Goble, J. C., & Kassell, N. F. (1994). Passive real-world interface props for neurosurgical visualization. Proceedings of CHI'94: ACM conference on Human Factors in Computing Systems, Boston.

Immersion Corp. http://www.immerse.com/.

Knep, B., Hayes, C., Sayre, R., Williams, T. (1995). Dinosaur input device. Proceedings of CHI'95: ACM Conference on Human Factors in Computing Systems.

Orlansky, J. (1949). Psychological aspects of stick and rudder controls in aircraft. Aeronautical Engineering Review(January), 22-31.

Parsons, L. (1995). Inability to reason about an object's orientation using an axis and angle of rotation. Journal of Experimental Psychology: Human Perception and Performance, 21(6), 1259-1277.

Perry, T. S., Voelcker, J. (1989). Of mice and menus: Designing the user-friendly interface. IEEE Spectrum, 1989(September), 46-51.

Poulton, E. C. (1974). Tracking skill and manual control. New York: Academic Press.

Puppetworks, http://www.puppetworks.com/.

Schmidt, R. A. (1988). Motor control and learning - A Behavioural Emphasis. ( 2nd ed.): Human Kinetics Publishers,Inc.

SenSable, http://www.sensable.com.

Venolia, D. (1993). Facile 3D direct manipulation. Proceedings of INTERCHI'93: ACM Conference on Human Factors in Computing Systems, Amsterdam, The Netherlands.

Ware, C. (1990). Using hand position for virtual object placement. The Visual Computer, 6, 245-253.

Zhai, S. (1993). Investigation of feel for 6DOF inputs: isometric and elastic rate control for manipulation in 3D environments. Proceedings of The Human Factors and Ergonomics Society 37th Annual Meeting, Seattle, WA.

Zhai, S. (1995). Human Performance in Six Degree of Freedom Input Control. Ph.D.Thesis, University of Toronto. http://vered.rose.toronto.edu/people/shumin_dir/papers/PhD_Thesis/top_page.html

Zhai, S., Milgram, P. Buxton, W.,. (1996). The influence of muscle groups on performance of multiple degree of freedom input control. Proceedings of CHI'96: ACM Conference on Human Factors in Computing Systems.

Zhai, S., Milgram, P. (1998). Quantifying coordination in multiple DOF movement and its application to evaluating 6 DOF input devices. Proceedings of CHI'98: the ACM Conference on Human Factors in Computing Systems.

Zhai, S., & Milgram, P. (1993a). Human performance evaluation of isometric and elastic rate controllers in a 6 DOF tracking task. Proceedings of SPIE Vol.2057 Telemanipulator Technology, Boston, MA.

Zhai, S., & Milgram, P. (1993b). Human performance evaluation of manipulation schemes in virtual environments. Proceedings of VRAIS'93: the first IEEE Virtual Reality Annual International Symposium, Seattle, USA.