Tech articles

Evolution of Hand-Free Navigation/Control of Digital World 

Human-machine interaction has always been central to the design and manufacturing process. The most natural way of such an interaction can either be speaking to the machine or expect it to react according to visual observations of the user's body pose, gestures and postures. The most convenient features of human body to establish the human-machine interaction include hands and the head in particular. Hand assistance in the navigation is quite wide-spread in today's modern hand-held electronic devices. Yet, the facial expressions and the gaze point and direction could be more powerful navigation tools in the future. Although the idea may look like a straightforward  extension to hand controlled navigation paradigms, the essential challenge lies in precision and accuracy of algorithm and overall system designs that could make things work properly.


The way we manage our electronic devices on the digital screens  navigation paradigm has alttered in so many different ways to this date. For example in back old days, people used to control their computers using keyboards through writing sequences of long lists of commands. The trend did not last long and is followed by the introduction of little arrow head. It was a single-touch utility that can choose and move icons and buttons on the screen. It was seen to be a revolutionary step at the time because a span of few pixels using wired mouses made it possible to run programs on computers that saved usrers from writing long list of codes using  their keyboards. The "arrow" is now everywhere on our digital electronic screens. Yet, the traditional arrow head has gone through many dramatic modifications since its first appearence. This article is aimed to illustrate some of historical evolutionary  steps of navigation paradigm taken in the past years and mention about the very likely  future possibilities.

Evolution of Multi-touch technology

Graphical User Interface (GUI) has become the inevitable part of any electronic device with a color digital display. After evolving from command-line prompt for text-based user interfaces, the moving ball and later the arrow become a universal pointing sign that floats over the buttons of nice glossy GUIs. Although, most of the navigation is handled by the movements of the arrow head, today's most popular operating systems are hybrid navigation designs in which the arrow is being used in conjunction with keyboard commands. But the current trend is that the need for the use of keyboard decreases as more easier-to-control gadgets are utilized such as wired and wireless mouse. 

Apple's leopard has a very nice and glossy look with smooth arrowhead navigation
which makes it fun to use the computer
In what follows after these introductory steps of navigation history, the  appearance of GUIs has  changed dramatically and accompanied by improved visual quality. As the display technology has evolved, the gloosy look and alluring  bright control panels and buttons made the navigation tools fun to use. Additional capabilities are later added to the arrow pointer in the following years of this evolution. For example, the arrow head become handy to use in standard key stroke applications and can even be used instead of the keyboard buttons saving the unexperienced user from reaching out the keyboard and looking for the right key.  However, people begin to have difficulties with using arrow heads as they desired to have easier access to their favorite applications and control over their hand-held devices.  Some times they do not want to bother to arrange the arrow location precisely over the small buttons. This could be time-consuming and tiring.  As the computers are sold in more compact and pocket size, the buttons become smaller and therefore, our magic assitant "the hand" or a plastic "stylus" become the primary control devices to navigate through the glossy look menus of these gadgets. The technology behind this innovation was not extraordinary as the resistance sensible-display technology made it possible to replace the arrow on the screen. Following years, small and clever software programming tricks opened up a new dimension in the navigation paradigm. Now we are able to do almost all the daily works in our hand-held computers, cell phones and PDAs through the help of rapid and easy hand movements using our fingertips.


Pressure sensitive screens are the next big thing in
the companies' agenda to provide more intuitive navigation. 
The resistive touchscreen technology is based on multiple layers of conductive material to detect pressure driven finger movements. Layer this older technology is replaced by newer capacitive touchscreen technology first commercialized in Iphones. In these electrically sensitive screens, there is no much of a need to press down a button on the screen since capacitive touchscreen displays detects electrical changes caused by a fingertip. Thus, this tech is more responsive and precise in terms of user-device interaction. As capacitive touch screen is very popular in those days, companies look for ways to use pressure sensitive  touchscreens with multi-touch capability.  This will introduce a third dimension to capacitive touchscreens as the user push harder on the screen, the pages will be turned faster or the movements get faster in a game. The intuition behind is straightforward.  As the use exert more force on the screen, pressure sensitive switch enables more current to flow unlike the capacity touchscreens where the pressure is sensed as on/off button and the skin is used to convey the electricity. Also, pressure sensitivity have a huge potential for a revolution in gaming on mobile devices. The third dimension will definitely make the games more exciting and provide a better user gaming interfaces that cannot be matched to older touchscreen technologies. With this pressure sensitivity can be enemurated as the current trend on navigation, the dependence upon our hands and fingertips is more pronounced.

Cool Stuff - Future of Navigation Paradigm


Iphone has pressure sensitive display that
allows hand to be  a handy tool in navigatio
The natural questions are to ask "what extent the hand can be used as a navigation tool?" and "what are the limitations that can bring forward the need for a new navigation paradigm?". First of all, let us consider what could be the limitations of hand movement-based control paradigm. First of all, if your hands are busy, you probably will not be able to manage things on the screen. Again, the handicapped who are unable to use their hands are not the primary customer group intended for such devices.  Although little impact it could have, the frequent rubs, dubs and having our hands in contact with the screen for a long time could be hazardous as the hands are the most frequent used limbs and could easily transport the germs around and be the source of infection for us and others as well. 

Dream about the scenario in which you get on your bicycle and mount your iphone in some appropritate location of your bike so that you can see the screen clearly. You open your favorite radio station and start listening as you ride. You remember that you were supposed to e-mail somthing for your work mate and it would be late if you do so after you are done with your biking. You open up your e-mail inbox on web and compose her back by simply using your head and iris movements maybe with occasional hand assistance. It might look a fantasy right now, but with the growing iris and head/face tracking algorithms, this would be a near-future objective of the new navigation paradigm. Its commercialization might not even be too far off.  Let us elaborate on how this could be possible with the current technology and associated challenges it could bring.. The number applications of this near-future technology is  abundant and it can be utilized in almost every single piece of our daily life. 


Methodology


Hand-free navigation technology is based on face morphology change and iris detection & tracking. It has two engines that tracks the the face movements and iris location in the eye in real time. As the head swings from left to right, the tracking result gives the location of a polygon that encapsulates the location of the face. The poylgon can shrink and enlarge depending on the motion of the face. On the other hand, iris location in the eye is being tracked to extract the gaze direction information. These tracking engines provides some of the inputs to the next stage of information processing such as the coordinates of eye ball in the eye and the angle of head movement with respect to the cameras horizontal axis etc. Based on the inputs of tracking algorithms processing block makes a decision and creates the necessary inputs to control the GUI menu. For example moving face with the right gaze direction in perpendicular with the cam axis can produce an effect of turning the page of a document. 

Facial feature, iris and bounding box tracking can be instrumental in hand-free navigation systems
There are many different face tracking algorithms in the literature, yet for efficient design the best application-specific algorithm should be utilized. For example, a simple affine motion model based on a square geometrical shape is shown in the above figure. As can be seen as the head swings right, the shape of the square changes producing different motion parameters. Similarly, since the location of the iris or certain facial features give us information about the gaze direction of the subject, the location of the iris is tracked and processing unit make decisions about whether the subject is really looking at the screen or not. However, 3D object motion and associated changes in shade or possible occlusions can introduce irrecoverable errors in feature tracking algorithms unless some other mechanisms interrupt and recovers the tracker from further error propagations. Thus the robust trackers should have an error recovery mechanism embedded in the over all system design. Finally, the processing unit decides on the next move based on the information obtained from both trackers that passed the error detection/recovery process succesfully. If the gaze direction changes suddenly, the navigation unit  disconnect itself from further user commands. 


As the screens of our generation are getting smaller, the navigation buttons become hard to distinguish and harder to manage. Smaller buttons necessitate more precise tracking and estimation. Precision can not be guaranteed most of the time depending on the image and video capture quality and/or the efficiency/complexity trade off of the algorithms run on the device. On the other hand, the hand-free navigation idea can be used in conjunction with pressure sensitive capacitive touchscreen technology to create a hybrid navigation system.   That way one can avoid some of the limitations of the current state-of-the-art tracking algorithms and camera technologies that can be manufactured into the cell phones.  


Head and gaze direction utilization can also revolutionalize the gaming on hand-held gaming consoles. For example, a first person shooter game can easily be controlled through the use of  head and iris movements. Consider a racing game, the car can be steered to right or left by aligning the car in the gaze direction through the movements of  our head. 

Conclusions


Most of the charm associated with new electronic devices of our era has somthing to do with the ergonomy  and the usableness of the human- machine interaction. This has been accomplished by traditional navigation tools in the past and renewed in current technology. Hand is observed to be an easier tool to meet humans navigation needs. Yet, the future technology will yield increased capabilities of even the smallest hand held devices to meet the human expectations. Ofcourse commercialization of such a technology requires time and effort to overcome some of the challenges that comes with the mechanics of new navigation paradigms. 



Related Links

Touch Screen Technology – How Does it Work?
3D touchscreens? Why Samsung think pressure-sensitive screens are the next big thing
              - Related tread*
Pressure Sensitive: A Mood Pad - Heat Sensitive Surface Review
PARATECH - website

ARTICLE 2:


Cooperation in Multimedia Networks: download here