Gestures on Mobile Devices

The introduction of the Multi-Touch technologyon mobile devices has changed the way people interact with these devices by introducing the gestures. The gesture metaphor allows a user to interact with the device with "natural" gestures that require the movement of the fingers on the screen.

(point your mouse or tap on the icons to get more info about gesturs)

Whereas embossed buttons, changing cursors and changing targets indicate possible interactions in desktop interfaces, the corresponding indicators for gestures and hence interactions are often missing in mobile devices. As a result, users are often unaware of less obvious interaction possibilities, and may therefore never notice certain functionalities, possibly becoming frustrated because they do not find the way to accomplish their tasks. We argue that gesture hinting is a means to solve this problem.

The gesture metaphor consequences.
CAUSES gestures are not really "natural" is not always obvious and easy to understand which gestures are available
PRIMARY CONSEQUENCES users may miss possible way of interacting and new functionalities users can be frustrated if they don't find a way to accomplish their tasks
SECONDARY CONSEQUENCES unpleasant user experience lead the user to renounce or leave the app

Current Solutions

Gesture hinting in current applications is made trough different methods. None of these examples could be applied in a general framework that can help the user with gesture hinting through all the systems and apps. However some examples show that these methods are really good in some situations.

In the following figure is presented a table that summarizes the good and bad aspects fo current solutions for gesture hinting in mobile devices. It follows a list of examples of current gesture hinting solutions in mobile apps.

In the table are summarized the pros and cons of the gesture hinting techniques in current applications.

This project doesn't have the aim to find a gesture hinting solution that will replace all these methods. These methods have reason to exist, and the solution proposed in the thesis should work along with them. The advantage with the proposed solution is that in any moment, wherever the user is, he can ask for gesture hinting in a quick and dynamic way.

Alchemy Concept

The Alchemy concept Lundgren and Hjulstrom suggests a possible approach to the gesture hinting that could be used as a framework through different apps and OSes. Alchemy is presented not as a perfect solution, but rather as a proof of concept to explore and discuss issues related to gesture hinting. Alchemy had at that point in time not been fully implemented and thus never been tested with users. Since a first part of the project is focused on the implementation and user testing of Alchemy concept, the next paragraph presents the concept and exposes some early pros and cons related to it, as stated by the authors.

“In short, our solution is to overlay semi-transparent gesture hints on top of the interface – any interface – which means that the design (graphics and code) can be used together with any application”
- Lundgren and Hjulstrom (2011).

The Alchemy solution proposes a dynamic gesture hinting system that use a set of semi-transparent icons that appear under the finger of the user, on top of the interface, that show which gestures are available for the target object under the user’s finger. For a detail description of Alchemy read the paper Alchemy: Dynamic Gesture Hinting for Mobile Devices.

Alchemy icons combined under the user finger On the left the full icon set used in the Alchemy; on the top right the icon that combine a tap, double tap and swipe; on the bottom right an image from Alchemy that shows the real use with the user’s finger on top.

To summarize and give a first overview of the pros and cons identified in the first analysis of the Alchemy concept I prepared the following table, where I listed the main design decisions described in the article and for each I explain some pros and cons.

Alchemy characteristics and relative pros and cons.
Characteristics Pros Cons
Touch and Hold - familiar for users - occlusions
- already used in apps and OSes
Icons Surround the User's Finger - close to the focus - occlusions
- combined icons
- size and color
- partially visible close to the edges
Idiomatic Icons - flexible
- extensible
- less intuitive
- require time to be learned
Combined Icons - small space
- fast once learned
- increase the cognitive workload
- require time to be lerned
- specifically designed for combined icons
- not extensible for more gestures

The main issues with Alchemy concern the occlusion problem and the use of combined icons. The touch and hold gesture, used to activate the gesture hinting mode, is familiar to the users, but has the two important disadvantages. The user holds his finger on the target and therefore occludes part of the screen. The gesture is already used in apps and OSes. The icons surround the user's finger, therefore the information is displayed close to the user's eyes focus. However this means that the finger occlude part of the icon. The choice of size and color could be problematic since the background could be not uniform. If the finger is close to one edge of the screen the icon can be only partially visible.

Alchemy icons combined under the user finger Alchemy icons combined under the user finger.

With Alchemy it is necessary using combined icons in order to show many available gestures at the same time. This will save space, but will require more effort for the users to understand the icons. The use of combined icons force the design of icons that could be combined together, making the concept less extensible for more gestures. Using idiomatic icons we have more freedom to design icons, and the icon set can be easily extended. However the downside is that the icons are less intuitive and need to be learned. For these reasons I started to explore new alternative ways to solve the gesture hinting problem.