Touchless interaction â why the future is about letting go: part 1/2

Part 1 of 2

It seems like just the other day we were discussing the move away from the mouse to the touchscreen, but such is the current pace of technological change and innovation, that now the talk is of a ‘post touchscreen’ world. New technologies are pushing the boundaries of what is possible without touching or clicking an interface – this gives us the opportunity to create more natural-feeling interactions than ever before, in scenarios that wouldn’t have previously been possible.

This blog post outlines some of the main technologies and interactions on offer, and how we can all benefit from them today.

From GUI to NUI
Since the dawn of computing we have had to adapt our natural human behaviour in order to interact with computers. Hunching over and poking at buttons on our glowing mobile and desktop computer screens, interacting with a graphical user interface (GUI) has become synonymous with modern life.

But the GUI is a workaround, albeit a very clever one, that enables us to control a computing device. It’s a compromise between how a computing system works and what a human can easily understand; by abstracting the inner workings of a system into a set of buttons and other controls, we can command it to do what we want it to do. As much as interaction designers like myself try to ease the pain by designing user-friendly GUIs, controlling a computing device using a GUI still feels less natural than other activities in our day.

But what if computers and their software were designed to fit in more with natural human behaviour, rather than the other way around? For years this hasn’t really been possible, largely because of the limitations of technology, but now a new generation of sensors and intelligent software means that natural interactions are more possible than ever, and we no longer need to rely solely on touching or clicking a GUI on a screen.

Such seamless interactions are often referred to as natural user interfaces (NUIs), and they could just sound the death knell for the GUI.

Can’t touch this
Touchless interaction doesn’t just open up the possibilities for more natural interaction however, it also provides a way of interacting during those times when you can’t touch a GUI – perhaps you have messy hands in the kitchen, are driving a car, holding a baby or are a surgeon carrying out an operation. With touchless interfaces we are now able to control a device and enjoy its benefits in nearly any scenario, many of which would not have been practical before.

It also means that we can now interact with devices that have very small screens (where GUI interaction is particularly problematic), or even devices that have no screen at all. From talking to smart watches to making physical hand gestures at your TV, it’s the dawn of a new age of computing, where connected devices are even more integrated into our lives.

So how can we control a device without touching or clicking it? Witchcraft? Smoke and mirrors?

Most of the methods available involve sensors that are already built into modern smartphones, such as the camera, microphone, accelerometer, gyroscope, and wireless communication technologies such as Bluetooth or near field communication (NFC). Below is a rundown of four key opportunities for interacting with a device without touching or clicking it, and some examples of their implementation.

1. Voice
Talking – it’s one of the main methods that we use to communicate with each other, so interacting with a device using your voice seems a natural step for software designers. Human speech and language is highly complex and hard for a machine to interpret, so although it’s existed in its infancy for a long time, it’s only relatively recently that the technology has become a serious contender for touchless interaction.

When Apple acquired Siri and built it into the iPhone 4S, voice interaction came of age – for the first time, people seriously considered talking to their computing devices instead of touching them. What’s particularly interesting about Siri is that it uses a natural language user interface, allowing the user to have a conversation with the device.

Apple gives the following example: “When you ask ‘Any good burger joints around here?’ Siri will reply ‘I found a number of burger restaurants near you.’ Then you can say ‘Hmm. How about tacos?’ Siri remembers that you just asked about restaurants, so it will look for Mexican restaurants in the neighbourhood.” It allows for a more natural, conversational interaction. Although Siri isn’t perfect, it’s made huge inroads into that tricky beast: intelligent software capable of understanding human language.

Siri and other natural language interfaces such as Google Now move our relationship with computers away from typing long search queries and endless button clicking, and replace it with the kind of conversations that we might have with each other.

Voice control can be particularly useful in certain scenarios, for example, driving a car (Apple is working with various car manufacturers to integrate Siri into cars), in the kitchen when your hands are mucky (The Nigella Lawson recipe iPad app allows you to move through ingredients and recipe instructions with voice commands). As anyone who has used a touchscreen keyboard knows, it’s hard work and it’s easy to make mistakes, so voice input can offer a welcome relief from keyboard input – providing that it works flawlessly.

On the web, HTML5 introduces the Web Speech API, technology giving modern web browsers the capability of speech recognition. Combined with voice output from the system (iOS7 features the Web Speech Synthesis API), it’s now possible for a user to have a conversation with a website. The possibilities are endless, particularly on mobile, and uses like language translation and dictation are just the start.

The other growing importance of voice control is more extreme. As technology is getting ever smaller and more disseminated around our homes, there’s a need to control it in new ways. A device may be too small to have a screen or keyboard, and buttons may be impractical. For instance, Google Glass can be controlled through voice commands, which is, in theory, easier than fiddling around with the tiny buttons that live on the side of the device. Through the voice command “OK Glass”, the device starts listening and the user can then use natural language voice interactions to control the device.

Google has also employed this ‘always listening’ feature in a recent phone, the Moto X, which can be woken up from sleep and used freely with voice commands, without the user ever going near the device, as demonstrated in this advert. The growing trend for smart watches such as the Samsung Galaxy Gear and long-rumoured Apple iWatch further underlines how technology is getting ever smaller, and requires ever more sophisticated voice controls.

Voice-based interaction does have one downside which I feel is a barrier – using it in public. Personally, I feel very self-conscious telling my phone what to do, but I’m sure this is something that will change as it becomes more and more common.

2. Accelerometer and gyroscope
Two sensors that are built into nearly all smartphones and tablets are the three-axis accelerometer and the gyroscope, which can be used to detect rotation and movement of the device. Usually used to switch between portrait and landscape mode based on screen orientation, they also have the potential for touchless interaction. One example is the Google Maps mobile app – if you angrily shake your phone while using the app, it opens up a dialogue box asking if you would like to give feedback – presuming that something went wrong.

IOS has several accelerometer-based interactions built into the operating system, including ‘shake to undo’, where a shake movement gives the option of undoing the last thing you typed. Good gestures feel natural, as they relate to things that we do already, such as shaking your phone in frustration when it’s not working, or resetting something with a shake. More subtle uses of the accelerometer and gyroscope can also feel like a natural way of controlling it, eg tilting and rotating your device to steer in a driving game. Parallax.js reacts to the orientation of your device, allowing for tilt-based interactions in the web browser.

Read part 2

By DMA guest blogger James Reece, User Experience Specialist, The Real Adventure

This is an edited version of a blog that first appeared on The Real Adventure