Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics

Touching Windows 7 - Informative post, video, and comments from the Engineering Windows 7 blog

Touching Windows 7 post on the Engineering Windows 7 blog provides a good overview of Microsoft's work with touch and gesture interfaces over the past several years, building upon earlier work with the TabletPC.

There are quite a few opinions shared in the post's comment section. It is worth the read.

Below are two videos from the Windows 7 post:

Windows 7 Gestures
<a href="http://video.msn.com/?mkt=en-US&playlist=videoByUuids:uuids:891c68b3-a534-4159-b6b2-8e4ac56b6890&showPlaylist=true" target="_new" title="Windows 7 Touch Gestures">Video: Windows 7 Touch Gestures</a>

My comments will be forthcoming.

More for Multi-touch: NextWindow Plug-in for Natural User Interface's Snowflake Multi-touch Software -and more.

Those of you have an HP TouchSmart, Dell Studio One PC, or NextWindow displays might be interested in the new NUI plug-in that supports NUI Suite Snowflake software. Here are the features of the plugin, according to information from the Natural User Interface website:

• Detailed user manual included with FAQ
• Developed on fast and reliable C++ platform
• Intuitive
• Customizable
• Gesture recognition library
• TUIO/OSC (Open Sound Control) support (sending and receiving events)
• Low level API
• Hardware accelerated rendering
• Support for wide variety of media types
• Advanced window handler that supports scaling and rotation
• Suitable for Windows® XP and Windows® Vista (Mac OSX and Linux can be developed on request)
• Audio support
• Single, dual support
• Multi-threaded resource handler (For fast data visualization)

"NUI has partnered up with NextWindow™, an international leader in the development of optical multi-touch technology and the manufacturer of optical multi-touch screens, overlays and OEM touch components."

"NextWindow™'s integrated technology allows for natural and intuitive interaction of digital content on flat TFT, LCD and Plasma solutions."

"The NUI NextWindow™ plug-in can be used with any programming language that supports TUIO, i.e. C/C++/C#, Java, Flash, Python, VVVV etc, meaning that software developers can run their own applications on NextWindow™, utilizing the NUI NextWindow™ plug-in."

Comment:
I became a fan of NextWindow touch-screen displays in early 2007 when I worked on a couple of touch-screen projects in my HCI and Ubicomp classes at UNC-Charlotte.

I've been using my HP TouchSmart PC at work with students with disabilities. I'm experimenting with the NUI Suite SnowFlake on my TouchSmart, and found that interacting with the Particles application delighted students with severe autism. The activities provided opportunities to establish joint attention. I also noticed an increase in the number of vocalizations and/or verbalizations among the students. Of course, this was NOT a scientific study.

RELATED
Definition of Joint Attention from UConn:

"Joint Attention is the process of sharing one’s experience of observing an object or event, by following gaze or pointing gestures. It is critical for social development, language acquisition, cognitive development…"

Establishing joint attention is an important step in the development of social interaction skills among young people who have autism spectrum disorders.

More about joint attention:

Joint Attention Study Has Implication for Understanding Autism
Science Daily, 9/29/07

Asperger-Advice: Joint Attention

Autism Games: Joint Attention and Reciprocity

Why is joint attention a pivotal skill in autism?
Tony Charman

Philos Trans R Soc Lond B Biol Sci. 2003 February 28; 358(1430): 315–324. doi: 10.1098/rstb.2002.1199.

Ideum's High Resolution Multitouch Table for Museum Exhibits, running NUI's SnowFlake Software Package.

The Don Harrington Discovery Center in Amarillo, Texas, and the Vulcan Park and Museum in Birmingham, Alabama will both have large, high resolution multi-touch table exhibits provided by Ideum.

Below is a demonstration video from Ideum:



Ideum's Portfolio Site
Ideum's multi-touch, multi-user mashup, created with NUI Suite 1.0 Snowflake, a gesture recognition multi-touch software package.
Ideum's mt2 Table Information & Specs (pdf)

Ideum's mt2 Table was tested at a hands-on museum during the development process.

SPARSH: DYI demo of an open-source multi-touch table and applications by NUI-group members

The following video is a demonstration of "Sparsh", an interactive multi-touch FTIR table built in eight weeks by a group of engineering students in India. Most of the information regarding the hardware and software you see running on this low-cost system can be found on the open-source NUI-group website, forums, and wiki.


Sparsh Multitouch Display from anirudh on Vimeo.

I especially like the multi-touch DJ application!

For more information, view the posts related to the NUI group on this blog.

Sparsh Website

Children at the Surface Table: BETT 2009

The picture below was posted on the Shakeout blog, and shows children gathered around Microsoft's Surface at the recent BETT 2009 educational technology conference in the UK. Read "Interesting tidbits from BETT 2009" for more information.

Also posted on the TechPsych blog

More Multi-touch Multimedia: Video demonstration of applications created with Snowflake and Flash

This video showcases the work of Natural User Interface-AB, using NUI Suite 1.0 Snowflake and Flash.

Here is the plug from the company's website:
"Natural User Interface (NUI) is a Swedish innovative emerging technology company specializing in commercially available advanced multi-touch software, hardware and service solutions. NUI's solutions can convert an ordinary surface into an interactive, appealing and intelligent display that creates a stunning user experience."

For more information and links:

For Techies and the Tech Curious: Multi-touch/Gesture from the NUI-Group

Search this blog!

Touch Screen Interaction in Public Spaces: Room for Improvement, if "Every Surface is to be a Computer"

Via Technology Supported Human-World Interaction

With Win7's "multi-touch" capabilities, Bill Gate's statement that every surface will be a computer might come true, but from what I can see, we have a long way to go before we get it right!

One of the concerns I have about "touch" screen interaction in public spaces is that not much thought is given to the environmental factors surrounding the system. The system that is tested for the first time after installation might work just fine, but not when it is put to use under a variety of conditions or situations.

EXAMPLES (I've recycled a few of these from previous posts.)

Touch Screen Kiosk/Display in Upscale Shopping Area, St. Thomas(?)

I came across this system when I was walking around in St. Thomas with my husband. I wasn't sure what to make of it. There wasn't anything on the large display, and I couldn't tell from a short distance if the two smaller displays were active. I noticed that the display had an awning of sorts, so the fact that the display would be used in a sunny location had been considered. Unfortunately, the awning didn't solve this problem, as you can see from the pictures below:

CLOSER UP

As we approached the display, it was almost impossible to see what was on the screen. In fact, we weren't sure what the display was all about. It looked like something you'd see at an airport check-in, not in a lush tropical setting.

Below is a shot of my husband, trying to see what is on the screen. This photo shows exactly what the screen looked like at the time:

The next picture tells it all. You can see that there is some sort of alphabet on the right side of the screen, but the reflection of my husband is easier to see. We couldn't figure this one out.

The following clip demonstrates how environmental concerns, such as wind and "sideways" rain, can make an interactive display at a shopping center difficult to use. During this clip, you can hear me muttering about the difficulties I had interacting with the information on the touch-screen display. It is possible that the system wasn't working well because of the exposure to the elements.

WIND AND SIDEWAYS RAIN

You can see that the navigation system was confusing. At the end, the entire screen turned red, and I gave up.

The next video-clip is of a touch-screen map at the Cleveland Clinic hospital. I was with my elderly aunt and my sister, and we were trying to figure out how to get to a restaurant. We never found it.

Although the screen was easy to see, the application was... crazy!

In my opinion, an interactive wayfinding application should run on a larger display and support two or more people, since many hospital visitors often run in pairs or small groups.

Usability, Accessibility, and User Experience in a Win7 Environment

Microsoft's newest operating system, Win7 (beta), allows for multi-touch interaction and application development. From what I can tell from the news from the Consumer Electronic Show (CES 2009), touch and gesture interaction holds quite a bit of interest among consumers and application developers alike.

Hopefully people will think carefully about user experience and usability factors when developing the applications we'll be using in the not too distant future!

Because of my background as a school psychologist, I’m interested how touch/gesture applications can address accessibility and universal usability issues.

I wasn't able to locate user experience or usability information for Win7. Here are a few links from the Microsoft website that I think are worth reading:

Touch Interaction Guidelines

How to Design a Great User Experience

Designing with Windows Presentation Foundation

Accessibility

According to the research from Microsoft, ”more than half of all computer users experience difficulties or impairments related to accessibility, and are likely to benefit from the use of accessible technology. Moreover, approaching software design with the flexibility and inclusiveness that are the hallmarks of accessibility often results in overall improved usability and customer satisfaction.”

Regarding touch/gesture interaction, know that “19% (24.4 million) of computer users have a mild dexterity difficulty or impairment, and 5% (6.8 million) of computer users have a severe dexterity difficulty or impairment.”

Christian Moore, from the NUI-Group, shared this nice graphic that depicts the various fields that intersect or converge with interface design, which I think is useful when thinking about how we will design, develop, and use emerging technologies:



Concepts such as accessibility and universal usability can fit into this design!

For more information regarding multi-touch, see my previous post, "For Techies and the Tech-Curious: Multi-touch/Gesture from the NUI Group"

Tech-Savvy Teachers at Classroom 2.0; Google Sketch-up for math, and other creative ideas...including multi-touch

Classroom 2.0

"Guzman", a member of Classroom 2.0, is a math teacher in Florence, Italy who teaches math to middle school students. In addition to math, he has an IT background. He used Google Sketchup to introduce and work with solids. He thinks that Sketchup would be a good tool to use to teach volumes, sections, and more.

Guzman's student's worked on this project one hour a week for five weeks, and created all of the models using Sketchup. The animation was created with Sketchup by Guzman.



You can find the models Guzman's students made at Google 3D Warehouse

Tom Barret, at Classroom 2.0 has started a Multi-Touch Interactive Desk Development group. Tom is involved in the SynergyNet: Multi-touch in Education research, which is part of Durham (UK) University's Technology-Enhanced Learning Research Group.

You can find related pictures on Flickr: Multi-Touch Interaction: Applications and Gesture Ideas

Multi-touch and Flash: Links to resources, revisiting Jeff Han's TED 2006 presentation

Despite the increase in interest in systems that support multi-touch, multi-user multimedia interaction, there is a need for creative, tech-savvy types to develop innovative applications. Why? This technology has the potential to make a powerful impact on how people learn, communicate, solve "big picture" problems, and do their various jobs.

CNN's Magic Wall was one of the first applications to gain the attention of the masses, as it was used as an interactive map during the US presidential election process. Touch-screen interaction gained even more notice after the recent SNL parody by Fred Amisen.

If you think about it, the multi-touch applications you see on the news aren't much different than what you'd get from a "single-touch" program.

Fancy, yes. Truly innovative, no.

Just imagine a 3D multi-touch, multi-user, multimedia version of Google Search. I did. I put my sketches in my idea book and hurt my brain thinking about how it could be coded.

Jeff Han, the man behind Perceptive Pixel and CNN's magic wall, had much more up his sleeve when he demonstrated his work at TED 2006. Even if you've previously seen this video, it is worth looking at again. (I've provided a link to the transcript below.)



Transcript of Jeff Han's TED 2006 Presentation

This video presentation had a transformational effect on me as I watched for the first time. Jeff Han brought to life ideas that were similar to my own as a beginning computer student thinking about collaborative educational games and multimedia applications that could be played on interactive whiteboards.

Here are some selected quotes from the video:

"I really really think this is gonna change- really change the way we interact with the machines from this point on."

"Again, the interface just disappears here. There's no manual. This is exactly what you kind of expect, especially if you haven't interacted with a computer before."

"Now, when you have initiatives like the hundred dollar laptop, I kind of cringe at the idea that we're gonna introduce a whole new generation of people to computing with kind of this standard mouse-and-windows pointer interface. This is something that I think is really the way we should be interacting with the machines from this point on. (applause)"

"Now this is going to be really important as we start getting to things like data visualization. For instance, I think we all really enjoyed Hans Rosling's talk, and he really emphasized the fact that I've been thinking about for a long time too, we have all this great data, but for some reason, it's just sitting there. We're not really accessing it. And one of the reasons why I think that is, is because of things like graphics- will be helped by things like graphics and visualization and inference tools. But I also think a big part of it is gonna be- starting to be able to have better interfaces, to be able to drill down into this kind of data, while still thinking about the big picture here."

So now what?

A recent post by "Alex", on the AFlex World blog discusses a few solutions. Alex had a chance to meet with Harry van der Veen and Pradeep George from the NUI Group, and Georg Kaindl, a multi-touch interaction designer from the Technical University of Vienna. The focus of the discussion was to come up with ideas to encourage Adobe/Flash designers and developers to learn more about multi-touch technology and interaction, and take steps to create innovative applications.

I especially like the following quote from the post:

"...A quick quote from our conversations: “When our children will walk up to a display, they will touch it and expect to do something.”

As a techie and a school psychologist, I see an immediate need for innovative applications. I know that there is a built-in market in the schools, at least for low-cost applications. Despite economic constraints, many school districts continue to invest in interactive whiteboards (IWB's). They are cropping up in preschool and K-12 settings, and teachers are searching for more than what's currently available.

Interactive, collaborative applications are needed in fields such as health care, patient education, finance & economics, urban planning, civil engineering, travel & tourism, museums & exhibitions, special events, entertainment, and more.

Smart Technologies, the company behind SmartBoards, has a new interactive multi-touch, multi-user table designed for K-6 education, the Smart Table. Hewlett Packard has several versions of the TouchSmart PC, which can support at least duo-touch, if not multi-touch, multi-user applications. There are numerous all-in-one large screen displays on the market that support multi-touch and multi-user interaction.

Quotes from Harry van der Veen, of Multitouch NL:

"In 10 years from now when a child walks up to a screen he expects it to be a multi-touch screen with which he can interact with by using gestures."

"...multi-touch screens will be as common as for children is the internet nowadays, as common as mobile phones are for us."


Here is a quote from a conversation I had with Spencer, who blogs at TeacherLED.

"It was interesting this week as I was in a classroom with a teacher who I've not worked with before... he had 2 students using the whiteboard who kept touching it together by mistake. The teacher, exasperated, said to himself, "Why can't they make these things to accept 2 touches without going crazy!"

Proof of the demand! I think you are right when teachers spot the limitations and then see the technology on visits to museums, that might stimulate demand."

Spencer creates cool interactive mini-applications, mostly for math, using Flash, that teachers (and students) love to use on interactive whiteboards. (He's interested in multi-touch, too.)


So what are we waiting for?!

Related:
Natural User Interface Europe AB meets Adobe
Georg's Touche Framework
NUI Group
TeacherLED
Interactive Touch-Screen Technology, Participatory Design, and "Getting It".
Hans Rosling's 2007 TED talk

HP to launch TouchSmart Developer's Kit! via Rich Brown at CNET Crave's Blog

If you follow my blog, you know that I have a TouchSmart, and that I'm a little obsessed about surface-gesture-touch-screen-multi-touch-multi-user computing....

Here is some good news!


According to a recent article from Rich Brown's Crave blog (CNET news), programming guidelines for the TouchSmart require developers to be know C# and Windows Presentation Foundation. That's great for me, since I can program in C# and I've been playing with WPF for nearly a year.



Despite what Rich Brown says, I know that the TouchSmart has at least duo-touch capabilities, since the touch-screen technology was provided by NextWindow, and the TouchSmart has been used for multi-touch demos created by FingerTapps. It is possible for the TouchSmart to handle input from multiple mice, too.

Hopefully Windows 7 will make it easier for developers to harness the power of the TouchSmart. I have so many ideas I'd like to see come to fruition.

If you go to the HP website, you can join the HP TouchSmart developer community.

Even more touch! Paper Four - Capacitive Touch Paper; NUI at DreamHack 2008...

When I first started writing about interactive touch-screen technology and multi-touch, there really wasn't much going on. Things have changed over the past couple of years.

Below is a video clip of Paper Four, via Touch User Interface, of a capacitive touch paper developed by researchers at the Mid Sweden University. Sensing electrodes are printed on the paper with conductive ink:


Natural User Interface will be demonstrating multi-touch applications at the upcoming DreamHack Winter 2008 festival November 27-30. DreamHack is known as the world's largest LAN party and computer festival. For more information, see my post on the Technology Supported Human World Interaction blog.

Multi-touch & Surface Computing Thoughts:

Despite the downturn in the global economy, I truly feel in my heart that collaborative multi-touch, or surface computing, will be an important "movement", on many levels. I think that it will enable people to gain a deeper understanding and enjoyment of ideas & concepts.

It is new way of expression and communication, with the potential to support learning, exploring, music, art, cooperation, gaming, and more effective information visualization & data-driven decision making.

Not too many people really "get" the concept of multi-touch - surface computing. Yet.

Those who do seem to be creative, forward thinking, thoughtful, and optimistic. Important characteristics to have during these trying times!

New multi-touch "Gravitoy" application from NUI; Multi-touch Space Invaders from Barcelona; "Open source" touch-less multi-touch SDK on CodePlex

NUI (Natural User Interface) recently unveiled Gravitoy, a multi-touch physics application that offers two modes of interaction. Below is a picture of children engaged in collaborative play on an interactive table running Gravitoy in the Sandbox mode. The children can draw objects that interact using principles of physics. (Photos are from NUI co-founder Harry van der Veen's blog)



Below are table-shots of the Sandbox mode:



See the triangle?

The second mode of Gravitoy is "Stacks". Two players play a game of stacking objects of various sizes. The difficulty level can be adjusted by changing gravity options.



This would make science lab engaging for students!

In addition, NUI developed a 3d model viewer application, which looks like it would be fun for budding architects and designers.

Visit the NUI website to learn more about the company and NUI's partners.

Take a look at Multi-touch Space Invaders from Multi-touch Barcelona. It looks like a fun game to play during a rainy recess:

Multitouch Space Invaders! from Multitouch Barcelona (NUI-Group members) on Vimeo.

If you are interested in learning more about developing multi-touch applications and systems, visit the NUI Group website for opensource resources, code, and "how-to" support forums.

Video clips and links related to the work of NUI-Group members, such as Christian Moore, Harry van der Veen, and Seth Sandler, can be found on a previous post:
More Multi-touch from Members of NUI Group!




Touchless Multi-touch...
I recently learned of another "open-source" touch-less "multi-touch" resource. The Touchless SDK was developed for a college project by a Columbia University student, Mike Wasserman.
You will need Visual Studio 2008, the code, and a webcam to get started. From what I can tell, the code will work with in the XNA development environment.

The Touchless SDK website has links to a video demonstration, demo code, FAQs, and more. The discussion list has a lot of good information.

Mike Wasserman is now working at Microsoft Labs.

Ideum's multi-touch, multi user table prototype

The above picture is from Jim Sapadaccini's post about Ideum's mutli-touch table prototype, using NUI (Natural User Interface) software, "Snowflake". Ideum is working on gesture-recognition software for use in future projects.

Ideum will be designing for museum installations, and will include user-testing as they work towards applications that will support collaboration among museum visitors as they interact around the exhibits.

For more information see the Ideum blog, and also Ideum's portfolio page.

Natural User Interface new website shares information about the company's innovative multi-touch solutions...

Frequent visitors to this blog know that I've been following the development of the NUI multi-touch system since it was in the gestational stage, a university project of Harry van der Veen. I'd like to mention that as the NUI multi-touch table has evolved, so has the NUI website.

Take a look at the user-friendly, visually appealing website that showcases the Natural User Interface company's accomplishments, new products, and services. Here is some information from the site:

• "Natural User Interface (NUI) is an innovative emerging technology company specializing in advanced multi-touch software and service solutions. NUI's solutions can convert an ordinary surface into an interactive, appealing and intelligent display that creates a stunning user experience."

• "NUI provides both standardized as well as customized horizontal, vertical and angled multi-touch hardware solutions. With the wide variety of market leading suppliers and partner, NUI gives warranty on all our products."

For those of you who can't wait to dive into programming multi-touch, collaborative applications, take a look at NUI's Snowflake 1.0 software for OEM partners:

• "NUI Suite 1.0 Snowflake is an easy to use, robust, fast and reliable gesture recognition, computer vision, image processing, motion sensing multi-touch software package.It has been tested and developed for over 1,5 year and valued as best in the industry by our global hardware partners."

Interactive User Interfaces: Nat Torkington's list of links to new and cool technologies.

Quick post:

Nat Torkington recently wrote a post on O'Reilly Radar with links and descriptions about new interactive interfaces, such as a multi-touch Rubik's Cube, a weather map with haptic feedback, NextWindow's multi-touch screens, Cyber Goggles, NUI, and more.