"Microsoft is Imagining a NUI Future". You can, too!

Microsoft is Imagining a NUI Future
Steve Clayton, Next at Microsoft Blog, 1/26/11


"Our research shows that the vast majority of people polled in both developed and emerging markets see great potential for NUI applications beyond entertainment. This is especially true in China and India, where 9 out of 10 respondents indicate they are likely to use NUI technology across a range of lifestyle areas – from work, education and healthcare, to social connections, entertainment and the environment. We believe that taking technology to the next billion can be aided by NUI – making technology more accessible and more intuitive to a wider audience". - Steve Clayton, Microsoft


The people at Microsoft don't own the concept!  I'm a member of the NUI Group (May, 2007) and SparkOn.  Both are on-line communities where you can find people who live and breathe NUI, learn about their work, and even share designs and code. If you are intrigued by NUI - as a designer, developer, or user, please join us.

Note: 
I've been an evangelist and cheerleader for the NUI cause for many years.  If you search this blog for "post-WIMP", "NUI", "multi-touch", "gesture", "off-the-desktop", "natural user interaction", "natural user interface", or even "DOOH", you'll be provided with an overwhelming number of posts that include videos, photographs, and links to NUI-related resources, including scholarly articles.  There is a small-but-growing number of people from many disciplines, quietly working on NUI-related projects.

RELATED
Microsoft Plans a Natural Interface Future Full of Gestures, Touchscreens, and Haptics
Kit Eaton, Fast Company, 1/26/112
Rethinking Computing (video)
Craig Mundie, Microsoft
Interactive Touch-Screen Technology, Participatory Design, and "Getting It" - Revised
Touch Screen Interaction in Public Spaces:  Room for Improvement, if "every surface is to be a computer".

Exploring Post-WIMP Interaction- "Manual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input

The video tells it all.

From Ken Hinckley's YouTube Channel

Thanks to Tony Uhrick for the link!>

Interactive Touch-Screen Technology, Participatory Design,and "Getting It"....

PLEASE SEE THE UPDATED VERSION OF THIS POST:
Interactive Touch Screen Technology, Participatory Design, and "Getting It", Revisited

There's been some discussion over the reasons why so many people don't understand touch screen, or "surface" computing, even though research in this area has been going on for years.

As the new owner of the HP TouchSmart, I know that I get it.

The research I've conducted in this area suggests that people will "get-it" only if there is a strong commitment to develop touch-screen "surface" applications through a user-centered, participatory design process. In my view, this should incorporate principles of ethnography, and ensure that usability studies are conducted outside of the lab.

This approach was taken with Intel's Classmate PC. Intel has about 40 ethnographic researchers, and sent many of them to work with students and teachers in classrooms around the world. (A video regarding ethnographic research and the Intel Classmate project can be found near the end of this post.)



Where to start?
K-12 classrooms and media centers. Public libraries. Malls. Hospital lobbies and doctor's offices. Any waiting room. Staff lounges in medical centers, schools, and universities. Community festivities and events. Movie theater lobbies. Museums and other points of interests.

I believe we need to take a "touching is believing" approach.

Here are some thoughts:

When I try to explain my fascination with developing touch-screen interactive multimedia applications, (interactive whiteboards, multi-touch displays and tables, and the like), many of my friends and family members eyes glaze over. This is particularly true for people I know who are forty-ish or over.

Even if you are younger, if you never saw the cool technology demonstrated in the movie Minority Report, or if you have limited experience with video games, or if you haven't came within touching distance of an interactive whiteboard, the concept might be difficult to understand.

The reality?

Even people who have the opportunity to use surface computing technology on large screens do not take full advantage of it. Multi-touch screens are often used as single-touch screens, and interactive whiteboards in classrooms are often serve as expensive projector screens for teacher-controlled PowerPoint presentations.

Most importantly, there are few software developers who understand the surface computing approach, even with the popularity of the iPhone and iPod Touch. Most focus on traditional business-oriented or marketing applications, and have difficulty envisioning scenarios in which surface computing would be a welcome breath of fresh air.

Another factor is that not all people entrusted to market surface or touch screen computing fully understand it.


Despite a cool website showing off the goods, Microsoft's Surface multi-touch table has been slow to take off, limiting hope of bringing down the price tag to a price most families or schools could afford. (The picture above depicts an application for the Surface designed for health care professionals, not K-12 science education.)

Although you can't buy a Surface table for your family room, it is possible to buy a TouchSmart.
HP's TouchSmart website is engaging and highlights some examples of touch-screen interaction, but most people don't seem to know about it.

Unfortunately, you wouldn't have a clue that the HP TouchSmart exists browsing the aisles at Circuit City or Best Buy!

When I was shopping for my new TouchSmart, I noticed that from a distance, the TouchSmart looked just like the other larger flat-screen monitors filling up the aisles. The salespeople at both stores were not well-informed about the system. The only reason I knew bout the new TouchSmart was related to my obsession with interactive multimedia touch-screen applications- designing them, developing them, studying them, reading about them, blogging about them.... ; }

More thoughts:

After studying HCI (Human-Computer Interaction), and relating this knowledge to what I know as a psychologist, my hunch is that the "Window Icon Mouse Pointing-device" (WIMP) and keyboard input mind-set is embedded in our brains, to a certain extent. Like driving a car, it is something automatic and expected. This is true for users AND developers.

Think about it.

Suppose one day, you were told that you no longer were allowed to control your car by turning on the ignition, steering the wheel, or using your feet to accelerate, slow down, or stop the car! Instead, you needed to learn a new navigation, integration, and control system that involved waving your hands about and perhaps speaking a few commands.

For new drivers who'd never seen a car before, this new system would be user-friendly and intuitive. Perhaps it would be quite easy for 16-year-old kids to wrap their heads around this concept. For most of us, no. Imagine the disasters we would see on our streets and highways!

When we think about how newer technologies are introduced to people, we should keep this in mind.

In my mind, spreading the word about surface computing is not a "if you build it, they will come" phenomenon, like the iPhone. We can't ignore the broader picture.

From my middle-age woman's vantage point, I believe that it is important that the those involved with studying, developing, or marketing surface computing applications realize that many of us simply have no point of reference other than our experiences with ATMs, airline kiosks, supermarket self-serve lanes, and the like.

(The video clip at the very end of this post provides a good example of touch-screen technology gone wrong.)

Be aware that there are substantial numbers of people who might benefit from surface computing who prefer to avoid the ATMs, airline kiosks, and self-serve grocery shopping.

Realize that the collective experience with technology, in many cases, has not been too pretty. Many people have had such user-unfriendly experiences with productivity applications, forced upon them by their employers, that any interest or desire to explore emerging technologies has been zapped.

My own exposure to interactive "surface" related technology was somewhat accidental.

A few years ago, a huge box was deposited into the room I worked in a couple of days a week as a school psychologist at a middle school. After a week or so, I became curious, and found out that it was a SmartBoard. Until then (2002!), I did not know that interactive whiteboards existed.

The boxed remained unopened in the room for the entire school year, but no worry. I played with the only other SmartBoard in the school, and found a couple at the high school where I also worked. I hunted for all of the applications and interactive websites that I could find, and tried them out. That is when I was hooked. I could see all kinds of possibilities for interactive, engaging subject area learning activities. I could see the SmartBoards potential for music and art classes. With my own eyes, I saw how the SmartBoard engaged students with special needs in counseling activities.

(By the way, if you are working with middle school students, PBS Kid's ItsMyLife website activities work great on an interactive whiteboard.)

A few years have passed, and reflecting on all of my fun experiences with interactive whiteboards, with and without students, I now understand that many teachers still have had limited exposure to this technology.

This school year, many teachers are finding themselves teaching in classrooms recently outfitted with interactive whiteboards, scrambling along with educational technology staff development specialists, to figure out how it works best with various groups of students, and what sort of changes need to be made regarding instructional practice.

For the very first time, interactive whiteboards were installed in two classrooms at one of the schools I work at. One of the teachers I know thanked me for telling her about interactive whiteboards and sharing my resources and links.

If I hadn't let her know about this technology, she wouldn't have volunteered to have one installed in her classroom. It has transformed the way she teaches special needs students.

In the few months that she's used the whiteboard, I can see how much it has transformed the way the students learn. They are attentive, more communicative, and engaged. The students don't spend the whole day with the whiteboard - the interactive learning activities are woven into lessons at various times of the day, representing true technology integration.

Now let's see what happens when all-in-one touch-screen PC's are unleashed in our schools!

Some resources:
HP TouchSmart PC website, with demo
HP's TouchSmart YouTube videos
lm3labs (catchyoo, ubiq'window)
NUI Group (See member's links)
NextWindow
Fingertapps
thirteen23
SmartTechnologies
Perceptive Pixel - Jeff Hans
Microsoft Surface
iPhone
(More can be found by doing a search on this blog or The World Is My Interactive Interface.)

Value of ethnographic research:
Ethnographic Research Informed Intel's Classmate PC
"Intel looked closely at how students collaborate and move around in classroom environments. The new tablet feature was implemented so that the device would be more conducive to what Intel calls “micromobility”. Intel wants students to be able to carry around Classmate PCs in much the same way that they currently carry around paper and pencil." -via Putting People First and Ars Technica

The video below is from Intel's YouTube Channel. Information about Intel's approach to ethnographic research in classrooms during the development of the Classroom PC is highlighted. This approach uses participatory design and allows the set of applications developed for the Classmate PC to reflect the needs of local students and teachers. Schools from many different countries were included in this study.




FYI:

Need for Improvement: User-Unfriendly Information Kiosk Interactive Map

Here are some interesting pictures from lm3labs, which are in my interactive usability hall of fame:

Samsung's new Omnia SDG i900 was re-created in a much larger size, using lm3lab's Ubiq'window touchless technology.

For more about lm3labs, including several videoclips, take a look at one of my previous posts:
Lm3Labs, Nicolas Leoillot, and Multimedia Interaction

Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics

Digital Out of Home (DOOH): Screens Large and Small at the Mall (and some touch interactive Coke machines!)

I was at the Southpark Mall in Charlotte yesterday and noticed that screens of all sizes were everywhere I went. I happened to have my little HD video camera with me and thought I'd share what came across my path.

Most of what I saw wasn't too innovative or interactive. Many of the smaller video displays were located on the market karts in the main traffic areas of the mall. Scattered about the mall are cozy living-room like areas, with comfy couches, WiFi access, and in on spot, a few large-screen HD televisions, perfect for watching sports or the news while other members of your social/family network do some serious shopping. I especially liked the infomercial about North Carolina's beaches around Wilmington.

I wasn't too excited about the information display about the mall, which provides what looks like a version of the Southpark Mall website, shown at :44 on the first video clip.  Located about 20 feet from a static mall directory, not a single soul looked at the screen or used the keyboard and mouse to find out more about what the stores in the mall had to offer.  The static directory, on the other hand, had groups of people looking at it all of the time as I observed.  (I added screen-shots and pictures of the keyboard-and-mouse display near the end of this post.)

At the end of the first video clip, you'll see a new interactive touch-screen Coke vending machine, but the one featured at the end of the first video is out of order.

Not to worry. I stopped to rest in another area of the mall, and in right in my line of sight was another Coke machine, just as a young man was trying to figure out how to get a Coke out of the machine. It took him 93 seconds. That might not seem like too long, but if you watch the second video, you'll see that it was almost painful to watch.

As the young man finished purchasing his soda, a family with two young children were nearby and figured out that the display wasn't just for ads. The second video clip has a few shots of the younger child playing with the touch screen, and later on, his dad.  The little guy probably would have played with the touchable spinning Coke bottle for a long time! The dad commented, "They should have something like this for the home!", and mentioned that his kids liked the SMARTboards at school.

In my opinion, the interactive Coke Machine didn't know what it wanted to be.  An eye-catching, attention grabbing infomercial?   A useful interactive information display?   A fun toy to touch and play?  A system to make it difficult to quickly reach your goal of getting your thirst quenched, better to get the ads into your brain?


Marketers, designers, and developers, listen to this:


A lot of people still do not know about larger interactive touch screens. Even if they have an iPhone!
I told the parents about touch-enabled all-in-one PC's, touch-screen netbook/laptops, and the rumor that Apple might come out with a touch-screen tablet. They'd never heard of such things. This mall is very upscale, and the families that come to shop there have money.  They still can buy shoes at Nordstrom and drink specialty coffees, and of course, crowd around in the Apple Store.

The Videos


Note:  The participants in the following two videos gave permission for me to video. The videos were not staged.

A Young Guy and an Interactive Coke Machine


A Kid and an Interactive Coke Machine


RELATED
Below is a picture of the web-connected directory at the Southpark Mall from about a year ago.  No-one used it then, and at the time, the display was not working.  If you look closely, you'll see the keyboard and mouse set up.  Although this display contains a lot of information about the mall, via the web, it does not meet the needs of most shoppers, who travel in pairs, groups, families, and extended families during the holiday season. 

Below are two screen shots of the SouthPark website, which can be accessed by using the keyboard and mouse on the information display, as I previously mentioned.

In my opinion, there is enough screen space on the touch-screen Coke machine to provide interactive information about the mall.  Ripping content from the mall's website won't do, since it is text-based, boring, and oh-so WIMP-y!

Better yet, the mall should transform the large static directories into something useful, keeping in mind that most of the time it will need to support two or more people deciding where to go and what to do while they are at the mall.  Beam a mini-map of the mall to the shoppers to use on their iPhones/Smart phones, and give them a shopping advisor app while you are at it.

Psssst....
There are too many talking head screens in the mall.  Make them interactive, add some value, and see what might happen, especially if you want to target reluctant shoppers like myself.

For fun:
I Want the Giant iPhone! (Short Glimpse of the Apple Store)


RELATED
Coca-Cola Testing Interactive Vending Machines Patricia Odell, Promo, 4/2/09:

"Shoppers will come upon the units in high traffic locations and can use the large format touch screen displays to interact with and buy Coca-Cola products. People will also be learning about specials and promotions available at the mall and will be able to purchase the beverages using Simon Giftcards.
 

"The flat screen is set in the vending machine doors and is divided into three sections. The machines feature functionality similar to an iPhone. For example, the mid section of the screen is where people can buy drinks. Clicking on a product lets the shopper rotate the bottle to see the label. The top and bottom sections of the screen are used for running commercials for Coke and other Coca-Cola brands and for Simon Mall promos...This is just preliminary to see how the functionality goes," Coca-Cola spokesperson Ray Crockett said. Next-gen models of the machines will offer mobile phone downloads in the form of music files, ringtones and wallpaper, along with cashless payment and more, Coca-Cola said...The machines were first Introduced at the Summer Olympics last year in Beijing and one the Simon dTour...The new machines incorporate sight, sound and motion video to take the vending experience from transaction to true interaction,” Anthony Phillips, global brand manager at Coca-Cola said in a release. “We wanted the machines to be eye-catching in a way that would turn heads and command attention.” The new venders were developed by The Coca-Cola Co. in partnership with Samsung and interactive marketing agency Sapient".

Sapient Interactive Services
Sapient Interactive Mobile Group

Update:  Some links to Bill Gerba's blog posts:

Digital Signage Screen Placement: Modeling Consumer Behavior http://bit.ly/4oXPWM
Digital Signage Screen Placement: Angle, Height and Text Size http://bit.ly/7hG6NZ
Making great digital signage content: A quick reference guide http://bit.ly/74rNL5