In Chapter 1 of Natural User Interfaces in .NET, Josh Blake asks and answers a question posed by many people who have been under the spell of keyboard input and GUI/ WIMP interaction:
Why bother switching from GUI to NUI? The answer? Read Chapter 1 (pdf) of the book - the chapter is free.
Here are a few of my personal reasons: 1. I want to buy the next version of the iPad or something like it. 2. I want to buy a new large-screen Internet HD TV.
3. I want to buy a Kinect.
4. I do NOT want to interact with my new TV with a Sony remote. Too many tiny buttons!
5. I do NOT want to interact with my new TV with a keyboard, because it reminds me of...work.
6. Most importantly:
I want to design apps for the people I care about, and others with similar needs:
My mom.
My grandson.
Moms and dads with kids in tow. People with special needs and/or health concerns, and the people who care and guide them. Knowledge sharers and (life-long) learners....
I love to dance- I studied dance through college, and off and on as an adult. I have a DDR (Dance Dance Revolution) game-floor pad somewhere in my attic gathering dust. I'm ready for new challenges.
I'm planning on buying a couple new dance games for the Wii and the Kinect. There is more to this story, given my interest off-the-desktop, post-WIMP HCI (human-computer interaction), interactive multimedia and games, and a career as a school psychologist dedicated to young people with disabilities, I'm excited to see where new technologies, interfaces, and interactions will take us.
So what do the wise men of usability have to say about new ways of interacting with games and other applications?
"Kinect has many great design elements that clearly show that the team (a) knows usability, (b) did user testing, and (c) had management support to prioritize usability improvements, even when they required extra development work." -Jakob Nielsen
Jakob Nielsen, one of the godfathers of usability, shared a few words of wisdom about the Kinect in his 12/27/10 Alertbox post: Kinect Gestural UI: First Impressions. Although he did not review Dance Central, he concludes that the game he reviewed, Kinect Adventures, was fun to play, despite usability problems.
If this is a topic that interests you, I recommend you read Neilsen's post, and also take a look at which are outlined in the post. Also take a look at recent essay Neilsen co-authored with Don Norman, another godfather of usability: Gestural Interfaces: A Step Backwards In Usability
Why is this topic important to me?
I have been involved in the Games for Health and Game Accessibility movement for many years. Lately I've been exploring the OpenKinect project with an aim to create ways of making movement-oriented games accessible for young people with more complex disabilities. For example, there is a need to have dance and movement games modified for students (and adults!) who need wheelchairs or walkers. There are students who have milder mobility challenges who love to dance, and the current games don't address their needs. Some of my students have vision or hearing impairments, too. They deserve a chance to play things designed for the Kinect.
"OpenKinect is an open community of people interested in making use of the amazing Xbox Kinect hardware with our PCs and other devices. We are working on free, open source libraries that will enable the Kinect to be used with Windows, Linux, and Mac."
Note: I currently work as a school psychologist with students up to age 22. My main office is adjacent to a large OT and PT room at Wolfe, a program for students who have special needs. We just had a large interactive whiteboard installed in the room that is begging for us to connect it with the school's Wii, and soon (we hope), a Kinect. If we are going to use dance games to help promote healthy activities among our special students, the games need to be accessible for students with cognitive, motor, and other limitations.
FIRST STEPS Although I can dance, I understand what the world is like through the eyes of many of the young people I work with who have motor coordination and sensory integration problems that interfere with their ability to move and dance, let alone access fast-paced dance games on the Wii or Kinect.
My initial plan is to look at what the new dance games might be like from the view of someone who doesn't know how to dance, and admits that they have "two left feet" - an perhaps, no sense of rhythm. Where would I start?
Wii's Just Dance2 seems to offer some support for learning how to dance through the use of simple movement icons, in the form of outlined figures, that provide information about how to move with the dancer on the screen. As you can see from the video below, the gamer is provided with information about upcoming moves throughout the game.
I decided to take a look at Just Dance2's MIKA "Big Girl" (You Are Beautiful) because some of the adolescent females I work with have weight concerns that interfere with their health. During the teen years, this can become a vicious cycle, resulting in less movement, and less participation with peers in physical activities, such as playing dance games. If a teen has depression as part of this mix, we know that exercise can help, and a fun dance game might be a life-saver, in more ways than one.
The screen shots below show how the movement icons are used in the game:
I thought it would be useful to learn more about the story behind the making of JustDance2. At 2:22, Alexia, the project's usability expert, makes her presence known. From what I can tell, she focused on aspects of the game that would make it more usable for non-dancers, including those with "two left feet", to play the game. (I don't know if there was anyone consulted about accessibility concerns for the game.)
Kinect Dance Central
Dance Central uses a different approach when it comes to "teaching" people how to dance along through the game. It would be interesting to test out Dance Central and JustDance 2 with the same set of people to get a better feel for what works and what doesn't. Below is a video that previews, in split-screen, the interaction that takes place in Dance Central:
Dance Central Full Motion Preview
In Dance Central, gamers are provided with information about the moves through icons that cycle up the right hand side of the screen. The level of dance-coordination to keep up with the moves is challenging at times, even for people who are OK at dancing. Players can select dances according to level of difficulty.
Kinect Usability with Regular People
Steve Cable (CX Partmers) shared his team's look at usability issues related to the Kinect by testing several games, including Dance Central, with groups of people in his article, "Designing for XBox Kinect - a usability study". The quote below is from the Steve's article:
"We’ve loved playing with the Kinect. There’s no doubt that the game play is lots of fun. In-game menus are a barrier to that fun. Kinect should allow players to move through menus quickly and compensate for inaccuracy.
We felt the Kinect would benefit from some standardised global controls – much like a controller uses the A button to select and the B button to move backwards. We also think it needs a more responsive pause gesture – one that doesn’t interfere with the user’s game play.
Most of our participants found the Dance Central menu to be more effective, more efficient and more satisfying to use. Here are our recommendations for designing a Kinect menu interface:
Allow users to make selections through positive gestures, rather than timed positions
Place options on a single axis to make them easier and quicker to select
Allow users to control menus with the game pad if they prefer
Use large easy to read text
Don’t make users scroll through options unnecessarily – it takes too long
Users will be distracted if used in a social setting – test your menus in a social context to see if they are prone to errors
Avoid the cursor metaphor, it’s not what gamers are used to seeing in game menus, and makes it harder to implement alternative joypad controls"
Below are screen shots that provide examples of how the movement icons are displayed in Dance Central:
I just got back from a meeting of the Charlotte UX Book Club. The group meets in the loft office of the Charlotte branch of Atlanta-based Macquarium, and provides a great opportunity for like-minded folks to socialize and toss around informed ideas.
Tonight's meeting featured Cennydd Bowles and James Box, the authors of Undercover User Experience. They joined the group via Skype, on a large screen TV. After a minute or so, it was as if they were right in the room, even though they were in the UK. What a delightful experience!
Cennydd and James both work at Clearleft, a team of designers and creative technologies that provide user-experience consulting. They are bright, insightful, and know their stuff. The book packs in a lot of good information in an easy-to-read, concise, and thought provoking manner. The author's address tools of the trade within the context of how work gets done across disciplines in organizations.
In my opinion, this book would be useful to people working on traditional projects as well as those who focus NUI design & development for emerging technologies.
BTW: Cennyd Bowles will be the closing plenary speaker at the IA Summit 2011 in March.
"Deliverables are a step on the journey, not the end of the line" -Undercover UX Design
Here are a couple of quotes from the Amazon.com editorial reviews:
Product Description
"Once you catch the user experience bug, the world changes. Doors open the wrong way, websites don't work, and companies don't seem to care. And while anyone can learn the UX remedies usability testing, personas, prototyping and so on unless your organization 'gets it', putting them into practice is trickier. Undercover User Experience is a pragmatic guide from the front lines, giving frank advice on making UX work in real companies with real problems. Readers will learn how to fit research, ideation, prototyping and testing into their daily workflow, and how to design good user experiences under the all-too-common constraints of time, budget and culture."
From the Back Cover
"Once you catch the user experience bug, the world changes. Doors open the wrong way, websites don't work, and companies don't seem to care. And while anyone can learn the UX remedies: usability testing, personas, prototyping and so on unless your organization 'gets it', putting them into practice is tricky. Undercover User Experience is a pragmatic guide from the front lines, giving frank advice on making UX work in real companies with real problems. Readers will learn how to fit research, ideation, prototyping and testing into their daily workflow, and how to design good user experiences under the all-too-common constraints of time, budget and culture."
"Our research shows that the vast majority of people polled in both developed and emerging markets see great potential for NUI applications beyond entertainment. This is especially true in China and India, where 9 out of 10 respondents indicate they are likely to use NUI technology across a range of lifestyle areas – from work, education and healthcare, to social connections, entertainment and the environment. We believe that taking technology to the next billion can be aided by NUI – making technology more accessible and more intuitive to a wider audience". - Steve Clayton, Microsoft
The people at Microsoft don't own the concept! I'm a member of the NUI Group (May, 2007) and SparkOn. Both are on-line communities where you can find people who live and breathe NUI, learn about their work, and even share designs and code. If you are intrigued by NUI - as a designer, developer, or user, please join us.
Note: I've been an evangelist and cheerleader for the NUI cause for many years. If you search this blog for "post-WIMP", "NUI", "multi-touch", "gesture", "off-the-desktop", "natural user interaction", "natural user interface", or even "DOOH", you'll be provided with an overwhelming number of posts that include videos, photographs, and links to NUI-related resources, including scholarly articles. There is a small-but-growing number of people from many disciplines, quietly working on NUI-related projects.
I came across this demonstration of I/O DataCenter's 12 by 7 foot interactive video wall that makes playing around with views of data center modules...interesting! The display is a gesture-based "multi-touch" system. (I'll update this post when I get more information.)
Here is the description from the Datacenter YouTube channel:
"Instead of hauling a 40-foot long modular data center to a trade show, i/o Data Centers is taking a high-tech approach to customer tours of their i/o Anywhere modular data center. The i/o team has created a 12-foot by 7-foot touchscreen video wall to provide interactive tours of the company's facilities. Selecting a "hot spot" pops up a virtual data center, complete with cross sections and product info, following the concept of the touch screens in the sci-fi movie "Minority Report.""
FYI: I/O Data Centers has an application that runs on the Surface.
Josh Blake recently interviewed Tamir Berliner, one of the founders of PrimeSense. If you haven't heard, Microsoft's Kinectwas based on work by PrimeSense, and licensed their technology. PrimeSense provides consumer electronics with natural user interaction capabilities. The good news is that the company recently released open-sourced middleware for natural interaction and depth-camera drivers. It will be interesting to see how this will play in the near future!
In the interview, Tamir discussed a number of topics related to postWIMP technologies. He also announced the newly created OpenNI, "an industry-led, not-for-profit organization formed to certify compatibility and interoperability of Natural Interaction (NI) devices, applications, and middleware." It is good to see this level of support for the cause!
Here is a quote from the interview that I especially liked:
"I believe that till today the devices we’ve been using, made us learn greatly lot about them before we could use them and gain their value. I’m pretty sure everyone who is reading this has got at least 3 remotes sitting on his living room table, and at least once a week needs to help someone use their computer/media center/phone/etc. It’s time for that to change and it’s up to us, the technologists to make this revolution happen, it’s time for the devices to take the step of understanding what we want and making sure we get that, even without asking if it’s a trivial task as opening a door when we approach, closing the lights when we leave the room, even making sure we have hot water to shower with when we return from work or wake up in the morning, depends on what we normally do." -Tamir
RELATED Here are a couple of videos from the OpenNI website that demonstrate OpenNI-compliant applications:
OpenNI-compliant real time skelton tracking by PrimeSense
OpenNI-compliant real time SceneAnalyzer by PrimeSense
FYI:
Josh Blake is the author of the Deconstructing the NUI blog. Over the past couple of years, he's explored natural user interfaces and interactions through his work on applications designed for Microsoft Surface and Win7 with Windows Presentation Foundation.
About a month ago, Josh organized OpenKinect, an on-line community to support collaboration among people interested in exploring ways to use Kinect with PCs and other devices. An example of this effort is the open source code, libfreenect, which includes drivers and libraries for Windows, Linux, and OS X.
This evening I followed a tweet to an article written by Ben Parr, of Mashable:
Google Acquires Some Powerful Video-Streaming and DRM Technology. According to Google, Widevine's adaptive video streaming technology monitors and adapts to bandwidth changes as it delivers content. This technology make accessing high-quality video content across the web more seamless, consistent, and convenient across platforms and locations, and is known as "TV Everywhere".
A range of technologies developed by Widevane in the recent past look like they will be of benefit to Google. Widevine's intellectual property portfolio covers a lot of ground. The patent claims distribution, as outlined on the company's website, includes realtime piracy detection and response fingerprinting, forensic watermarking, media tracking, evolving detectors (monitoring and response to piracy), security renewals, QOS, cross domain content security, secure processor technology, trusted computing technology, grooming/transcoding, DCAS (Downloadable Conditional Access System), device certificates, application level encryption, adaptive streaming, and usage controls.
Google will have a wide reach with the acquisition of Widevane, as it plans to continue the company's partnerships with the "entire ecosystem" of businesses related to digital video content in some way. As more people access web-based video from smart phones and related devices, and discover they can access video whenever they want, the demand for Google's cloud computing support will grow, along with the need for additional centers and support to handle the demand for multimedia content and related software applications.
The acquisition of Widevane might provide Google with a great deal of power over the next generation of cable/airwaves. If so, this will be a boon to advertisers, if done well. As it is, viewers must wait patiently to watch an ad for 15 minutes or so before viewing a short video clip on websites such as the Wall Street Journal. For some, this just a minor annoyance, and certainly not as bad as garish banner ads and pop-ups. Marketers will have additional opportunities to reach potential customers through the use of product placement/embedded ads when people access more longer-playing videos and movies on-the-go. The technology exists to create customized embedded ads in videos based on data collected about the viewer, which is right up Google's alley.
I'm working on some ideas for web-based interactive educational videos and other interactive multimedia applications designed to be accessed across various screens and devices. Technology is changing rapidly, and to move forward, I need to know more as I make decisions in the future. I'll return to this topic in future posts as I research this topic further.
Thanks to Pawel Solyga (Solydzajs) for the tweet that sent me down this rabbit hole ; )
FYI: Pawel is a software engineer, focused on next gen mobile apps. He is the VP of Technology at Numote. He also is a NUI Group co-founder.
Moms and dads with kids in tow.
