Interactive multi-touch for sound design, dj-ing, and music creation

NUI-Group member Christian Bannister, a musician, designer, and developer behind Subcycle Labs has been experimenting with music and multi-touch technology and interactions. Here's a demonstration of what he's come up with so far:


multi-touch the storm - interactive sound visuals - subcycle labs from christian bannister on Vimeo.
.
"The big picture goal of this project is to bridge the gap between sound visualization and musical instrument. With multi-touch interaction it is possible to manipulate multiple characteristics of a sound—directly and visually. Right now a lot of electronic music involves staring at the back of the performers laptop. This is a shame because in many cases a lot of really interesting things are happening on the computer that the audience is completely unaware of. This project hopes to create a common visual language and experience for the electronic musician and the audience by enhancing the perception of sound and music on both sides...These sketches are built with Processing and Max/MSP networked with OSC on a single computer..." -Christian Bannister



JazzMutant Lemur
The multi-touch tech company now known as Stantum evolved from JazzMutant, Jazzmutant was founded in 2002 by Guilluame Largillier and Pascal Joguet, and joined by Julien Olivier in 2003. The original focus was to create a multi-touch controller for music applications, and the Lemur was born. It is now in version 2, with features such as a gesture object that provides three ways for people to interact with sound, extended scripting abilities, and remote control of your computer's mouse cursor or keyboard.  Stantum recently developed a next-gen multi-touch screen system for use in mobile devices such as smart phones and netbooks.  (I'll discuss this further in a future post.)



Mapping Ableton Live to Jazz Mutant's multi-touch Lemur for sound design:


Additional tutorials can be found the Jazzmutant YouTube channel.


Ableton Live, the software used in the above video, will include Max. Max/MSP is now known as MAX 5:
"In use for over twenty years by performers, composers, artists, scientists, teachers, and students, Max is the way to make your computer do things that reflect your individual ideas and dreams. Version 5 is a dramatic transformation of Max that begins a new chapter in the program's history."

RELATED
Video:  Max for Live
Max is a product of cycling74
Cycling74 created the Make Controller Kit, which includes fully programmable controllers. The kit is networked based. It is capable of working with actuators and can read sensor information into Max.
Jamoma, a platform for interactive art-based research and performance. Jamoma is the prototyping environment for SpatDIF, the Spatial Sound Description Interchange Format, and GDIF, the Gesture Description Interchange Format.
GDIF: Gesture Description Interchange Format, a tool for music related movements, actions, and gestures 
Stantum's Mobile Phone Multi-touch Interface:  Demonstration of precise interactions on a resistive touch screen

How the Stantum's Resistive Multi-Touch Screen Works

Thomas Hansen's Multi-Touch Wisdom: "Windows Touch vs PyMT - Why programming on Windows is too complicated!"

If you are interested in designing or programming multi-touch programs and like elegant, concise code, you'll appreciate Thomas Hansen's recent blog post:


Windows Touch vs PyMT:  Why multi-touch programming on Windows is too complicated!


Here is an excerpt from Thomas Hansen's blog post:


"If you read the rest of the blog post, I’ll show you what I mean about context, and why e.g.  Windows Touch makes life difficult if you want to program multi-touch.  I’ll show you how to rewrite a windows touch example project (5 C# source files and > 400 lines of code) in Python using PyMT (1 source file with 12 lines of code).  Yes 12 lines, you read correctly (and then there is the whole thing about it just running on Linux or OSX as well…but we’ll leave that for another blog post)."


At the end of his post, Thomas makes a comment that I think is worth considering:


"The interaction paradigm is so revolutionary, I think we need to adopt our development tools more to it and explore the interaction space.  Instead I think people are jumping the gun on trying to standardize the interface while using the development paradigms we used for the GUI."


Well said!


Thomas Hanson is a member of the NUI Group.  He is a graduate student at the University of Iowa, pursuing his PhD in Computer Science, specifically Human Computer Interaction.  

Interactive Motion Graphics Showreel from Filmview Services - great content!

Here is a showreel from Filmview Services that simulates how tech-usability in an interactive gesture/touch world should be!



Here is a quote from the Filmview Services blog:


What Are Screen Graphics?

"...So it works out more cost effective for the films to actually have someone put the graphics on the screens for real. It also greatly enhances the performance of the actors. You only have to watch any of the Star Wars Eps 1-3 to see how wooden acting is when you don’t actually know what is in front of you. Actors love to be able push buttons and bang touch screens during their scenes. Having to actually do it in a certain order can stretch their capabilities mind you, and I am pretty gob smacked at how absolutely computer illiterate some of them are. Don’t they use email?


Anyway, due to this diminished ability to hit and bang things in any certain order, it is our job to make it impossible to mess things up. That’s why they are all genius typers. We make it so they can type any old thing and the letters still come out the way they are meant to each time. We also put little locking codes into our programming so they can’t accidentally escape the graphic mid job. It’s amazing how many of them can type the Esc button when they are meant to be spelling LOGIN."

Thanks, Tim!

SOMEWHAT RELATED
Coincidentally,  when I was visiting the NUI-Group forums this morning, I came across a link to Jakob Nielsen's "Usability in the Movies -- Top 10 Bloopers", which are worth taking a look at. I've posted the list, but you'll need to go to Nielson's web page to read the descriptions. You'll smile.

1. The Hero Can Immediately Use Any UI
2. Time Travelers Can Use Current Designs
3. The 3D UI
4. Integration is Easy, Data Interoperates
5. Access Denied/Access Granted
6. Big Fonts
7. Star Trek's Talking Computer
8. Remote Manipulators (Waldo Controls)
9. You've Got Mail is Always Good News
10."This is Unix, It's Easy"

Applications for touch, multi-touch, and gesture interaction: What's Marching to Market?

Every week or so, I hear about a new multi-touch enabled laptop.Today, I came across a review on Gizmodo about the Lenovo ThinkPad T400, which is a 14.1 capacitive touchscreen laptop that can handle four touches at once:

Lenovo ThinkPad T400s with Multitouch Review: Finger Flicking Fun


The Multitouch ThinkPad in Action

Lenovo ThinkPad T400s with Multitouch from Gizmodo on Vimeo.

SCREENSHOT
 
"The screen is pure beauty. Like a capacitive touchscreen phone, light taps on the display are responsive and you don't have to think about positioning your finger in a certain way to open apps or rearrange windows." - Joanna Stern, Gizmodo, 9/15/09


OTHER EXAMPLES
 
THE FUJITSU LIFEBOOK T5010

Add the Fujitsu LifeBook t5010 to the Windows 7 Multitouch Laptop List
Gizmodo, Joanna Stern, 9/15/09



DELL LATITUDE XT2
 

 












TOUCH DEMO- HP TOUCHSMART TX2 & WINDOWS 7


AIR HOCKEY FOR WINDOWS 7: IDENTITY MINE


FINGERTAPPS





Want to create something multi-touch?

It looks like there will be a growing need for Windows 7 application developers.  Here's some code examples using Windows Presentation Foundation and the Windows 7 Multitouch API

Windows 7 Multitouch Application Development, Part I (The Code Project: Kunal Chowdhury)


Other Developer Resources

Touchlib Multi-touch Development Kit  (NUI GROUP)

MSDN Windows Touch: Developer Resources

nTrig's Multi-Touch Download for Windows 7

NUITech's Snowflake Suite for NextWindow

TouchKit Multi-touch Development Kit


GestureWorks Flash Multitouch SDK

3M Multi-touch Developer Kit


RELATED
Touch-less Interaction from lm3labs

Create the future you want. with ubiq'window from Nicolas Loeillot on Vimeo.

More about gesture interaction and multi-touch can be found on this blog- do a search!

Ron George's Interaction Design Toolbox

Ron George, an interaction designer, recently posted a comment on my blog. So I visited his blog and found that he has a great set of resources that many of my readers would appreciate!

For starters, take a look at Ron's Interaction Design Toolbox page, and then spend some time browsing around the rest of his site/blog.

Ron works at Microsoft on a team that is exploring Natural User Interface/Interaction (NUI). He previously was on the Surface team.

MICROSOFT: ARE YOU LISTENING? Cool Cat Teacher (Vicki Davis) Tests Out Microsoft's Multi-touch Surface Table

When Microsoft unveiled the Surface in 2007, I was disappointed that there were no immediate plans to market it for use in education. At the time, there were no immediate plans to release the Surface for research and development by university students. I had just completed a course in human-computer interaction, and with my background as a school psychologist, I KNEW this sort of technology would work well in education.

My head was brimming with ideas for this innovative technology for use with the students I work with every day.

The following video shows a demonstration of Microsoft's multi-touch, multi-user Surface table at the 2009 NECC conference, and also provides insightful comments from Vicki Davis, author of the very popular Cool Cat Teacher blog. Vicki discusses the value of surface/tabletop computing in education and shares her views about the need for user involvement in the educational software development process. She also gives great advice about how Microsoft or other developers of tabletop computing systems should proceed.

I agree with Vicky's comments, 100%, as my regular blog readers know!

MICROSOFT, ARE YOU LISTENING?



In the above video, it is apparent that the musical instrument applications do not provide a good touch response on the Surface. Vicki suggests that touch responsiveness is key, and that all Surface applications should be held to the high standard of Apple's iPhone and iPod Touch. Vicky goes on to say that Microsoft should support easy development of applications, and ensure that applications are very user-friendly and easy for school folks to install. If you work in a K-12 setting, you know what I am talking about ; )

Vicky is preaching to the choir when she stresses that Microsoft R&D (and others) should involve users in ongoing development, in a meaningful way, by using REAL people, REAL teachers, people who work with students. If you have ever battled with a "lame" educational application, you know why this is so very important!

Vicky's enthusiasm for the use of tabletop/surface computing in education can not be ignored. She absolutely knows what she is talking about, and she is the instructional technology voice for a multitude of educators around the world.

Here is Vicky's plea:

"I wish Microsoft would listen to these 3 things from a teacher in a classroom (me). I know Microsoft has bigger things to do than watch this video, but, I can dream, can't I?"

(I've paraphrased the following quotes.)

1. Understand the amazing potential for Surface devices in education. Look at three to five years out. You are looking at the future.

(This technology can engage students who have ADHD, etc.)

2. Harness the power of your users! Pull in your users. There are so many people in education would give their thoughts for free!

3. Create virtual and online ways for Microsoft to interact with teachers.


"If Microsoft decides to invest in this, and I do hope that somebody watching this video will understand the importance of integrating the world around us into the learning experiences and the learning environment, as part as how we remake and re-do education."

My sentiments exactly!

A few thoughts:

So where are we now?

Smart Technologies has come out with the SMARTTable, but it was designed for younger students. At this point, there are very few options, especially affordable options, for educators of students in the upper grades to use this technology.

Some members of the NUI-group are involved in creating educational applications for table-top systems, but they are few in number.

A few companies are using this technology for education, but the applications are mostly limited to interactive museum exhibitions.

From my research on this topic, there are very few developers that have the interest or the inclination to create educational applications for table-top computing.

My hope is that this will change soon! Join me in this conversation.

Surface Flight Tracker Video from fboweb labs / flightwise.com, with background music by Art of Noise for your NUI pleasure.

This flight-tracker application for the Surface, looks fun to use. As I watched the video, I realized that it wasn't the application itself that I liked. It was the music that accompanied the video. The choice of music was from the 80's synth-pop band, Art of Noise

Since I'm a music lover, the music got me thinking.

Wouldn't it be great if productivity/work-related applications like Flight Tracker could be developed to provide a means for incorporating a sound-track?

Several thoughts and ideas flashed into my mind:

• Surface and related natural user interface/interaction (NUI) applications have the potential to transform routine, ho-hum work tasks into activities that are a bit more pleasant. Since people often listen to music while they work, it stands to reason that NUI productivity applications should incorporate a music component, at least as an option.
  

• To support a user-centered music platform for NUI applications, the application could incorporate a "smart" music library within the system, with the capability of integrating music libraries and playlists from user's mobile devices, as well as the web, effortlessly.(Of course, there are privacy/security and firewall issues to address, but that is another story.)
  

• Users could have a choice of listening to their own music playlists (including a shuffle option, selecting from a variety of presets, or go for something like the iTunes genius effect, listening to music generated from an algorithm that takes into account music preferences and user interaction with the productivity application over time.
  

• Since many Surface/NUI applications are designed to support collaborative work and interaction between two or more people, the music situation could get a bit complicated, since people have differing tastes. If co-workers disagreed about the music selection, the program would automatically default to generic elevator music, or silence.
  

• NUI applications might even pave the way for a new genre of music. This concept isn't too far-fetched. Think of all the music we've come to love over the years that was composed for movies and even video games!
  

If you know of anyone that is working on this concept, or would like to collaborate with me sometime in the future on this concept, please let me know. I'm slowly working on an interactive timeline prototype, and I have some ideas about adding a music/sound track component.

RELATED
Art of Noise - Close To The Edit (Version 1):

Multi-Touch Musical Instruments- Surface Editor: Post via the NUI-Group Forum

Multi-touch Everywhere Technology in Action

Below is the video from Future Instruments, a company formed through Geneva's Conservatory of Music and the University of Applied Sciences Western Switzerland. It is worth watching the entire video clip.

The following information was posted in the NUI-Group Forum:
"The research group at Geneva’s Conservatory of Music presents the Surface Editor and Surface Tracker software applications for turning ordinary surfaces into multi-touch musical instruments. The Surface Editor allows users to custom design interfaces by dragging and dropping components, such as buttons, sliders, keyboards, and many others, onto the interface. The surface can then be used to control digital audio workstations, virtual instruments, hardware synthesizers and samplers, VJ software, and other applications, via either the MIDI or OSC protocol. The Surface Tracker is a multi-touch tracking application created for tracking movements on a surface using high speed infrared OptiTrack cameras. These cameras have on-board image processing, which allows them to process 100 frames/sec while sparing the host computer’s CPU. The applications are available at http://www.surface-editor.com."

Here are more details from the Future Instruments website:

Surface Tracker

"The Surface Tracker is an open source application for low latency multi-touch finger tracking. It was designed initially for our Multi-Touch Everywhere system (MUTE), but it should work more generally as well to track fingers which have been illuminated using the laser light plane method. It only supports low latency OptiTrack cameras; these cameras have frame rates of 100 frames per second, but due to their onboard image processing, these cameras only consume 3-5% of your computer's CPU (depending on your processor)."

"The Surface Tracker is a standalone application, which sends Open Sound Control messages to any connected client application at a user defineable IP address and port. It implements the TUIO protocol, which was specially designed for transmitting multi-touch events on a table surface."

"The Surface Tracker is currently only available on Windows. This is due to the fact that the driver for OptiTrack camera's is currently only available on Windows. We are, however, currently having discussions with the makers of the OptiTrack camera about extending support of their cameras to OS-X."

You can download the Surface Tracker from the Future Instruments download page.

I wish this system was around when I was taking a computer music technology class!

RELATED
Greg Kellum's Website: Surface Editor

A Flexible Mapping Editor for Multi-touch Musical Instruments(pdf)
Greg Kellum, Alain Crevosier (9th International Conference for New Interfaces for Musical Expression - NIMES

Sneak Peek: Musical Instruments of the Future
Eliot Van Buskirk, Gadget Lab, Wired 2/25/09

NUI-Group Members: What are they doing now?

Multitouch Media Application Pro v3.0 from Falcon4ever on Vimeo.

MMA Pro is a multitouch photo and video organizer build in Adobe AIR (Flex3) and has new features such as Google Maps, support for uploading pictures on the fly using blue-tooth. For more information, visit Laurence Muller's website, Multigesture.Net. There you can download the application. Make sure you read the install instructions that are included in the readme.text, and also make sure that you have the latest Adobe AIR 1.5.x. Laurence also recommends installing BlueSoleil to handing the pairing of devices and file transfers. (If you've never programmed for Bluetooth, take his advice!)

Laurence Muller (M.Sc.) is a Scientific Programmer at the University of Amsterdam who develops scientific software for multi-touch devices. He is a member of the NUI-Group.

The following video highlights some of the applications from the University of Amsterdam from about a year ago:

Multitouch Applications from Falcon4ever on Vimeo.

Feel free to leave a comment and a link or two if you are a NUI-Group member and like to share your recent projects!

Lonely Planet Travel App by Amnesia-Razorfish for the Surface

I came across this video on the Microsoft Surface blog about a demo created for Lonely Planet by Amnesia-Razorfish, from REMIX 2009 Australia this past June:


Lonely Planet proof-of-concept at REMIX Australia 2009

Following a link from the Surface blog, I found yet another video:



Design in the Round. Creating Compelling User Experiences for Surface

The video provides an overview of the history of human computer interaction and look towards the future of NUI (natural user interface). The Surface is viewed as only as a step towards NUI, which follows a person as they go about the day, interacting with technology via a variety of devices and settings.

"Designing for multi-touch, gesture-based and tangible experience like Microsoft Surface presents a new set of challenges. How do you design for a user interface that doesn't have a top? How to allow for multiple simultaneous users without them getting in each other's way?..."

Human Media Lab's Organic User Interfaces and Interactive Sphere

The following video is of an interactive sphere prototype created by Josh Marble and John Bolton (members of NUI Group), from Human Media Lab at Queens University in Canada:

"DYI Multi-Touch Spherical Display"


I shared some information about the Human Media Lab's emerging technology in a previous post,
My Summer Emerging Technology "Wish to Play" List. Here is some more information about the lab:

"The Human Media Lab is Canada's premier media laboratory. Its mandate is to develop disruptive technologies and new ways of working with computers that are viable 10 to 20 years from now. We are currently working on the design of Organic User Interfaces (Oui!), an exciting new paradigm that allows computers to have any shape or form."

I would love to visit this lab. Perhaps they are working on flexible haptic interfaces, or other stuff of dreams.

(I'm on a cruise ship with limited Internet access, so I'll post more when I have the opportunity!)

Information about Touch Screens, Multi-Touch, & Gesture Interaction is Spreading

Since the news about Windows 7 multi-touch capabilities has spread around, I haven't had enough time to keep up all of information related to multi-touch interaction. Fortunately there are a few bloggers out there who are doing a great job filling in the gaps.

The Touch User Interface blog has a wealth of information in the form of pictures, video clips, slides, and links that I'd like to share.

The following slideshow/videos were highlighted in the Touch User Interface blog post, "Touch UI: HCI Viewpoint":

Untold Stories of Touch, Gesture, & NUI
Joe Fletcher, Design Manager, Microsoft Surface

Touch and Gesture Computing, What You Haven't Heard
Dan Saffer

Tap is the New Click

Other posts of interest on the Touch User Interface blog:
Touch screens and vision impairment
Link: Designing the Palm Pre: An Interview with Michelle Koh

Touch User Interface Overview

I've updated some additional information about UX,interactive multimedia, multi-touch, and gesture interaction on my Multimedia and Interaction Resources page, which is a work in progress.

MeeTTable: Kevin O'Mahoney's Multi-Touch Meeting Table Project

Kevin O' Mahoney is a multimedia developer who is working on his Masters in Media Design at Cork University in Ireland. The MeeTTable, pictured above, supports collaboration among 2 or more people who are developing 3D design projects. (Kevin is a member of the NUI-Group.)

More information about Kevin's project, include a video, can be found on his blog.

Future of Interaction: Microsoft Lab's Vision 2019 Montage; Future Healthcare Montage

The following video is from Microsoft Office Vision 2019. If you visit the YouTube source, you'll find an interesting discussion about this five-minute+ clip.



Video Presentation from MIX09: Untold Stories of Touch, Gesture, & NUI
Joseph Fletcher, Design Manager, Microsoft Surface


"Learn about new patterns, behaviors, and design approaches for touch and gesture interfaces from a practitioners point of view. Learn early lessons from applied knowledge of touch applications, devices, and design methods."



LukeW's notes from Joseph Fletcher's presentation
LukeW's post: Design Principles at Microsoft

LukeW is the Senior Director of Product Ideation & Design at Yahoo! Inc.

Rhizome 2009: A Lovely Interactive Multi-touch App on a Flexible Lycra Screen

Loran Bey is a member of the NUI group. He created Rhizome 2009 using Unity 3D and tBeta, now know as CCV (Community Core Vision). The screen in the video is made from flexible lycra, and this provides a tangible interaction effect. The music in the background is Aphix Twin's Avril 14th.

MultiTouch Screen Lycra from Loran Bey on Vimeo.

Unity 3D is a game development tool for browser-base games, including games optimized for the iPhone. (If you visit the Unity 3D website, be sure to download their 3D web plugin and visit their relaxing on-line Tropical Paradise.)

The screen displayed in the video was inspired by the 2005 Khronos Projector installation, by Alvero Cassinelli, an assistant professor at the Ishikawa Komuro Laboratory at the University of Tokyo. Khronos is described as a "video time-warping machine with a deformable screen."

The Khronos Projector website provides several simulation applets built with Processing that you can play with. They were fun to interact with on my HP TouchSmart PC. I liked Behind the Door the best.

The following video demonstrates how the Khronos application works:


Take a look at Alvaro Cassinelli's archive of his interactive media art if you have the chance!

Cassinelli's Meta-Perception research group is doing some interesting things, too:

"The goal of this group is to research methods for capturing and manipulating information that is normally inaccessible to humans and machines. In doing so, we hope to create new ways of perceiving the world and interacting with technology. Our research methods span fields such as human-computer interaction, media arts, physiology, and ethics."

Good news about the San Jose Interactive Displays Conference (that I couldn't attend)

Sadly, I was unable to attend the Interactive Displays conference that was held last week in San Jose, California. Jeff Han, the guy in the video clip in my sidebar, was one of the keynote presenters. I really wish I could have seen - and touched - the action!

Update (4/27/09) from Thomas Hansen, a member of NUI Group, who attended IDC:

"..It was certainly an interesting event. I got to talk (albeit briefly) to both Andy Wilson and Jeff Han. Both of whom produce very inspiring and world class UI/HCI research on a consistent basis. Further, it was a great honor to meet some of the other members of the nuigroup community in person, some of whom where inspiring not only because of their amazing intellect and artistic talents, but especially due to their friendliness, benevolence, and maturity."

Here are a few excerpts from people who were fortunate to attend and then write about the experience:

Putting our arms around the future of touch Ina Fried , CNET 4/23/09

"..if you used one of the interactive displays here to show a heat map of this industry, it would glow red hot. That's because touch displays, for years relegated to kiosks and industrial uses, are quickly becoming mainstream. Hewlett-Packard and Dell already have touch-capable machines, while Microsoft is set to make gesture input standard with Windows 7...For his own part, Han said he was inspired by seeing a PBS documentary in the early 1980s that showed Microsoft researcher Bill Buxton, then at the University of Toronto, using multitouch to compose music on a computer. The computer itself was a green screen with an ancient processor and little memory, but the key underlying concept was already there..

"Sometimes it takes that long for these things to marinate and gestate," Han said....And while things are now taking off, Han urged the crowd not to forsake quality in the rush to take advantage of a hot market. "That will ruin it and mess it up for all of us, and that would be a real shame," Han said."

Interactive Displays Conference Highlights Kevin Arthur, Touch Usability 4/22/09

"...He (Jeff Han) showed a great clip from an early 80s TV show called Bits and Bytes that featured a young Bill Buxton demonstrating some of his tablet work at the University of Toronto...Jeff Han's point in showing this clip was not just to share what inspired him as a kid to pursue computer science as a career. He also wanted to make the point that none of this stuff is really new. He urged the audience not to jump on whatever tech is cool this week, but to be aware of the history and to do the research. Be thoughtful and careful about what you're making. He said one of his fears with the multitouch craze is that the waters will be poisoned by bad and poorly conceived implementations. He said "don't add noise" to the ecosystem by using terms sloppily -- like "multitouch"...

The importance of being more thoughtful and mindful of prior work are not exactly new to most of us with design, HCI, or CS backgrounds, but the audience here is largely made up of marketing or other business types, I believe, who sometimes tend to get a bit carried away, you might say. I mean no disrespect to my friends in marketing..."

Interactive Displays Conference San Jose Harry van der Veen, 4/25/09
" Big thumbs up for Pira tech for managing to get so many multi-touch industry professionals (Wacom, Mindstorm, NextWindow, 3M, Jeff Han (Perceptive Pixel), LG, Tyco Electronics, Stantum and more) and hobbyists together."

From the NUITEQ (Natural User Interface) Gallery, via Harry van der Veen

Kids take to multi-touch interaction naturally!

The following photos are from Harry van der Veen's Multi-touch blog. (Harry was one of the founding members of the NUI-Group, and also is the CEO of NUITEQ-Natural User Interface)

The last two pictures are of the HP TouchSmart running NUI Suite Snowflake software, developed by the Natural User Interface Europe AB (NUITEQ) for think LCD, Plasma, and FT displays.

NUI-Group's Christian Moore Interview-Podcast on the Interactive Display 2009 Conference website

As I write this post, the 2009 Interactive Displays conference is underway in San Jose, California. The home page of the conference website provides a series of interviews of many of the conference participants and presenters, including Chris Moore, a member of the NUI group. Chris discusses the importance of the open-source NUI (natural user interface) community in moving interactive display and NUI-based technology forward.

Interactive Displays 2009 Conference Website

Interview Participants:
Mats Johansson - EON Reality
Guillaume Largillier - Stantum
Christian Moore- NUI Group
Brent Bushnell- Tapcode
Steven Bathiche - Microsoft
Jerry Bertrand - Microscent
Henry Kaufman- Tactable

If you are at the conference and you have some video clips to share of a presentation, demonstration, or anything else that might be of interest, please give me the link so I can share it on my blog.

Below are some pictures from the companies represented by the people who were interviewed for the Interactive Displays 2009 conference:


Eon Reality


Tactable


Microsoft's Surface


NUI-Group


Tapcode


I don't usually put company logos on my blog, but if you are a visiting my blog searching for links to companies that are involved with interactive display technologies, this will save you some time. (I received no payment for this gesture.)

Pervasive Checkers on Microsoft's Surface: The Gamepack Video

In early 2007, before we knew of the existence of Microsoft's multi-touch surface, I worked on a "Pervasive Checkers" project with Johnny Hopkins, a classmate in my Ubiquitous Computing class. I'd previously worked with XNA Game Studio Express in an AI for Games class, and thought that it would be cool to make a casual checkers game on a multi-touch table that could be played in gathering spots such as coffee houses and neighborhood cafes.

Below is a screen shot of what we created using Inspiration software - (in the application, you can click on an item and it expands to reveal additional information.)

Two years later, and the Pervasive Checkers idea is reality - but I wasn't involved in the process. Checkers is one of the games that is included in a game-pack created specifically for the Surface.

Take a look at the video:



From Surface Computer News:

"The Microsoft Surface Games Pack is a clear illustration of where the Natural User Interface of Windows 7 has the potential to take games. Windows is traditionally the number 1 gaming platform around the world. With the introduction of the NUI, allowing players to literally have titles at their fingertips via touch, Windows 7 can kick the gaming experience up a notch. Provided that developers rise up to the challenge."

Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics