For the Tech Curious: Multi-Gesture Net: A Multi-touch and Multi-gesture Research Blog

Laurence Muller, M.Sc. is a scientific programmer at the Universiteit van Amsterdam who develops scientific software for multi-touch devices. His blog, Multigesture.Net, provides good information regarding multi-touch and gesture interaction hardware and software applications.


Laurence links to the DYI tabletop computing bootcamp that was held at
IEEE Tabletops and Interactive Surfaces 2008. From there, you can find a linked list of the organizers of the events, and additional information.

Picture below is from MTC Multi-touch Console:


Here is a link to the group's libavg wiki that includes open-source code and "how-to" instructions.

If you are interested in multi-touch and multi-gesture computing from an academic point of view, Florian Echtler, of the Technische Universitat Munchen has a series of publications listed on his website. Here is the abstract of one of his papers. He is on the right track. I especially like the fact that he's thought about widget layers. (I have, too, but they are only sketches in my idea book.)

TICH: Tangible Interactive Surfaces for Collaboration between Humans (Sourceforge website, with links to libtisch.

F. Echtler, G. Klinker
A Multitouch Software Architecture
NordiCHI 2008: Using Bridges, 18-22 October, Lund, Sweden. (bib)

"In recent years, a large amount of software for multitouch interfaces with various degrees of similarity has been written. In order to improve interoperability, we aim to identify the common traits of these systems and present a layered software architecture which abstracts these similarities by defining common interfaces between successive layers. This provides developers with a unified view of the various types of multitouch hardware. Moreover, the layered architecture allows easy integration of existing software, as several alternative implementations for each layer can co-exist. Finally, we present our implementation of this architecture, consisting of hardware abstraction, calibration, event interpretation and widget layers."

Update from Multi-Touch Barcelona and Guten Touch: Multi-Touch Space Invaders

Multitouch Space Invaders Basic Demo from multitouch-barcelona on Vimeo..

For more sights and sounds from this group, visit the Red Bull Music Academy Barcelona 2008 website.

For more about multi-touch technology, including DYI instructions for creating multi-touch tables and displays, open-source code,and tutorials, visit the NUI Group website.

Multi-touch and Flash: Links to resources, revisiting Jeff Han's TED 2006 presentation

Despite the increase in interest in systems that support multi-touch, multi-user multimedia interaction, there is a need for creative, tech-savvy types to develop innovative applications. Why? This technology has the potential to make a powerful impact on how people learn, communicate, solve "big picture" problems, and do their various jobs.

CNN's Magic Wall was one of the first applications to gain the attention of the masses, as it was used as an interactive map during the US presidential election process. Touch-screen interaction gained even more notice after the recent SNL parody by Fred Amisen.

If you think about it, the multi-touch applications you see on the news aren't much different than what you'd get from a "single-touch" program.

Fancy, yes. Truly innovative, no.

Just imagine a 3D multi-touch, multi-user, multimedia version of Google Search. I did. I put my sketches in my idea book and hurt my brain thinking about how it could be coded.

Jeff Han, the man behind Perceptive Pixel and CNN's magic wall, had much more up his sleeve when he demonstrated his work at TED 2006. Even if you've previously seen this video, it is worth looking at again. (I've provided a link to the transcript below.)



Transcript of Jeff Han's TED 2006 Presentation

This video presentation had a transformational effect on me as I watched for the first time. Jeff Han brought to life ideas that were similar to my own as a beginning computer student thinking about collaborative educational games and multimedia applications that could be played on interactive whiteboards.

Here are some selected quotes from the video:

"I really really think this is gonna change- really change the way we interact with the machines from this point on."

"Again, the interface just disappears here. There's no manual. This is exactly what you kind of expect, especially if you haven't interacted with a computer before."

"Now, when you have initiatives like the hundred dollar laptop, I kind of cringe at the idea that we're gonna introduce a whole new generation of people to computing with kind of this standard mouse-and-windows pointer interface. This is something that I think is really the way we should be interacting with the machines from this point on. (applause)"

"Now this is going to be really important as we start getting to things like data visualization. For instance, I think we all really enjoyed Hans Rosling's talk, and he really emphasized the fact that I've been thinking about for a long time too, we have all this great data, but for some reason, it's just sitting there. We're not really accessing it. And one of the reasons why I think that is, is because of things like graphics- will be helped by things like graphics and visualization and inference tools. But I also think a big part of it is gonna be- starting to be able to have better interfaces, to be able to drill down into this kind of data, while still thinking about the big picture here."

So now what?

A recent post by "Alex", on the AFlex World blog discusses a few solutions. Alex had a chance to meet with Harry van der Veen and Pradeep George from the NUI Group, and Georg Kaindl, a multi-touch interaction designer from the Technical University of Vienna. The focus of the discussion was to come up with ideas to encourage Adobe/Flash designers and developers to learn more about multi-touch technology and interaction, and take steps to create innovative applications.

I especially like the following quote from the post:

"...A quick quote from our conversations: “When our children will walk up to a display, they will touch it and expect to do something.”

As a techie and a school psychologist, I see an immediate need for innovative applications. I know that there is a built-in market in the schools, at least for low-cost applications. Despite economic constraints, many school districts continue to invest in interactive whiteboards (IWB's). They are cropping up in preschool and K-12 settings, and teachers are searching for more than what's currently available.

Interactive, collaborative applications are needed in fields such as health care, patient education, finance & economics, urban planning, civil engineering, travel & tourism, museums & exhibitions, special events, entertainment, and more.

Smart Technologies, the company behind SmartBoards, has a new interactive multi-touch, multi-user table designed for K-6 education, the Smart Table. Hewlett Packard has several versions of the TouchSmart PC, which can support at least duo-touch, if not multi-touch, multi-user applications. There are numerous all-in-one large screen displays on the market that support multi-touch and multi-user interaction.

Quotes from Harry van der Veen, of Multitouch NL:

"In 10 years from now when a child walks up to a screen he expects it to be a multi-touch screen with which he can interact with by using gestures."

"...multi-touch screens will be as common as for children is the internet nowadays, as common as mobile phones are for us."


Here is a quote from a conversation I had with Spencer, who blogs at TeacherLED.

"It was interesting this week as I was in a classroom with a teacher who I've not worked with before... he had 2 students using the whiteboard who kept touching it together by mistake. The teacher, exasperated, said to himself, "Why can't they make these things to accept 2 touches without going crazy!"

Proof of the demand! I think you are right when teachers spot the limitations and then see the technology on visits to museums, that might stimulate demand."

Spencer creates cool interactive mini-applications, mostly for math, using Flash, that teachers (and students) love to use on interactive whiteboards. (He's interested in multi-touch, too.)


So what are we waiting for?!

Related:
Natural User Interface Europe AB meets Adobe
Georg's Touche Framework
NUI Group
TeacherLED
Interactive Touch-Screen Technology, Participatory Design, and "Getting It".
Hans Rosling's 2007 TED talk

RENCI at Duke University: Multi-Touch Collaborative Wall and Table utilizing TouchLib; More about UNC-C's Viz lab...

RENCI is a multi-disciplinary collaboration between several universities in North Carolina, with centers located at the Europa Center, Duke University, N.C. State, UNC Chapel Hill, East Carolina University, UNC-Asheville, UNC-Charlotte, and the Health Sciences Library at UNC-Chapel Hill. Many of the centers focus on visualization and collaborative technologies, and have been involved in multi-touch "surface" computing.

The pictures below are from the RENCI center at Duke University:

Duke Multi-Touch Collaborative Wall

The multi-touch wall is 13 x 5 feet, and utilizes six high-definition projectors, resulting in a combined resolution of 5760-2160, and supports multiple users. According to information on the RENCI website, the design is scalable and applicable to non-flat surfaces. The wall system runs on Windows and Linux.

(Photo by Josh Coyle)

(Photo by Josh Coyle)

DI, or Direct Illumination is used for touch detection in both the wall and the table for detecting touch. A separate instance of Touchlib runs for each of the 8 cameras used to detect touch. A gesture engine interprets the information about touches on the screen as gesture events. Each camera is handled separately for image processing and blob tracking tasks.

Graphics from the RENCI Vis Group Multi-Touch Blog

Here is cool picture of the "Multi-touch Calibration Device", which uses a built-in TouchLib utility.

Additional information can be found on the RENCI Vis Group Multi-Touch Blog.

FYI

Touchlib is a multi-touch development kit that can be found on the NUI-Group website.

"Touchlib is a library for creating multi-touch interaction surfaces. It handles tracking blobs of infrared light, and sends your programs these multi-touch events, such as 'finger down', 'finger moved', and 'finger released'. It includes a configuration app and a few demos to get you started, and will interace with most types of webcams and video capture devices. It currently works only under Windows but efforts are being made to port it to other platforms."

If you are interested in creating your own multi-touch table, the NUI-Group website and forums are a great place to start.

Related:

If you follow my blog, you probably know that I've taken several graduate courses at UNC-Charlotte. Some of my professors and a classmate or two have been involved in some exciting visualization research over the past year. (If you are serious about multi-touch and other visually-based applications, it is worth taking some time to familiarize yourself with visualization and interaction research.)

News from the UNC-Charlotte Vis Center:

At the University of North Carolina at Charlotte, RENCI is a collaboration between the UNC Charlotte Urban Institute, the Center for Applied Geographic Information Science, and the Charlotte Visualization Center.

11/06/2008
Robert Kosara’s group wins two awards at IEEE VisWeek Caroline Ziemkiewicz and Robert Kosara won Honorable Mention (the second highest award) at the IEEE InfoVis Conference for their paper, “The Shaping of Information by Visual Metaphors”. Also, Alex Godwin, Kosara’s student, won Best Poster for his submission, “Visual Data Mining of Unevenly-Spaced Event Sequences”.

The Vis Center is pretty fascinating, as you can see by the group of visitors at an open house.

If you are just as fascinated by this stuff as the guys in the picture, here are links to some recent papers by UNC-Charlotte faculty affiliated with the Vis Center:

The Shaping of Information by Visual Metaphors (Caroline Ziemkiewicz and Robert Kosara)

Evaluating the Relationship Between User Interaction and Financial Visual Analysis (Don Hyun Jeong, Wenwen Dou, Felsia Stukes, William Ribarsky, Heather Richter Lipford, Remco Chang)

Visual Analytics for Complex Concepts Using a Human Cognition Model (Tera Marie Green, William Ribarsky, and Brian Fisher)

Steven Sinofsky Discusses Multi-touch and the HP TouchSmart, Windows 7, and more at PDC2008

Steven Sinofsky at the PDC2008 Open Space


In this video, Steven discusses Windows 7 and confirms that Windows 7 works with the HP TouchSmart: "The hardware is multitouch." Sinofsky also discusses how there might be a need for multi-touch drivers for the HP TouchSmart.

Take the 25 minutes or so to watch the video. It is worth it, even the more technical aspects, including how events will be managed to minimize boot time.

Dude!
Sinofsky says you gotta say "dude!". Watch the video to get it!

New multi-touch "Gravitoy" application from NUI; Multi-touch Space Invaders from Barcelona; "Open source" touch-less multi-touch SDK on CodePlex

NUI (Natural User Interface) recently unveiled Gravitoy, a multi-touch physics application that offers two modes of interaction. Below is a picture of children engaged in collaborative play on an interactive table running Gravitoy in the Sandbox mode. The children can draw objects that interact using principles of physics. (Photos are from NUI co-founder Harry van der Veen's blog)



Below are table-shots of the Sandbox mode:



See the triangle?

The second mode of Gravitoy is "Stacks". Two players play a game of stacking objects of various sizes. The difficulty level can be adjusted by changing gravity options.



This would make science lab engaging for students!

In addition, NUI developed a 3d model viewer application, which looks like it would be fun for budding architects and designers.

Visit the NUI website to learn more about the company and NUI's partners.

Take a look at Multi-touch Space Invaders from Multi-touch Barcelona. It looks like a fun game to play during a rainy recess:

Multitouch Space Invaders! from Multitouch Barcelona (NUI-Group members) on Vimeo.

If you are interested in learning more about developing multi-touch applications and systems, visit the NUI Group website for opensource resources, code, and "how-to" support forums.

Video clips and links related to the work of NUI-Group members, such as Christian Moore, Harry van der Veen, and Seth Sandler, can be found on a previous post:
More Multi-touch from Members of NUI Group!




Touchless Multi-touch...
I recently learned of another "open-source" touch-less "multi-touch" resource. The Touchless SDK was developed for a college project by a Columbia University student, Mike Wasserman.
You will need Visual Studio 2008, the code, and a webcam to get started. From what I can tell, the code will work with in the XNA development environment.

The Touchless SDK website has links to a video demonstration, demo code, FAQs, and more. The discussion list has a lot of good information.

Mike Wasserman is now working at Microsoft Labs.

iPhone: Multitouch control of a laptop screen!

The Media Computing Group at RWTH-AACHEN University has developed an application to allow users to control a laptop via a multi-touch iPhone. Although the demo shows how to rotate and resize images, the "multiple views" section is interesting. Like an Etch-a-Sketch, you shake the iPhone to reset the images.

I've been toying with ideas about ways to create user-friendly interactions between screens of all sizes. This approach intrigues me.



You can find more information about the iPhone and Cocoa on the Multi-touch Framework website. The contact for this project is Stefan Hafeneger, a student assistant

Natural User Interface: Overview of multi-touch technology and application development by Harry van der Veen,- Business to Buttons

Harry van der Veen from Natural User Interface Europe AB, was one of the keynote speakers at the Business to Buttons: Designing for Effect conference, held in June 2008. In this presentation video, Harry discusses the past, present, and future of multi-touch technology, and reviews the importance of multi-touch over single touch displays. He also provides a good overview of gesture interaction, something that he researched when he was a student. This presentation includes several video examples of multi-touch applications in action.

The presentation is well worth the 30-minute view!

"Harry van der Veen is a Bachelor of Multimedia, derived from the Dutch education Communication, Multimedia and Design, focused on Interaction Design and Project Management. He is CEO, co-founder and co-owner of the Sweden based commercial company Natural User Interface Europe AB, which focuses on delivering standardized and customized multi-touch hardware / software solutions and services to the global market. In addition to that, he co-founded the NUIGroup community, which is the worlds largest online platform where a global network of people share their ideas and information in an open source community, focused on multi-touch hardware and software solutions."


NUIGroup Community


Harry van der Veen's blog

Natural User Interface Europe AB (Harry van der Veen's company)

NUIGroup Wiki: This wiki includes tutorials for developing multi-touch applications, building your own low-cost multi-touch table, and information about current projects that are in progress.

Related Information:

The Business to Buttons: Designing for Effect conference was held on June 12-12 in Malmo, Sweden, organized by Malmo University and inUse, a user experience consultancy. Partners in this conference included Adaptive Path, a product experience strategy and design company, Patrick W. Jordan, a design, marketing, and brand strategist, the cocktail, a user experience and interaction design studio, cooper, a product design company, and OresundIT, a non-profit network.

Don Norman, the author of books such as "Design of Everyday Things" and "The Design of Future Things", presented at this conference. Don Norman is one of the founding fathers of the Human-Computer Interaction and related fields, and is the co-founder of the Nielsen Norman Group, a consultant firm that helps company create human-centered products.

Videos of Don Norman's Presentations:
Emotional Design: Total User Experience
Cautious Cars and Cantankerous Kitchens

Other:
Business to Buttons 2008 Recorded Sessions
Business to Buttons 2008 Downloads

My posts about the work of NUI Group members:

Multi-Touch Plug-in for NASA World Wind?!

More Multitouch: NUI Group's Christopher Jette's multi-touch work featured in Engaget ; Croquet?

More Multi-Touch from members of the NUI group!

Multi-touch Crayon Physics from multitouch-barcelona, inspired by Crayon Physics by Kloonig Games

Cross Post: Seth Sandler's YouTube Video, "How to Make a Cheap Multi-touch Pad" goes viral

NUI-Group Member Bridger Maxwell Receives High School Science Fair Award for Multi-Touch Screen Project

Look, touch, listen, and play: Seth Sandler's interactive Audio Touch Table video; NUI Group and Google's Summer of Code

V12's Dual-Touch-screen Canova: Multi-touch could support applications for universal design, education, and assistive technology.

V12 Design is a company with offices in Italy and Australia. It is developing the second-generation version of Canova . Since the product will not be on the market for at least 16 months, no pictures of the newest version were available. The first version is depicted in the photos below:

Canova Generation 1#


(Photo via Gizmodo; depicts the first version of the dual LCD laptop.)

According to information from LaptopMag, Conova was..The first-generation model....was envisioned to transform from a sketch pad, to a writing pad with an electronic pen, to a newspaper."



From the Gizmodo AU website:

"An interview with Valerio Cometti, the founder and managing director of V12 Design, confirmed that the upcoming version would support multitouch input and that a microphone would be built into the design for voice commands."

With multi-touch and audio input, this device would be a godsend for young people with learning difficulties and others who require assistive technology. It would also be very useful for people in various fields of work. I assume that the Canova will come with speakers, a DVD player, a video camera, Blue-tooth and Wi-Fi capabilities.

I was excited about the new dual-touch screen laptops in development for the One Laptop Per Child project. I'm even more excited about the Canova. Hopefully, the company is devoting attention to usability studies with a wide range of people, using a variety of interactive demo applications.

I wonder if I can get my hands on a second-generation prototype of Canova to test out my touch-screen application "experiments". I'm bursting with ideas to try out, including gaming applications.

I want one now!

The second version will be developed by a US manufacturer. If anyone knows more about this leave a comment!

Related:
V12 Designs' Dual-Touchscreen Notebook Coming Within Two Years

V12 DEsign Dual-Touchscreen Notebook will make ebook readers drool

V12 Design Delivering Dual Touchscreen Laptop Within Two Years

More Multi-Touch from members of the NUI group!

It is always exciting to see what members of the NUI group are doing!

Here is a new video of a multi-touch creation by some of the members of the NUI group. Although this is a proof-of-concept example, it is fun to see how it is played out, using the little iPhone-like touch-pad widgets as a navigation tool for the large screen.


Read the "Multi-touch Goodness" article in Gizmodo of an interview with Christian Moore about this demo and his Lux open-source framework. (Christian is a colleague of Harry van der Veen, both members of the NUI group.)

Here is an excerpt from the interview:
"JD: Why Flash?
CM: Because it's fast to prototype in. However, the software is broken into several segments. One C++ application that tracks hands that talks to Flash... WPF... or another C++ app... and basically everything you can imagine. You can enable multitouch in any environment, like Cocoa."

High-resolution screen shots and additional information can be found on the nuiman website.

For my tech-minded readers:
I'm pretty sure that the C++ application that track hands and fingers in the video demo uses Touchlib, a library for creating multi-touch interaction. Touchlib can work with TUIO, a protocol for tabletop tangible user interfaces. Applications such as Flash and Processing support TUIO. For more information about TUIO, read "TUIO: A Protocol for Table-Top Tangible User Interfaces".
(Information from the NUI group website mentions that OpenCV, or Open Computer Vision Library, found on SourceForge, can support blog detection and tracking.)

The people behind TUIO are from the Reactable project, of the Music Technology Group at Pompeu Fabra University in Barcelona:

Here is my plug for the NUI group, once again!

"The NUI group, or Natural User Interface Group, is an interactive media group researching and creating open source machine sensing techniques to benefit artistic and educational applications.

We offer a collaborative environment for developers that are interested in learning and sharing new HCI (Human Computer Interaction) methods and concepts. This may include topics such as: voice/handwriting/gesture recognition, touch computing, computer vision, and information visualization."


FYI
I came across the Harry van der Veen of the NUI group in early 2007 when I was working on touch-screen projects for my HCI and Ubicomp classes, and I'm inspired by all of the creativity I've seen coming from this group.

If you'd like to see more demos, visit the Natural User Interface website, a commercial out-growth of Harry and his colleague's work, where you can view a reel that includes a few touch-screen games. I love the vision statement on this site:

"Technology should enable us to interact with computers, in the same way we interact with the real world; in a way which is natural to us, namely through gestures, expressions, movements, and manipulations. Our vision is to change the way people interact with computers."

Multi-touch Crayon Physics

Also posted on the TSHWI blog:
Watch how you can draw simple shapes that can instantly turn into a game!


Multitouch Crayon Physics from multitouch-barcelona on Vimeo.
For a better version of this video, see http://www.vimeo.com/980528 For more information, see the RXSurface blog post. On May 18, Multi touch crayon physics will be offered as an alphabeta opensource! The people behind RXSurface are members of the Natural User Interface (NUI) group:
"Natural User Interface or ~ NUI Group is an interactive media group researching and creating open source machine sensing techniques to benefit artistic and educational applications."

Seth Sandler's "How to Make a Cheap Multitouch Pad" YouTube video is going viral...

Seth Sandler, a member of the NUI group, has worked very hard at putting together low-cost multi-touch screen surfaces. Seth has a background in music, so his projects focus on interactive music applications. Over 315,000 people watched this video within the week that it was uploaded.

Thanks Seth, for sharing this vision with the world!



Seth's AudioTouch Blog: "An Interactive Multi-user, Multi-touch Musical Table and More"

Hint for high school teachers: This sort of thing would be a great project for an after-school technology club!

Link to video: Microsoft Research UK's multi-touch research team

"I think that the sort of grander vision is that 5 years to 10 years down the line, there will be lots of new displays and display technologies, and displays will become more ubiquitous and cheaper. What we're trying to do is actually explore what the interaction techniques might be on these displays, because not all of them will be connected to mice and keyboards." -Shahram Izadi, Microsoft Research UK


Direct link to the wmv file

"Inside MultiTouch: Team, Demo, Lab Tour"

Take a look at a post on the MSDN Channel 9 website about an interview of members of Microsoft Research UK's multi-touch team. The researchers on the video are Shahram Izadi, Alex Butler, and Steve Hodges. The video contains some interesting demos. This team's approach to multi-touch is different than approach taken by the Microsoft Surface team. This video is well worth the 30 minutes!

This technology would be great for interactive educational games and 3-D applications. Microsoft has plans for this technology to be used in the home and for shared applications.