Multi-touch and Flash: Links to resources, revisiting Jeff Han's TED 2006 presentation

Despite the increase in interest in systems that support multi-touch, multi-user multimedia interaction, there is a need for creative, tech-savvy types to develop innovative applications. Why? This technology has the potential to make a powerful impact on how people learn, communicate, solve "big picture" problems, and do their various jobs.

CNN's Magic Wall was one of the first applications to gain the attention of the masses, as it was used as an interactive map during the US presidential election process. Touch-screen interaction gained even more notice after the recent SNL parody by Fred Amisen.

If you think about it, the multi-touch applications you see on the news aren't much different than what you'd get from a "single-touch" program.

Fancy, yes. Truly innovative, no.

Just imagine a 3D multi-touch, multi-user, multimedia version of Google Search. I did. I put my sketches in my idea book and hurt my brain thinking about how it could be coded.

Jeff Han, the man behind Perceptive Pixel and CNN's magic wall, had much more up his sleeve when he demonstrated his work at TED 2006. Even if you've previously seen this video, it is worth looking at again. (I've provided a link to the transcript below.)



Transcript of Jeff Han's TED 2006 Presentation

This video presentation had a transformational effect on me as I watched for the first time. Jeff Han brought to life ideas that were similar to my own as a beginning computer student thinking about collaborative educational games and multimedia applications that could be played on interactive whiteboards.

Here are some selected quotes from the video:

"I really really think this is gonna change- really change the way we interact with the machines from this point on."

"Again, the interface just disappears here. There's no manual. This is exactly what you kind of expect, especially if you haven't interacted with a computer before."

"Now, when you have initiatives like the hundred dollar laptop, I kind of cringe at the idea that we're gonna introduce a whole new generation of people to computing with kind of this standard mouse-and-windows pointer interface. This is something that I think is really the way we should be interacting with the machines from this point on. (applause)"

"Now this is going to be really important as we start getting to things like data visualization. For instance, I think we all really enjoyed Hans Rosling's talk, and he really emphasized the fact that I've been thinking about for a long time too, we have all this great data, but for some reason, it's just sitting there. We're not really accessing it. And one of the reasons why I think that is, is because of things like graphics- will be helped by things like graphics and visualization and inference tools. But I also think a big part of it is gonna be- starting to be able to have better interfaces, to be able to drill down into this kind of data, while still thinking about the big picture here."

So now what?

A recent post by "Alex", on the AFlex World blog discusses a few solutions. Alex had a chance to meet with Harry van der Veen and Pradeep George from the NUI Group, and Georg Kaindl, a multi-touch interaction designer from the Technical University of Vienna. The focus of the discussion was to come up with ideas to encourage Adobe/Flash designers and developers to learn more about multi-touch technology and interaction, and take steps to create innovative applications.

I especially like the following quote from the post:

"...A quick quote from our conversations: “When our children will walk up to a display, they will touch it and expect to do something.”

As a techie and a school psychologist, I see an immediate need for innovative applications. I know that there is a built-in market in the schools, at least for low-cost applications. Despite economic constraints, many school districts continue to invest in interactive whiteboards (IWB's). They are cropping up in preschool and K-12 settings, and teachers are searching for more than what's currently available.

Interactive, collaborative applications are needed in fields such as health care, patient education, finance & economics, urban planning, civil engineering, travel & tourism, museums & exhibitions, special events, entertainment, and more.

Smart Technologies, the company behind SmartBoards, has a new interactive multi-touch, multi-user table designed for K-6 education, the Smart Table. Hewlett Packard has several versions of the TouchSmart PC, which can support at least duo-touch, if not multi-touch, multi-user applications. There are numerous all-in-one large screen displays on the market that support multi-touch and multi-user interaction.

Quotes from Harry van der Veen, of Multitouch NL:

"In 10 years from now when a child walks up to a screen he expects it to be a multi-touch screen with which he can interact with by using gestures."

"...multi-touch screens will be as common as for children is the internet nowadays, as common as mobile phones are for us."


Here is a quote from a conversation I had with Spencer, who blogs at TeacherLED.

"It was interesting this week as I was in a classroom with a teacher who I've not worked with before... he had 2 students using the whiteboard who kept touching it together by mistake. The teacher, exasperated, said to himself, "Why can't they make these things to accept 2 touches without going crazy!"

Proof of the demand! I think you are right when teachers spot the limitations and then see the technology on visits to museums, that might stimulate demand."

Spencer creates cool interactive mini-applications, mostly for math, using Flash, that teachers (and students) love to use on interactive whiteboards. (He's interested in multi-touch, too.)


So what are we waiting for?!

Related:
Natural User Interface Europe AB meets Adobe
Georg's Touche Framework
NUI Group
TeacherLED
Interactive Touch-Screen Technology, Participatory Design, and "Getting It".
Hans Rosling's 2007 TED talk

A new kind of SmartBoard: Multi-User Multi-Touch SMART Table for Education

I heard a rumor a few months ago that this was coming...and it was true!



The SMART Table "Touch, Learn, Together"

"The SMART Table interactive learning center lets students get hands-on with collaborative activities – and the more hands the merrier. The SMART Table is a versatile learning center where groups of students can create, explore and discover together on the durable screen. The interface is so intuitive that even the youngest child can play games and plunge into learning activities in no time. You can choose from a wide variety of ready-to-go lesson activities, or customize them to suit your own lessons."

SMARTTablerelease

If you are interested in developing for the SMART Table, visit the SMART Table Developer site.

Here are some of my other posts about multi-touch, surface, and table-top computing:

Interactive Touch-Screen Technology, Participatory Design, and "Getting It".

Emerging Interactive Technologies, Emerging Interactions, and Emerging Form Factors

The atracTable Multi-Touch System from Atracsys

Time for More Touch! NASA's collaborative multi-touch table by Gesturetek and Inhanced Digital Corporation; HP touch-screen notebook; NextWindow

New multi-touch "Gravitoy" application from NUI; Multi-touch Space Invaders from Barcelona; "Open source" touch-less multi-touch SDK on CodePlex

Natural User Interface new website shares information about the company's innovative multi-touch solutions...

Time for More Touch! Part Two: Microsoft's "Oahu", a hypothetical (?), affordable version of the Surface multi-touch table..

Creative Programming: openFrameworks - AWESOME for interactive multimedia applications

Ideum's Multi-touch, Multi-user Prototype

SynergyNet Project: Multi-touch tables and displays designed for education...

Rich White at Greenbush Labs is a strong supporter of interactive whiteboards and displays through his work with EduSim, an interactive 3-D virtual collaborative environment. Here is a link to one of his most recent posts about SynergyNet, a multi-touch table that is designed for use in education:

In the Classroom of the Future: Every Desk is an Interactive Surface



"SynergyNet will integrate ICT into the fabric of the classroom. The new desk with a ‘multi-touch’ surface will be the central component; the desks will be networked and linked to a main smartboard offering new opportunities for teaching and collaboration."

"Several students will be able to work together at a desk as the desks allow simultaneous screen contact by multiple users using fingers or pens. Durham researchers want to create a ‘natural way’ for students to use computers in class. The system encourages collaboration between students and teachers, and a move away from teacher-centric learning."

More about the SynergyNet project:

Smart desks make sci-fi a reality in the classroom

Research is underway at Durham University, located in the UK, to see how the interactive tables can support learning IT concepts. According to the SynergyNet project website, the goals of the research are as follows:

• Aim 1: To create a radically new technology-rich learning environment that integrates with traditional classroom layouts and collective activities.
• Aim 2: To design and implement a new form of user interface for educational multi-touch systems.
• Aim 3: To formulate a new pedagogy that eases transition and movement between teacher-centric and pupil-centric interaction.
• Aim 4: To analyse pupils’ learning strategies to inform fundamental research by capturing data as pupils use the SynergyNet environment.

"The system will also boost equal access in school. In IT, we have found that males have been the dominant actors - interactive classrooms will encourage more females to take part in lessons. It will also enable more disabled students to participate in lessons and allow more personalized learning." - Dr. Elizabeth Burd, Principle Investigator, Director of Active Learning in Computing

Related:
jME Physics - the open-source java-based physics engine used in the SynergyNet system

jMonkeyEngine - the open-source java-based game engine use in the SynergyNet system (the video below contains some clips from the SynergyNet project)

Pupils test multi-touch screens
(BBC Article)

More touch screen "surface" display musings...

I had my first chance to use an interactive touch-screen SmartBoard, by Smart Technologies in 2002-03. Since I work mostly with kids and teens, I wondered why large-display touch screen technology wasn't more widespread, since there are so many free, interactive websites that provide pretty engaging activities for users.

One of the things I learned was that large-display touch-screen technology is in the preschool stage. There are problems with screen responsiveness, screen resolution, durability, and input.

In recent years, the idea of a touch screen has evolved to table-tops and drafting boards, embedded within wireless systems that allow for interoperability with mobile devices and remote applications.

Great technology exists, but no-one has pulled all the components together in a way that can easily scale for the people who would benefit from this sort of technology the most - people who spend most of day time teaching, learning, or both. I had a great experience using a NextWindow Human Touch large-screen display for some of my projects last semester. It was difficult for me to track one down, but once I got my hands on it, I liked it, even though it did not have multi-touch capabilities.

One laptop for each child? That was a good idea for the late 1990's and early 2000's. One high-quality, affordable, large touch-screen display or table for each classroom would be more effective.

One touch-screen display/table for every 4-6 students would be better. Is there anyone out there who is up for the challenge?

Next Post: Updated links to interactive multimedia websites appropriate for large touch screen surfaces.