Multi-touch, multimedia, multi-modal: Fujitsu LIFEBOOK T4310 has a multitude of possibilities!

Multi-touch, multi-media, multi-modal... the Fujitsu LIFEBOOK T4310 looks like it provides multiple possibilities for people from all walks of life:

-GestureWorks

The LIFEBOOK T4310 comes with an integrated web cam and fingerprint reader, and a variety of I/O options, including HDMI, USB, Firewire, BlueTooth,  LAN, analog video output, SD card reader, line in/out, a wireless switch for the integrated 3G and UMTS, and an express card reader. There is an optional modular bay that can accommodate an additional drive or battery.


The fun part is that the Fujitsu LIFEBOOK T4310 hands-on comes with Microsoft's Touch Pack applications, which are demonstrated in the video below:

• Microsoft Surface globe
• Surface Collage
• Microsoft Rebound touch game
• Surface Lagoon screensaver, which provides a water-ripple effect and little fish that respond to touch interaction.

As demonstrated in the video, the LIFEBOOK supports gesture interaction, multi-touch interaction, stylus interaction, and traditional keyboard interaction. The capacitive display has a bi-directional hinge, allowing it to be turned 360 degrees, and also positioned so that the display can be set facing up horizontally over the keyboard. (This feature would be welcomed in educational settings, if the notebook was used in education settings, as it could support paired and group collaborative learning activities.)


What I like about the LIFEBOOK is that it has an integrated ambient light sensor, which automatically adjusts the brightness of the display according to the level of light in the environment. This feature is important for people like me who are on the go and must use their computing devices under a range of lighting conditions.


I would love to get my hands on the Fujitsu LIFEBOOK T4310 for a month and test the system in-depth in my day-job as a school psychologist, and in my leisure pursuits as a UX/designer/developer/musician/gamer/etc,  "hobbyist".


Can anyone make this dream come true?


RELATED:
Fujitsu's multitouch LIFEBOOK T4310 tablet makes quick work of Microsoft's Touch Pack 
(Paul Miller, Engadget, 9/6/009)
Microsoft Touch Pack Gallery (Engadget)
Engadget's Reader Comments


I visited Fujitsu's website to gather additional information, and found a couple of press releases in German:


IFA 2009:  Fujitsu setzt auf Fingerspitzengefuhl und prasentiert  Notebook mit Multiple-Touch-Function (pdf)


Here is the press contact:


Fujitsu Technology Solutions
Melanie Wolf, Senior PR Manager
Tel.: + 49 (0) 89 62060 4458, Mobile: + 49 (0) 171 33 42 882,
Melanie.wolf@ts.fujitsu.com

How a multi-touch table is made: NUI Group members Aaron Bitler and Brady Simpson on HAK.5

If you want to know more about the inner workings of DYI multi-touch, you'll be interested in the video below.

Aaron Bitler and Brady Simpson learned how to build a multi-touch table through their participation in the NUI Group, and recently formed a company, 3M8 to distribute multi-touch solutions.  The 1/2 hour video provides an in-depth tour of multi-touch technology, in an an on-line broadcast on the HAK.5 website.


"Brady Simpson explains the different multi-touch methods. Aaron Bitler gives us a tour of the x19 multitouch tables construction. Brady explains the software used to power the x19 lcdmultitouch table, including the nuigroup CCV application. Aaron wraps up by explaining to us how he got involved in multitouch. He shares with us some sites where others can get started in their own homebrew projects, as well as his own multitouch company 3M8s at LCDMultiTouch.com."

UPDATE:  New website for Brady Simpson's company: http://www.lcdmt.com/

More Multi-Touch from members of the NUI group!

It is always exciting to see what members of the NUI group are doing!

Here is a new video of a multi-touch creation by some of the members of the NUI group. Although this is a proof-of-concept example, it is fun to see how it is played out, using the little iPhone-like touch-pad widgets as a navigation tool for the large screen.


Read the "Multi-touch Goodness" article in Gizmodo of an interview with Christian Moore about this demo and his Lux open-source framework. (Christian is a colleague of Harry van der Veen, both members of the NUI group.)

Here is an excerpt from the interview:
"JD: Why Flash?
CM: Because it's fast to prototype in. However, the software is broken into several segments. One C++ application that tracks hands that talks to Flash... WPF... or another C++ app... and basically everything you can imagine. You can enable multitouch in any environment, like Cocoa."

High-resolution screen shots and additional information can be found on the nuiman website.

For my tech-minded readers:
I'm pretty sure that the C++ application that track hands and fingers in the video demo uses Touchlib, a library for creating multi-touch interaction. Touchlib can work with TUIO, a protocol for tabletop tangible user interfaces. Applications such as Flash and Processing support TUIO. For more information about TUIO, read "TUIO: A Protocol for Table-Top Tangible User Interfaces".
(Information from the NUI group website mentions that OpenCV, or Open Computer Vision Library, found on SourceForge, can support blog detection and tracking.)

The people behind TUIO are from the Reactable project, of the Music Technology Group at Pompeu Fabra University in Barcelona:

Here is my plug for the NUI group, once again!

"The NUI group, or Natural User Interface Group, is an interactive media group researching and creating open source machine sensing techniques to benefit artistic and educational applications.

We offer a collaborative environment for developers that are interested in learning and sharing new HCI (Human Computer Interaction) methods and concepts. This may include topics such as: voice/handwriting/gesture recognition, touch computing, computer vision, and information visualization."


FYI
I came across the Harry van der Veen of the NUI group in early 2007 when I was working on touch-screen projects for my HCI and Ubicomp classes, and I'm inspired by all of the creativity I've seen coming from this group.

If you'd like to see more demos, visit the Natural User Interface website, a commercial out-growth of Harry and his colleague's work, where you can view a reel that includes a few touch-screen games. I love the vision statement on this site:

"Technology should enable us to interact with computers, in the same way we interact with the real world; in a way which is natural to us, namely through gestures, expressions, movements, and manipulations. Our vision is to change the way people interact with computers."

libTISCH, a multi-touch development framework with multi-touch widgets and more!

For techies and the tech-curious who like technologies that support collaboration and multi-touch interaction,  this is great news!

Florian Echtler announced the first stable releas of libTISCH, a multi-touch development framwork, which can be found on Sourceforge.  TISCH stands for Tangible Interaction Surfaces for Collaboration between Humans.  libTISCH, a C++ software framework, is included in this project.  It provides a means for creating GUIs based on multi-touch and/or tangible input devices.

Here is how it works:

Here is information from libTISCH announcement:


Highlights of this release are, among others, the following features:

- ready-to-use multitouch widgets based on OpenGL
- reconfigurable, hardware-independent gesture recognition engine
- support for widely used (move, scale, rotate..), pre-defined gestures
 as well as custom-defined gestures

- hardware drivers for FTIR, DI, Wiimote, DiamondTouch..
- TUIO converters: source and sink

- cross-platform: Linux, MacOS X, Windows (32 and 64 bit)
- cross-language: C++ with bindings for C#, Java, Python

libTISCH has a lot to offer for the multitouch developer. For example, 
the textured widgets enable rapid development of applications for many
kinds of multi-touch or tangible interfaces. The separate gesture
recognition engine allows the translation of a wide range of highly
configurable gestures into pre-defined or custom events which are then
acted on by the widgets. While the lower layers of libTISCH provide
functionality similar to tbeta, touche etc. (you can interface existing
TUIO-based software with libTISCH in both directions), it goes far
beyond.

More information about the library and underlying architecture can be
found on http://tisch.sf.net/ and in the Sourceforge wiki at
http://sourceforge.net/apps/mediawiki/tisch/

TISCH Project Wiki

RELATED
Florian is on the scientific staff at the Technisch Universitat Munchen in Germany. Be sure to check out his  webpage.

I especially like the concept of the MeTaTop: "A Multi-Sensory Table Top System for Medical Procedures" that is linked from Florian's website.

Want to make some multi-touch? Try PyMT- Python Multitouch. Featured in Make. (via Sharath Patali)

Sharath Patali, a member of the NUI-Group, has been working with Python Multitouch, otherwise known as PyMT, to create multi-touch applications.  He shared a link to a recent post in Make, featuring PyMT.  Sharath is the author of the UI Addict blog, and is currently doing his internship at NUITEQ (Natural User Interface Technologies).

I've been told that the beauty of PyMT is that it makes it "easy" to create multi-touch prototype applications using very few lines of code, which is great for trying out different ideas in a short period of time.  It helps if you already know Python!


PyMT - A post-WIMP Multi-Touch UI Toolkit from Thomas Hansen on Vimeo.

"PyMT is a python module for developing multi-touch enabled media rich applications. Currently the aim is to allow for quick and easy interaction design and rapid prototype development. PyMT is written in Python, based on pyglet toolkit."


PyMT Programming Guide


PyMT Website

Note: 
Christopher, author of The Space Station blog, is a member of the NUI-Group, and is building his own multi-touch table running his PyMT-based applications. Christopher is a student in Koblenz, Germany, studying computational visualistics, known as information visualization in the US.

Cyan Design's Multi-touch Experiments, Tutorials, and More (Collider Exhibition Series- Interactivity and New Media)

Chris Yanc, of Cyan Design, has been creating multi-touch applications for a while. His work, "Multi-touch Experiments" was included in the Collider Exhibition Series in Akron, Ohio. Below is a video of his experiments, a video of a demo app for an interactive touch conference map,  info about his work with the 36 Views of a Bridge project, and links to his tutorials and code.


Collider Series - Multitouch Table Experiments from Chris Yanc on Vimeo.


About the Collider Exhibition Series (info taken from the website):
"The Collider Exhibition Series examines the impact, implications and inspi- ration of the phenomenon generally categorized under the umbrella term New Media within the design practice and fine arts.Collider: Interactivity and New Media is an initial exploration into this realm. The exhibition seeks to provoke an awareness of the pervasive nature of New Media as it is applied in every function of our society and immerges as a forum for the highest expressions of our contemporary culture. It explores the results of collisions between humans and machines, biology and com- putations, art and technology, thought and process. What is New Media? And what are the implications to artists and designers when worlds, cul- tures and even identities collapse, build and collide."


Chris Yanc's Multi-Touch Demo App:  Tokyo Game Show 09 Conference Map

multitouch demo app - tokyo game show 09 conference map from Chris Yanc on Vimeo.

Chris was involved with the 36 Views of a Bridge at the Bridge Project in Cleveland, Ohio.  His post, 36 Views of a Bridge, explains in detail how his work was created, and also observations of groups of people interacting with the multi-touch table that was part of this project.

The sliding list widget demonstrated in the above video was created using Flash and can be downloaded from Chris's website: Sliding Lists ~ TUIO Flas App Widget.  The link will take you to "how-to" information, a code example,  and also to a link to download the widget.

Chris was kind enough to share his repository of tutorials and code:
http://www.cyancdesign.com/tutorials/   His tutorial page has lots of good links!

I'll be highlighting more work by innovative individuals, groups, and companies from time-to-time in future posts.

BP Oil Spill Multi-touch Map Mashup by Ideum

BP Gulf of Mexico Oil Spill Multi-touch Exhibit

In response to the BP oil spill in the Gulf of Mexico, the team at Ideum has created a free version of their multi-touch-enabled Google Map and Flickr mashup application to educational organizations, including museums and aquariums.  The information included on the map includes oil spill and fishing restriction data from the National Oceanic and Atmospheric Association (NOAA), along with pictures taken at the scene in and around the region of the spill.

Ideum is a company that "designs and creates interactive computer-based exhibits, multitouch installations, rich Internet applications, and social networking sites for museum and Web visitors. We work closely with museums, nonprofit organizations, and socially responsible companies to create memorable visitor experiences."   Ideum uses their own multi-touch framework called GestureWorks to create interactive applications.

Jim Spadaccini is the director and founder of Ideum. He formerly was the Director of Interactive Media at the Exploratorium in San Francisco.

Ideum is working on a version of the program optimized to work on 3M's  22" multi-touch system.  A single-touch version for the web might be available if there is interest.

RELATED

BP Oil Spill in the Gulf of Mexico Flickr Group

Ideum's oil spill-related pictures on Flickr

More from Multitouch Barcelona!

Multitouch Barcelona is up to more great work!

Multitouch Space Invaders XL

Multitouch Space Invaders XL from Multitouch Barcelona on Vimeo.

Guten Touch, by Multitouch Barcelona

GUTEN TOUCH from Multitouch Barcelona on Vimeo.

"Designed for the Red Bull Music Academy 08, Guten Touch is an interactive installation that involves people into a natural relationship with technology. A two projected display system plus a 3m x 2m multitouch wall showcase applications designed to engage us into human friendly experiences rather than flashy and jaw-dropping visualizations. Space Invaders hit by foam balls, pixel paintings created with brushes and digital objects held by hands try to blur boundaries between real and digital."

Multi-touch Barcelona (new site)
"Multitouch Barcelona is a recently born interaction design group that explores natural communication between people and technology. They design touch sensitive environments where real world interactions move to a digital context. Interfaces where people touch, play, move, feel...Where senses play their natural role, where everything just 'happens" as it does in the real world. "




RELATED
Offf 2009: International Festival for the Post-Digital Creation Culture

Update on Ideum: Multitouch, Multiuser Table at the California Science Center

Ideum is a company that has been around since 1999.  It focuses on the design and creation of "computer-based exhibits, multitouch installations, rich Internet applications, and social networking sites for museum and Web visitors."   Gestureworks is the software developed by Ideum, which is an authoring solution for Adobe Flash and Adobe Flex.

Below is some information, including a video clip, of one of Ideum's most recent installations a multiuser, multitouch table at the California Science Center.

Case Study:  Ideum's  L.A. Zone Multi-touch, Multiuser Table (Jim Spadaccini)

Here is an excerpt from Jim Spadaccini's post:
"This multiuser, multitouch table exhibit in the L.A. Zone allows visitors to explore a satellite image and map of the LA Basin and view overlays in thematic areas that explore fire, air, water, and earth. In addition, points of interest are placed on the map in the form of images and videos. All of the content and navigational controls are available in English and Spanish. The exhibit software runs on a custom-built 50" multitouch table that supports 60+ simultaneous points of input."

The Ideum team has spent a great deal of attention to the way groups of museum visitors interact with their multitouch applications. In the past, they've noticed that on a 50" surface, visitors would experience interaction conflicts. In the present application, more than one visitor can manipulate the content without changing the interaction of others.  Spadaccini points out that multi-user control of touch screen exhibits is new, so interface design concepts developed for single-user interaction are no longer useful.  Social interactions around a multiuser display are now something to consider.

In my opinion, what the Ideum team learns through their design process can be quite useful to those of us planning to develop multiuser educational applications for interactive surfaces and displays. What I liked about this exhibit is that it has a monitor that mirrors the display, so people can view the interaction from afar.  This feature would be useful in classroom settings that have an interactive table and an interactive whiteboard. The interactive whiteboard as the secondary display so the teacher and other students could see what the group at the table was exploring.



Multitouch Google Maps and Flickr Mashup Built with Flash



Update: A video showing how Ideum's multi-touch table can withstand all sorts of heavy-duty treatment:




Note:I have been pretty busy over the past couple of months- I have much that I haven't yet blogged about.  More to come!

Quick Link: 3M Invests in Perceptive Pixel, Jeff Han's Multitouch Tech Company

3M Invests in Perceptive Pixel

"3M, through its 3M New Ventures business, has invested in Perceptive Pixel Inc., a developer of advanced multi-touch solutions based in New York City. Terms of the transaction were not disclosed.Founded by multi-touch pioneer Jeff Han in 2006, Perceptive Pixel is dedicated to the research, development and production of multi-touch interfaces for the knowledge worker. The company's hardware and software products enable users to manipulate complex datasets through a new class of intuitive, powerful and visually rich interface techniques. The combination of its technologies with those of 3M will create incredible new opportunities for both companies."

"To see Perceptive Pixel multi-touch solutions in action on 3M Projected Capacitive Technology, see the video at http://www.3m.com/touchPPI. For more information about 3M MicroTouch products, visit www.3M.com/touch. For an overview of popular touch technologies and terminology, visit www.touchtopics.com."

RELATED
3M Touchscreen Products and Systems

Catching up with multitouch pioneer Jef Han Ina Fried, Cnet 10/22/10

Jeff Han's 2006 Ted Talk



Jeff Han, 2007



Thanks to Seth Sandler for the link!

Interactive Mobile Multimedia

A study in January of 2005 by Nokia indicated that there was a demand for interactive mobile multimedia services. That study was conducted about five years ago, before anyone had heard of the iPhone!   The technology to support interactive mobile multimedia has come a long way since then, and many of the new applications support multi-touch, or at least duo touch interactivity.

I'm very much interested in figuring out how to design web-based interactive content (and apps) that can be optimized for touch (and multi-touch/gesture) screens of various sizes, from SmartPhones/iPhones to the large interactive whiteboards that are now in a multitude of classrooms.

From my experience as a school psychologist, I know that there are many teens who have graduated from traditional cell phone to the next level.  If they don't have an iPhone, they have a smart phone.  I don't have the statistics on this, but my personal observations tell me that there are teens who come from families who are from lower economic status who are somehow able to own 3G smartphones.

What a great opportunity to provide casual interactive multimedia educational games to support student learning!  The games and activities could be assigned as homework from time to time, and with the appropriate LMS (Learning Management System), the teacher would have instant access to student progress.  In addition, the students would be provided with immediate feedback about their "work", which we know is an important factor in learning.

It is difficult to figure out the best path to forge, since nearly every week someone announces a new platform, technology, and programming approach!

At any rate, here are a few interesting things related to this topic that I'd like to share.  Many of these concepts are in the experimental phase, but are worth some attention.


SciLor's Open-Source Programs
SciLor's HD2/ Leo Multi-touch Demo v2 12/3/09 using VB.Net and Windows Mobile

SciLor's Comments: 
I have manged it to get Multitouch running in an vb.net app :)
There are still some bugs, which have to be resolved:
-Stop auto alignment!
-Identify the "Touches"


Google I/O 2009: Mastering the Android Media Framework


AT&T Interactive Mobile Website


Satellite-Terrestrial Network Delivering Mobile Video with Interactive Services - ICO mim





RELATED
Adding Multi-Touch to Your Windows Mobile Application's User Interface
Wei-Meng Lee,  DevX.com 3/24/08
Previous Post:  The new 3G iPhone:  Expanding the Possibilities of Interactive Multimedia Communication (Interactive Multimedia Technology,  6/9/08)
Note: I just skimmed the following articles - when I have more time, I'll post more of my reflections related to this set of topics.

Rethinking context framworks: Enabling cross-domain, cross-device experiences (pdf)

Daniel Stewart, Nitya Narasimhan,  Position Paper, CMPPC Workshop, Pervasive 2007

Interactive Mobile Multimedia Needs IP and Circuits
Brough Turner,  Internet Telephony, 9/09
Primetime for Mobile Television:  Extending the entertainment concept by bringing together the best of both worlds (pdf)  IBM Institute for Business Value
A Holistic Approach to Enhance Universal Usability in m-Learning
Vlado Glavinic, Sandi Ljubic, Mihael Kukec, 2008 The Second International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies
Seamless Mobility: A Continuity of Experiences across Domains, Devices, and Networks (pdf) 2005
Raghu Rau, Senior Vice President of Marketing, Motorola
Ambient Networks:  Cooperative Mobile Networking for the Wireless World
Norbert Niebert, John Wiley and Sons LTD  4/07 
Mobility management challenges and issues in 4G heterogeneous networks (link to ACM pdf)
Sadia Hussain, Zara Hamid and Naveed S. Khattak, InterSense '06.  Proceedings of the first International Conference on Integrated Internet Ad hoc and Sensor Networks
Mobile Multimedia: Tune in to Digital Convergence (pdf)
DVB-Scene (a trade magazine) 3/2008
HP OpenCall Media Plafform: a cost-effective, agile IP media server (pdf)
Whitepaper, Hewlett-Packard Development Company, L.P.  2/2009
A survey of wireless multimedia sensor networks (pdf)
Ian F. Akyildiz, Tommaso Melodia, Kaushik R. Chowdhury, 2006  Science Direct  Elsevier


Mobile Multimedia Companies/R&D
Stantum  (UMPC)

Movidity
MobIME: Mobile Internet for Media and Entertainment

Ideum's GestureWorks vs Adobe AIR 2 and Flash Player 1.0 comparison of multitouch and gesture support

Jim Spadaccini, of Ideum, shared information about his company's product, Gestureworks, highlighting how it provides better multi-touch and gesture support than Adobe AIR2 and Flash Player 10.1. Gestureworks supports multiple-point drag, rotate, and scale at the same time. In the video, the application is demonstrated on an HP Touchsmart 600 and a 3M multitouch screen.

Adobe AIR 2 and Flash Player 10.1  vs Gestureworks 1.0: A direct comparison of multitouch and gesture support


"A direct comparison between the built-in support for multitouch found in Adobe Flash Player 10.1 beta / Adobe AIR 2 and that of the Gestureworks multitouch framework for Flash. More about this comparison can be found on the Gestureworks website (http://www.gestureworks.com) and the Ideum website (http://www.ideum.com)  There is a blog post with more about this comparison and links to all of the example files at: www.ideum.com/2010/01/true-multitouch-wi th-adobe-flash/ "

True Multitouch with Adobe Flash - Jim Spadaccini


GestureWorks Supported Gestures

Example of Ideum's GestureWorks multi-touch, multi-user design for an exhibit a the Vancouver Aquarium:

Musings about NUI, Perceptive Pixel and Microsoft, Rapid Creative Prototyping (Lots of video and links) Revised

It just might be the right time for everyone to brush up on 21st century tech skills. iPads and touch-phones are ubiquitous. Touch-enabled interactive whiteboards and displays are in schools and boardrooms.  With Microsoft's Windows 8 and the news that the company recently acquired Jeff Han's company, Perspective Pixel, I think that there will be good support - and more opportunities- for designers and developers interested in moving from GUI to NUI.    


In the video below, from CES 2012, Jeff Han provides a good overview of where things are moving in the future.  We are in a post-WIMP world and there is a lot of catching up to do!

CES 2012  Perceptive Pixel and the Future of Multitouch (IEEE Spectrum YouTube Channel)



During the video clip, Jeff explains how far things have come during the past few years:

 "Five and 1/2 years ago I had to explain to everybody what multi-touch was and meant. And then, frankly, we've seen some great products from folks like Apple, and really have executed so brilliantly, that everyone really sees what a good implementation can be, and have come to expect it.  I also think though, that the explosion of NUI is less about just multi-touch, but an awareness that finally people have that you don't have to use a keyboard and mouse, you can demand something else beside that.  People are now willing to say, "Oh, this is something I can try, you know, touch is something I can try as my friendlier interface"."

Who wouldn't want to interact with a friendlier interface?  Steve Ballmer doesn't curb his enthusiasm about Windows 8 and Perceptive Pixel.  Jeff Han is happy how designs created in Windows 8 scales for use on screens large and small. He explains how Windows 8 can support collaboration. The Story Board application (7:58) on the large touchscreen display looks interesting.

I continue to be frustrated by the poor usability of many web-based and desk-top applications.  I like my iPad, but only because so many dedicated souls have given some thought to the user experience when creating their apps.  I often meet with disappointment when I encounter interactive displays when I'm out and about during the day.  It is 2012, and it seems that there are a lot of application designers and developers who have never read Don Norman's The Design of Everyday Things!

I enjoy making working prototypes and demo apps, but my skill set is stuck in 2008, the last year I took a graduate-level computer course.  I was thinking about taking a class next semester, something hands-on, creative, and also practical, to move me forward. I can only do so much when I'm in the DIY mode alone in my "lab" at home.  I need to explore new tools, alongside like-minded others.  


There ARE many more tools available to designers and developers than there were just four years ago.  Some of them are available online, free, or for a modest fee.  I was inspired by a link posted by my former HCI professor, Celine Latulipe, to her updated webpage devoted to Rapid Prototyping tools. The resources on her website look like a good place to start for people who are interested in creating applications for the "NUI" era.  (Celine has worked many interesting projects that explore how technology can support new and creative interaction, such as Dance.Draw.) Below is her description of her updated HCI resources:

"New HCI resource to share: I have created a few pages on my web site devoted to Rapid Prototyping tools, books, and methods. These pages contain reviews of various digital tools, including 7 different desktop prototyping apps, and including 8 different iPad apps for wireframing/prototyping. I hope it's useful to others. Feel free to share... and please send me comments and suggestions if you find anything inaccurate, or if you think there is stuff that I should be adding. I will be continuing to update this resource." -http://www.celinelatulipe.com (click on the rapid prototyping link at the top)

IDEAS
Below are just a few of my ideas that I'd like to implement in some way. I can't claim ownership to these ideas- they are mash-ups of what comes to me in my dreams, usually after reading scholarly publications from ACM or IEEE, or attending tech conferences. 

• An interactive timeline, (multi-dimensional, multi-modal, multimedia) for off-the-desktop interaction, collaboration, data/info analysis exploration.  It might be useful for medical researchers, historians, genealogists, or people who are into the "history of ideas".  Big Data folks would love it, too. It would handle data from a variety of sources, including sensor networks. It would be beautiful to use.
• A web-based system of delivering seamless interactive, multi-modal, immersive experiences, across devices, displays, and surfaces. The system would support multi-user, collaborative interaction.  The system would provide an option for tangible interaction.
• A visual/auditory display interface that presents network activity, including potential intrusions, malfunctions, or anything that needs immediate attention that would be likely to be missed under present monitoring methods. 
• Interactive video tools for creation, collaboration, storytelling.  (No bad remote controllers needed.)
• A "wearable" that provides new ways for people to express and communicate creatively, through art, music, dance, with wireless capability. (It can interact with wireless sensor networks.)*
• An public health application designed to provide information useful in understanding and sepsis prevention efforts. This application would utilize the timeline concept describe at the top of this list. This concept could also be useful in analyzing other medical puzzles, such as autism.

Most of these ideas could translate nicely to educational settings, and the focus on natural user interaction and multi-modal i/o aligns with the principles of Universal Design for Learning, something that is important to consider, given the number of "at-risk" learners and young people who have disabilities.

I welcome comments from readers who are working on similar projects, or who know of similar projects.  I also encourage graduate students and researchers who are interested in natural user interfaces to and move forward with an off-the-desktop NUI project.  I hope that my efforts can play a part in helping people make the move from GUI to NUI!  



Below are a few videos of some interesting projects, along with a list of a few references and links.

SMALLab (Multi-modal embodied immersive learning)

PUPPET PARADE: Interactive Kinect Puppets(CineKid 2011)



MEDIA FACADES: When Buildings Start to Twitter

HUMANAQUARIUM (CHI 2012)

 

NANOSCIENCE NRC Cambridge (Nokia's Morph project)






 
Examples: YouTube Playlists
POST WIMP EXPLORERS' CLUB
POST-WIMP EXPLORER'S CLUB II

Web Resources
Celine Latulipe's Rapid Prototyping Resources 
Creative Applications
NUI Group: Natural User Interface Group
OpenFrameworks and Interactive Multimedia: Funky Forest Installation for CineKid
SMALLab Learning

Microsoft Research Natural User Interface 

OpenExhibits: Free multi-touch + multiuser software initiative for museums, education, nonprofits, and students.
OpenSense Wiki 
CINEKID 2012 Website 
Multitouch Systems I Have Known and Loved (Bill Buxton)
Windows 8
Perceptive Pixel
Books
Natural User Interfaces in .NET  WPF 4, Surface2, and Kinect (Josh Blake, Manning Publications)
Chapter 1 pdf (Free)
Brave NUI World: Designing Natural User Interfaces for Touch and Gesture (Daniel Wigdor and Dennis Wixon)
Designing Gestural Interfaces (Dan Saffer)
Posts

Microsoft's"Magic Wall" Tech on Windows 8 Laptops and Tablets by Next Year

Bill Snyder, ReadWrite Web, 7/20/12

I noticed some interesting tools on the Chrome web store - I plan to devote a few more posts to NUI tools in the future.

For the Tech Curious: Multi-Gesture Net: A Multi-touch and Multi-gesture Research Blog

Laurence Muller, M.Sc. is a scientific programmer at the Universiteit van Amsterdam who develops scientific software for multi-touch devices. His blog, Multigesture.Net, provides good information regarding multi-touch and gesture interaction hardware and software applications.


Laurence links to the DYI tabletop computing bootcamp that was held at
IEEE Tabletops and Interactive Surfaces 2008. From there, you can find a linked list of the organizers of the events, and additional information.

Picture below is from MTC Multi-touch Console:


Here is a link to the group's libavg wiki that includes open-source code and "how-to" instructions.

If you are interested in multi-touch and multi-gesture computing from an academic point of view, Florian Echtler, of the Technische Universitat Munchen has a series of publications listed on his website. Here is the abstract of one of his papers. He is on the right track. I especially like the fact that he's thought about widget layers. (I have, too, but they are only sketches in my idea book.)

TICH: Tangible Interactive Surfaces for Collaboration between Humans (Sourceforge website, with links to libtisch.

F. Echtler, G. Klinker
A Multitouch Software Architecture
NordiCHI 2008: Using Bridges, 18-22 October, Lund, Sweden. (bib)

"In recent years, a large amount of software for multitouch interfaces with various degrees of similarity has been written. In order to improve interoperability, we aim to identify the common traits of these systems and present a layered software architecture which abstracts these similarities by defining common interfaces between successive layers. This provides developers with a unified view of the various types of multitouch hardware. Moreover, the layered architecture allows easy integration of existing software, as several alternative implementations for each layer can co-exist. Finally, we present our implementation of this architecture, consisting of hardware abstraction, calibration, event interpretation and widget layers."

Multi-touch Linux on a Stantum SlatePC & more

-Stephane Chatty, Benjamin Tissoires; Video by G. Tabart


Information from the YouTube video:
"This video, shot by ENAC's Interactive Computing Laboratory, demonstrates how one can use out-of-the-box Linux applications with multitouch displays, using the lab's kernel drivers and modified X.org evdev driver. See http://lii-enac.fr/en/projects/shareit/xorg.html for technical details. This work was done during project ShareIT, carried out with Thales Avionics, IntuiLab and Stantum, and sponsored by Aerospace Valley. The ShareIT project explores the use of multitouch interaction for future commercial aircraft cockpits."


The multi-touch software shown in the video was created by IntuiLab.  The Slate PC multi-touch hardware was built by Stantum.  Later in the video, the software is shown running on a MERL DiamondTouch


RELATED
Project ShartIT, ENAC:

ENAC Interactive Computing Laboratory, University of Toulouse, France


Multitouch Interaction: Some Background (from ENAC)


List & Description of some multi-touch devices (from ENAC)


IntuiLab Showroom Pictures


Stantum: Notes from Guillame:
"Key to the expansion of multi-touch is finding the killer app."  

London Design Festival Winners - Visualization Category: Multi-Touch Barcelona; Experiential Category: LM3Labs

Multitouch Barcelona and LM3Labs were among the winners in the London Design Festival!

Visualization Category:  Multi-Touch Barcelona

Human Interface (A Real Human!)

Hi from Multitouch Barcelona on Vimeo.
 

"With a client list that includes Red Bull, you know these guys aren’t messing around. Multitouch Barcelona designs "experiences that merge real and digital into a creative environment where people are invited to touch, play, move, feel as they do in the real world."  Clearly enjoying a tongue-in-cheek methodology, the group’s philosophy seems to grow out of an affinity for natural interaction. The recent work showcased at Offf (a festival for, um, "post-digital creation culture") perfectly shows both their whimsy and tech chops. For example, the Human Interface, a person you use as a computer by asking him to carry out tasks such as email using a keyboard made from cardboard and tubes that carry email around.  Nothing short of hilarious and yet strangely alarming." -Grand designers:  the world's best design work Paul Armstrong,  Wired UK,    8/17/09



I love Multitouch Barcelona. Especially their Space Invaders project.
(I posted this video previously, but I like it so much I decided to post it again!)

Multitouch Space Invaders XL from Multitouch Barcelona on Vimeo.




Another company I follow,  LM3Labs, won in the Experiential category:

Showreel:

Update from Multi-Touch Barcelona and Guten Touch: Multi-Touch Space Invaders

Multitouch Space Invaders Basic Demo from multitouch-barcelona on Vimeo..

For more sights and sounds from this group, visit the Red Bull Music Academy Barcelona 2008 website.

For more about multi-touch technology, including DYI instructions for creating multi-touch tables and displays, open-source code,and tutorials, visit the NUI Group website.

Sensory-Mind's Ring Wall, an interactive multi-touch wall you don't even have to touch!

Sensory-Minds is a small company in Germany that is focused on research and design in the field of Natural User Interfaces. If you visit the SENSORY-MINDS website, you'll find that it has been designed for touch interaction.

ring°wall from SENSORY-MINDS on Vimeo.

Information about the Ring Wall from Sensory-Mind's Vimeo site:

".....The two-piece ring°wall consists of a LED display and a multitouch information-wall and impresses by its size: a total surface of 425 square meters, which equals more than 6000 computer displays, is the biggest of its kind. An interactive World emerges out of 34 million pixels generated by 15 high definition projectors and is supported by 30 directional speakers.


Multitouch sensors basing on laser technology, convert the usage of the natural user interface into an experience. By direct touching, more than 80 users can simultaneously get informed about news and activities around the ringworld.


The interactive wall is not only a central information system, but also an innovative advertising tool and medium for public viewings."

Heiko Hoffman of Sensory-Minds recently joined the NUI-Group.  Here is a response to a question on the NUI-group forum about the way the system's sensors work:


"The sensor works like spinning radar gun, or like a wiper in a car, that means like a radar gun you get the distance to the object.  It’s not quiet difficult when you know the position of the sensor and the degree from the beam to get the X,Y position.  Yes, you don’t need to touch the surface but we arranged it that the radar beam is very close to the screen so it seems that you have to touch the surface.

At the moment the system (has) limitations because each sensor (gets) coordinates and this means that you got shadows.This is no problem because the people interact very fast. We are working on a system where the data from the sensors were put together and that would be the solution for this problem."


Note:

I'll be writing a few more posts updating the work of past and present NUI-group members as well as news from the commercial multi-touch & gesture community.  Be sure to check back soon, or subscribe to this blog!

Stantum's Multi-touch Slate PC, Windows 7 Certified

The Stantum Slate PC is based on the Dell Inspiron Mini netbook platform. The 10.1 Slate accepts handwriting, gestures, and text-based input. Is in the "proof-of-concept stage.

Slate PC Proof of Concept (pdf) Includes specifications. Stantum also offers Multi-Touch Development Kits (MDK's) that include a controller board featuring PMatrix, a controller chip. and a touch panel.

How PMatrix Works (from the Stantum website):


RELATED

Stantum's unlimited multitouch meets Mini 10 in a fight to the death
(Josephe L. Flatley, Engadget, 11/17/09)

JAZZ MUTANT
Jazz Mutant is the Music & Media product division of Stantum.  I offers multi-touch control surfaces for audio and media - Dexter and Lemur.

More information about Stantum and other multi-touch focused companies to come!

Multi-touch application developed by an all-women team from South Dakota School of Mines and Technology for the Imagine Cup Competition

According to a recent article from the Microsoft News Center, "statistics from high schools and universities suggest that percentage won’t change any time soon. Only 17 percent of Advanced Placement (AP) computer science test-takers in 2008 were women, even though women represented more than half of all AP test-takers. At the college level, fewer than one in five computer and information science degrees were awarded to women."


Microsoft's Imagine Cup competition is a way to encourage female students that they can use technology to help make the world a better place, and that computer science is a field that provides an outlet for creativity a innovation.

One of the teams that participated in the Imagine Cup Competition was "Team Blob". The young women in this team attend South Dakota School of Mines and Technology. Their work can be seen in the video presentation of Team Blob's Multi-touch Designer, which allows teachers to create multi-touch presentations for their students. The application was written in C# using Windows Presentation Foundation.


The team developed a interactive timeline to highlight history's famous women in math and science, and demonstrated it on a 40-inch multi-touch table to girls who visited their university campus. The time-line can be seen in the video clip at about 3:34.

"Team Blob members, from left, Lori Rebenitsch, Robyn Krage and Jaelle Scheuerman demo their application that aims to bring emerging multitouch technology into K-12 classrooms. The all-woman team is from the South Dakota School of Mines and Technology."