It is about time for an update about touch/gesture- interactive technologies.
I've been researching the latest in "touch" screens and new developments in interactive multi-media content. In just one year, a multitude of websites have been transformed from static to interactive.
Although the initial objective for some of these websites was to optimize the interface and navigation for people accessing websites via touch-screen cell phones, some are ideal for use on touch-enabled slates, the iPad, and even larger touch screen displays and surfaces.
Convergence seems to be the buzz word of the day. Interactive TV. Game sets with Internet access. Movies on your cell phone. Touch screen Coke machines displaying movie trailers. What's happening now, and what is next?
I welcome input from my readers in the form of links to websites, university labs with grad students and professors who are obsessed with emerging interactive technologies, proof-of-concept video clips, video clips of related technologies that are new-to-market, etc.
FYI: I'm also in the middle of writing a series of posts about 3D television technologies for the Innovative Interactivity blog, and welcome input from my readers about this topic.
-Stephane Chatty, Benjamin Tissoires; Video by G. Tabart
Information from the YouTube video: "This video, shot by ENAC's Interactive Computing Laboratory, demonstrates how one can use out-of-the-box Linux applications with multitouch displays, using the lab's kernel drivers and modified X.org evdev driver. See http://lii-enac.fr/en/projects/shareit/xorg.html for technical details. This work was done during project ShareIT, carried out with Thales Avionics, IntuiLab and Stantum, and sponsored by Aerospace Valley. The ShareIT project explores the use of multitouch interaction for future commercial aircraft cockpits."
The multi-touch software shown in the video was created by IntuiLab. The Slate PC multi-touch hardware was built by Stantum. Later in the video, the software is shown running on a MERL DiamondTouch
Globacore Globacore used four 72" Blue Ocean Low Gain Screens 16:9 to create their twenty-foot multi-touch wall, which I think was also used for the "holographic" screen in the first video. The tracker and base flash API from the NUI-group were used in this system. I wasn't sure if I'd heard about Blue Ocean screens. Much to my surprise, the company behind Blue Ocean is Nippura, located in Japan. Nippura has another office in Charlotte, N.C. For years, Nippura's core business has been focused on creating "the world's largest acrylic panels, tunnels, cylinders and spheres for the professional aquarium industry", but is also focused on screen systems as well.
US Nippura/Blue Ocean Projection Screens Blue Ocean screens were use in the second videos of this post. Below is some information the screens from the company's website: "Blue Ocean® Projection Screens are the first ever to cast the screen into a hi-tolerance viewing plane centered between two ultra-clear panels of cell-cast acrylic, the same type of material used in hi-optic fighter jet canopies and the world’s largest aquariums. Blue Ocean® Projection Screens also offer something never before found in screens: self-rigidity. The thickness of each screen can be fabricated so that it maintains its form without external frames or tension systems. Frameless high definition projected images without borders amplify the infinite depth viewing experience. Because of its durability, outdoor applications of entertainment around the garden and pool are also possible. Normal wear and tear scratching has no long-term effect on the screens’ performance, and a ten year old screen, simply re-polished, will literally look and perform exactly as it did the day it left the factory." Super High Definition Stereo Image using 4K Projection Systems (pdf) Tetsuro Ogi, Hiroaki Daigo, So Sato, Yoshisuke Tateyama, Yasuaki Nishida, International Conference on Artificial Reality and Telexistence 2008
NUITEQ NUITEQ offers single and duo-touch plugins for Snowflake Suite that enables the software to run on NEXIO and NextWindow screens, including NextWindow's screens that are integrated in the HP TouchSmart and Dell Inspiron all-in-one PC's and others. SnowFlake Suite is also compatable with 3M Touch Systems, N-Trig, and Lumino. You can view the SnowFlake Suite application in action on NUITEQ's YouTube channel. "NUITEQ's software product Snowflake Suite, which received a Stevie Awards finalist recognition for Best New Product or Service of the Year 2009 - Media & Entertainment, is available to end clients, software developers, system integrators, VAR's and OEM's. Snowflake operates on a wide variety of hardware technology platforms including rear camera based solutions, IR overlay systems, bending wave, surface acoustic wave (SAW), projected and surface capacitive, resistive and other touch technologies for small hand held mobile devices and larger scale interactive displays." Recent news: NUITEQ Assists LTU Skelleftea at SACO event with new educational multi-touch desk
NextWindow and All-In-One PC's: NextWindow, the company that made the large single-touch display I used for a couple of class projects nearly three years ago, has really grown over the past few years. NextWindow provides optical touch screens for the HP TouchSmart PC, and also for the Dell Inspiron Studio One All-In-One PC, and the Dell Sx2210T desktop monitor. The next Dell with a NextWindow touch screen will be the Inspirion One 19, an all-in-one PC. You can find NextWindow touch screens in the Medion X9613, the NEC ValueStar W, and the Sony L Series. (NextWindow has Windows 7 certification for touch screens)
STANTUM and GUNZE USA Stantum has partnered with Gunze USA, a manufacturer of resistive touch-panels, to co-design and deliver multi-touch products. "Gunze USA will provide these custom multi-touch modules to the North American market for use in industrial, automotive, defense, aerospace, medical, kiosk/point-of-sales, and other professional applications. Unlike capacitive multi-touch, Stantum’s technology is particularly well suited to these professional markets, thanks to its:
immunity to EMI issues, via easy coexistence with Wi-Fi or Bluetooth antennas around the screen;
stylus input and high precision, enabling handwriting recognition; and
ability to work flawlessly with gloves."
According to the press release, Stantum's multi-touch technologies provide advantages such as handling 10 or more touches at a time, finger and stylus input, finger-pressure consumption, and native Windows 7 support, with no need for a driver. Developers can order the "beta" version of the Slate PC, based on the Dell Inspiron Mini netbook platform, from the Stantum website.
GESTURETEK The video below is of an interactive 3D application created for the Gondwana Das Praehistorium in Schiffweler, Germany, and provides an simulation of what it might be like to be a flying ancient pterosaur of giant dragonfly. Two gamers can play at the same time. If you want to fly, you flap your arms, and navigate by leaning. You crouch to dive, and increase your speed by flapping your arms even faster: GestureTek's 3D Interactive Flying and Virtual Game Experience for Gonwana Museum
This company always has something going on that looks fun and interactive. To get an idea this company's interactive work, take a look at the information on the GestureTek website links below. GestureTek has a variety product videos on the website.
SNIBBE INTERACTIVE Snibbe Interactive was founded by Scott Snibbe to develop social immersive media as a communication medium that incorporates the language of cinema and engages participation and interaction with others (and technology) through unencumbered body movements.
Scott Snibbe's Social Immersive Media presentation at Stanford's Seminar on People, Computers and Design, 5/2009
'Scott Snibbe of Sona Research discusses a distinct form of augmented reality focused on social interaction: social immersive media. He discusses design principles and interaction techniques to create strong emotional responses and social engagement through visceral interaction. He also describes approaches to clearly communicate cultural and scientific ideas through the medium, and how to promote specific distinct social behavior in uses." -YouTube description
RELATED Globacore "Globacore was founded by Ben Unsworth and Jeff Williams out of a passion for technology, art, and business. Globacore began as a web hosting and development company supporting the the technical needs of sports apparel startup VC Ultimate and it's associated companies."
"Globacore Inc. was officially incorporated in April 2003. In 2007 Ben and Jeff were joined by programmer Kyle Davis and graphic designer (and trail biker) Matt Venhuizen. In 2008, the Flexmaster Lukasz Ruminski joined us and continues to be a huge member of our team.Update: We'd like to welcome our newest team member: Developer Dave Rapin!"
Globacore Interactive Technologies 67 Mowat Ave. Suite 431 Toronto, ON M6K 3E3 Phone: + 1 (416) 925-2082 Tollfree: + 1 (888) 309-7658 Fax: + 1 (647) 477-3270 General Email:info@globacore.com
Natural User Interface Asia Pte Ltd(Asia office)
12 Little Road
#06-01 Lian Cheong Industrial Building
Singapore 536986
Phone: +65 6748 5245
Fax: +65 6748 7961
FINGERTAPPS Contact: James Bell-Booth
james[at]fingertapps.com
+64 21 226 6399 www.fingertapps.com/
GUNZE USA Product Overview Contact:Gunze Electronics USA 2113 Wells Branch Parkway, #5400 Austin, Texas 78728 Tel: 512-990-3400 Fax: 512-252-1181
LM3LABS Website: http://lm3labs.com/ENG/index_eng.html info@lm3labs.com Europe: Antipolis Innovation Campus 300, route des Cretes 06560 Sophia-Antipolis France Hong Kong, China +852-8198.8406 Australia +61(02)8011.3342 USA: +1(646)461.6146 TGarden Square 409 3-5-26 Funabori Edogawa-ku 134-0091 Tokyo, Japan +81(0)3.3877.0779
Outdoor Media Company (China) Products Contact: Kuo Lin (?) 42x46 Multi-touch LCD Display
HIWAD TOUCH - Beijing Yhjn Science and Technology Co. LTD
Interactive Sandbox "An interactive sand box or sand table for use in tactical training includes an automated control system and image generator so that infinitely variable visual images may be projected onto the contoured surface of the sand for training and preparation for action in the military field and disaster control, industrial planning, environmental protection, and similar situation."
Interactive Holographic Touch Screen
Beijing Yihejianing Science and Technology Co., Ltd
Address: Room 621, Jinyuan Modern shopping Mall, NO.1 Yuanda Road, Haidian District, Beijing, China
TE:8610-88892897，88892895
FAX:8610-88892867-801
Email: hxrong2001@hotmail.com http://www.cnhci.cn
A study in January of 2005 by Nokia indicated that there was a demand for interactive mobile multimedia services. That study was conducted about five years ago, before anyone had heard of the iPhone! The technology to support interactive mobile multimedia has come a long way since then, and many of the new applications support multi-touch, or at least duo touch interactivity.
I'm very much interested in figuring out how to design web-based interactive content (and apps) that can be optimized for touch (and multi-touch/gesture) screens of various sizes, from SmartPhones/iPhones to the large interactive whiteboards that are now in a multitude of classrooms.
From my experience as a school psychologist, I know that there are many teens who have graduated from traditional cell phone to the next level. If they don't have an iPhone, they have a smart phone. I don't have the statistics on this, but my personal observations tell me that there are teens who come from families who are from lower economic status who are somehow able to own 3G smartphones.
What a great opportunity to provide casual interactive multimedia educational games to support student learning! The games and activities could be assigned as homework from time to time, and with the appropriate LMS (Learning Management System), the teacher would have instant access to student progress. In addition, the students would be provided with immediate feedback about their "work", which we know is an important factor in learning.
It is difficult to figure out the best path to forge, since nearly every week someone announces a new platform, technology, and programming approach!
At any rate, here are a few interesting things related to this topic that I'd like to share. Many of these concepts are in the experimental phase, but are worth some attention.
SciLor's Comments: I have manged it to get Multitouch running in an vb.net app :)
There are still some bugs, which have to be resolved:
-Stop auto alignment!
-Identify the "Touches"
Google I/O 2009: Mastering the Android Media Framework
AT&T Interactive Mobile Website
Satellite-Terrestrial Network Delivering Mobile Video with Interactive Services - ICO mim
The video is is of the Stantum Slate PC, via Netbooked's YouTube channel...The system in the video is running on a modded Dell Mini 10, and doesn't require calibration. Notice how the system easily handles a variety of interaction- fingers, thumbs, pinch, rotation, multiple finger swipes, brush strokes, fingernail action, stylus, and more.
At this time, the Stantum Slate is available for developers only.
The Stantum Slate PC is based on the Dell Inspiron Mini netbook platform. The 10.1 Slate accepts handwriting, gestures, and text-based input. Is in the "proof-of-concept stage.
Slate PC Proof of Concept (pdf) Includes specifications. Stantum also offers Multi-Touch Development Kits (MDK's) that include a controller board featuring PMatrix, a controller chip. and a touch panel.
JAZZ MUTANT
Jazz Mutant is the Music & Media product division of Stantum. I offers multi-touch control surfaces for audio and media - Dexter and Lemur.
More information about Stantum and other multi-touch focused companies to come!
NUI-Group member Christian Bannister, a musician, designer, and developer behindSubcycle Labs has been experimenting with music and multi-touch technology and interactions. Here's a demonstration of what he's come up with so far:
multi-touch the storm - interactive sound visuals - subcycle labs from christian bannister on Vimeo. . "The big picture goal of this project is to bridge the gap between sound visualization and musical instrument. With multi-touch interaction it is possible to manipulate multiple characteristics of a sound—directly and visually. Right now a lot of electronic music involves staring at the back of the performers laptop. This is a shame because in many cases a lot of really interesting things are happening on the computer that the audience is completely unaware of. This project hopes to create a common visual language and experience for the electronic musician and the audience by enhancing the perception of sound and music on both sides...These sketches are built with Processing and Max/MSP networked with OSC on a single computer..." -Christian Bannister
JazzMutant Lemur The multi-touch tech company now known as Stantum evolved from JazzMutant, Jazzmutant was founded in 2002 by Guilluame Largillier and Pascal Joguet, and joined by Julien Olivier in 2003. The original focus was to create a multi-touch controller for music applications, and the Lemur was born. It is now in version 2, with features such as a gesture object that provides three ways for people to interact with sound, extended scripting abilities, and remote control of your computer's mouse cursor or keyboard. Stantum recently developed a next-gen multi-touch screen system for use in mobile devices such as smart phones and netbooks. (I'll discuss this further in a future post.)
Mapping Ableton Live to Jazz Mutant's multi-touch Lemur for sound design:
Additional tutorials can be found the Jazzmutant YouTube channel.
Ableton Live, the software used in the above video, will include Max. Max/MSP is now known as MAX 5: "In use for over twenty years by performers, composers, artists, scientists, teachers, and students, Max is the way to make your computer do things that reflect your individual ideas and dreams. Version 5 is a dramatic transformation of Max that begins a new chapter in the program's history."
There has been a flurry of work in the computer music technology world that relates to what has been going on with interactive display technology, multi-touch & gesture interaction. I came across a link to the GDIF website when I was searching for information about interactive music and the use of multi-touch technologies for a future blog post.
So what is GDIF? Gesture description interchange format
"The Gesture Description Interchange Format (GDIF) is being developed as a tool for streaming and storing data of music-related movements, actions, and gestures. Current general purpose formats developped within the motion capture industry and biomechanical community (e.g. C3D) focus mainly on describing low-level motion of body joints. We are more interested in describing gesture qualities, performer-instrument relationships, and movement-sound relationships in a coherent and consistent way. A common format will simplify working with different software, platforms and devices, and allow for sharing data between institutions." (The Jamoma environment is used to prototype GDIF.)
Alexander Refsum Jensenius is the man who initiated the GDIF project. He's written a variety of articles about music, gestures, movement, and emerging technologies.
God oy, R. I., E. Haga, and A. R. Jensenius (2006b). Playing `air instruments':Mimicry of sound-producing gestures by novices and experts. InS. Gibet, N. Courty, and J.-F. Kamp (Eds.), Gesture in Human-Computer Interaction and Simulation, GW 2005, Volume LNAI 3881, pp. 256{267.Berlin: Springer-Verlag.
Jensenius, A. R., T. Kvifte, and R. I. God oy (2006). Towards a gesture description interchange format. In N. Schnell, F. Bevilacqua, M. Lyons, and A. Tanaka (Eds.), NIME '06: Proceedings of the 2006 International Conference on New Interfaces for Musical Expression, Paris, pp. 176{179. Paris: IRCAM { Centre Pompidou.}
Kvifte, T. and A. R. Jensenius (2006). Towards a coherent terminology and model of instrument description and design. In N. Schnell, F. Bevilacqua, M. Lyons, and A. Tanaka (Eds.), Proceedings of New Interfaces for Musical Expression, NIME 06, IRCAM - Centre Pompidou, Paris, France, June 4-8, pp. 220–225. Paris: IRCAM - Centre Pompidou. [PDF]
Marshall,M. T., N. Peters, A. R. Jensenius, J. Boissinot, M. M. Wanderley, and J. Braasch (2006). On the development of a system for gesture control of spatialization. In Proceedings of the 2006 International Computer Music Conference, 6-11 November, New Orleans. [PDF]
"Sonic Interaction Design is the exploitation of sound as one of the principal channels conveying information, meaning, and aesthetic/emotional qualities in interactive contexts."
SID Action has four working groups:
WG1: Perceptual, cognitive, and emotional study of sonic interactions
"SoundHack was my main thing for a long time, and I poured a lot of effort into it. It was the place I put my ideas. I did have something of a mission with SoundHack. I wanted to take some computer music techniques that were only used in academia, and get them out there so that all types of musicians could use them."-Tom Erbe SoundHack Spectral Shapers
"Csound is a sound and music synthesis system, providing facilities for composition and performance over a wide range of platforms. It is not restricted to any style of music, having been used for many years in the creation of classical, pop, techno, ambient, experimental, and (of course) computer music, as well as music for film and television."-Csound on Sourceforge
"For me, music is a medium through which the inner spiritual essence of all things is revealed and shared. Compositionally, I am interested in extending the voice of the traditional performer through technological means to produce a music which connects with the past, lives in the present and speaks to the future. Educationally, I am interested in helping students see technology as the most powerful instrument for the exploration, discovery, and realization of their essential musical nature - their inner voice."
Upcoming post about innovations at Stantum:
I'll be focusing on Stantum and its music and media technologies division, JazzMutant. in my next post. It is interesting to note that the co-founders of Stantum, Guilliam Largilleir and Pascal Joget, have a background in electronic music. Guiliam specializes in multi-modal user interfaces and human-machine interface technologies. Pascal has a background in physics and electronics, and has worked as a sound engineer.
My music back-story:
The very first computer-related course I took was Computer Music Technology (in 2003), since I play an electronic midi/digital keyboard and previously tried to teach myself a few things, long before computers and related technologies were "easy" for me to figure out. During the mid-90's, I tried my hand at Dr. Richard Boulanger's CSound, and tried to acquaint myself with tools from Cycling'74, but I gave up. Not long after that, bought the first version of MOTU's Freestyle, which nicely worked on my Performa 600, hooked up to my Ensoniq 32, after the nice people at MOTU sent me an update that was compatible with my set-up. Later on, I came across Tom Erbe's SoundHack freeware.
