HACKED KINECT MULTITOUCH using libFreenect and libTISCH (via Florian Echtler)

MULTI-TOUCH WITH HACKED KINECT
Here is NUI-Group member Florian Echtler's  proof-of-concept HD video of using a hacked Kinect camera for multitouch-like interaction.  The application was built on Ubuntu Linux written using libfreenect, by marcan42  and Florian's creation, libTISCH.



Florian decided to use picture-browsing interaction to demonstrate proof-of concept, so "everybody can focus on more interesting things :-)"


(I have SO many ideas for this!  I'll throw a few out there in an upcoming post....maybe someone can run with them!)


RELATED
Hacked Kinect taught to work as multitouch interface
Paul Miller, engadget, 11/11/10


FOR THE TECH-CURIOUS:
TISCH stands for Tangible Interactive Surfaces for Collaboration between Humans, and is a cross-platform, cross device multi-touch development framework.  You can download the source package for Windows, MacOS X, and Linux from the TISCH Sourceforge website. The Ubantu Lucid/Karmac version has "superquick installation via PPA" - the instructions can be found on the TISCH Sourceforge website.


LibFreenect- Open Source PC Drivers for Kinect
Xan Tium, XBLOG 360 11/10/10

Marcan is Hector Martin Cantero, the author of the Abort, Retry, Hack? blog.

For your convenience, I've reposted something I wrote about libTISCH back in 2009:

libTISCH, a multi-touch development framework with multi-touch widgets and more!

For techies (and the tech-curious) who like technologies that support collaboration and multi-touch interaction,  this is great news!

Florian Echtler announced the first stable releas of libTISCH, a multi-touch development framwork, which can be found on Sourceforge.  TISCH stands for Tangible Interaction Surfaces for Collaboration between Humans.  libTISCH, a C++ software framework, is included in this project.  It provides a means for creating GUIs based on multi-touch and/or tangible input devices.

Here is how it works:

Here is information from libTISCH announcement:

Highlights of this release are, among others, the following features:

- ready-to-use multitouch widgets based on OpenGL
- reconfigurable, hardware-independent gesture recognition engine
- support for widely used (move, scale, rotate..), pre-defined gestures
 as well as custom-defined gestures

- hardware drivers for FTIR, DI, Wiimote, DiamondTouch..
- TUIO converters: source and sink

- cross-platform: Linux, MacOS X, Windows (32 and 64 bit)
- cross-language: C++ with bindings for C#, Java, Python

libTISCH has a lot to offer for the multitouch developer. For example, 

the textured widgets enable rapid development of applications for many
kinds of multi-touch or tangible interfaces. The separate gesture
recognition engine allows the translation of a wide range of highly
configurable gestures into pre-defined or custom events which are then
acted on by the widgets. While the lower layers of libTISCH provide
functionality similar to tbeta, touche etc. (you can interface existing
TUIO-based software with libTISCH in both directions), it goes far
beyond.

More information about the library and underlying architecture can be found on http://tisch.sf.net/ and in the Sourceforge wiki at
http://sourceforge.net/apps/mediawiki/tisch/

TISCH Project Wiki

Note:

Dr. Florian Echtler is on the scientific staff at the Technisch Universitat Munchen in Germany. Be sure to check out his  webpage.

I especially like the concept of the MeTaTop: "A Multi-Sensory Table Top System for Medical Procedures" that is linked from Florian's website.

Interactive Touch-Screen Technology, Participatory Design, and "Getting It", Revisited

I've been planning on updating one of my popular posts, "Interactive Touch-Screen Technology, Participatory Design, and Getting It" for a while. 


Here is a compromise - since much still rings true two years since I wrote it, the bulk of this post remains the same.   I've updated a few sections with additional video examples of interactive touch-screen applications, good and bad, along with a few links and resources, located at the bottom of the post. 


(The missing piece of information?  An update about apps for the iPad and similar touch-screen tablets.)

Sit back and enjoy!

-Images: HP; Wired

There's been some discussion over the reasons why so many people don't understand touch screen, or "surface" computing, even though research in this area has been going on for years.

As the new owner of the HP TouchSmart, I know that I get it.

The research I've conducted in this area suggests that people will "get-it" only if there is a strong commitment to develop touch-screen "surface" applications through a user-centered, participatory design process. In my view, this should incorporate principles of ethnography, and ensure that usability studies are conducted outside of the lab.

This approach was taken with Intel's Classmate PC. Intel has about 40 ethnographic researchers, and sent many of them to work with students and teachers in classrooms around the world. (A video regarding ethnographic research and the Intel Classmate project can be found near the end of this post.)


-Images: ClassematePC


Where to start?
K-12 classrooms and media centers. Public libraries. Malls. Hospital lobbies and doctor's offices. Any waiting room. Staff lounges in medical centers, schools, and universities. Community festivities and events. Movie theater lobbies. Museums and other points of interests.

I believe we need to take a "touching is believing" approach.

Here are some thoughts:
When I try to explain my fascination with developing touch-screen interactive multimedia applications, (interactive whiteboards, multi-touch displays and tables, and the like), many of my friends and family members eyes glaze over. This is particularly true for people I know who are forty-ish or over.

Even if you are younger, if you never saw the cool technology demonstrated in the movie Minority Report, or if you have limited experience with video games, or if you haven't came within touching distance of an interactive whiteboard, the concept might be difficult to understand.

The reality?
Even people who have the opportunity to use surface computing technology on large screens do not take full advantage of it. Multi-touch screens are often used as single-touch screens, and interactive whiteboards in classrooms are often serve as expensive projector screens for teacher-controlled PowerPoint presentations.

Most importantly, there are few software developers who understand the surface computing approach, even with the popularity of the iPhone and iPod Touch. Most focus on traditional business-oriented or marketing applications, and have difficulty envisioning scenarios for surface computing.  There is a need for a breath of fresh air!

Another factor is that not all people entrusted to market surface or touch screen computing fully understand it.

Despite a cool website showing off the goods, Microsoft's Surface multi-touch table has been slow to take off, limiting hope of bringing down the price tag to a price most families or schools could afford. (The picture above depicts an application for the Surface designed for health care professionals, not K-12 science education.)

Although you can't buy a Surface table for your family room, it is possible to buy a TouchSmart.  HP's TouchSmart website is engaging and highlights some examples of touch-screen interaction, but most people don't seem to know about it. (Since this post was last written, there are many more touch and multi-touch options available to the public, such as the  Dell, the iPad, etc.)

Unfortunately, you wouldn't have a clue that the HP TouchSmart exists browsing the aisles at Circuit City or Best Buy!

When I was shopping for my new TouchSmart, I noticed that from a distance, the TouchSmart looked just like the other larger flat-screen monitors filling up the aisles. The salespeople at both stores were not well-informed about the system. The only reason I knew bout the new TouchSmart was related to my obsession with interactive multimedia touch-screen applications- designing them, developing them, studying them, reading about them, blogging about them.... ; }

More thoughts:

After studying HCI (Human-Computer Interaction), and relating this knowledge to what I know as a psychologist, my hunch is that the "Window Icon Mouse Pointing-device" (WIMP) and keyboard input mind-set is embedded in our brains, to a certain extent. Like driving a car, it is something automatic and expected. This is true for users AND developers. (Update - See The Post WIMP Explorers' Club: Update of the Updates for a review of a discussion among passionate post WIMP folks)

Think about it.

Suppose one day, you were told that you no longer were allowed to control your car by turning on the ignition, steering the wheel, or using your feet to accelerate, slow down, or stop the car! Instead, you needed to learn a new navigation, integration, and control system that involved waving your hands about and perhaps speaking a few commands.

For new drivers who'd never seen a car before, this new system would be user-friendly and intuitive. Perhaps it would be quite easy for 16-year-old kids to wrap their heads around this concept. For most of us, no. Imagine the disasters we would see on our streets and highways!

When we think about how newer technologies are introduced to people, we should keep this in mind.

In my mind, spreading the word about surface computing is not a "if you build it, they will come" phenomenon, like the iPhone. We can't ignore the broader picture.

From my middle-age woman's vantage point, I believe that it is important that the those involved with studying, developing, or marketing surface computing applications realize that many of us simply have no point of reference other than our experiences with ATMs, airline kiosks, supermarket self-serve lanes, and the like. (The video clip at the very end of this post provides a good example of touch-screen technology gone wrong.) -UPDATE: additional videos were added to this post.

Be aware that there are substantial numbers of people who might benefit from surface computing who prefer to avoid the ATMs, airline kiosks, and self-serve grocery shopping.

Realize that the collective experience with technology, in many cases, has not been too pretty. Many people have had such user-unfriendly experiences with productivity applications, forced upon them by their employers, that any interest or desire to explore emerging technologies has been zapped.

My own exposure to interactive "surface" related technology was somewhat accidental.
A few years ago, a huge box was deposited into the room I worked in a couple of days a week as a school psychologist at a middle school. After a week or so, I became curious, and found out that it was a SmartBoard. Until then (2002!), I did not know that interactive whiteboards existed.

The boxed remained unopened in the room for the entire school year, but no worry. I played with the only other SmartBoard in the school, and found a couple at the high school where I also worked. I hunted for all of the applications and interactive websites that I could find, and tried them out. That is when I was hooked. I could see all kinds of possibilities for interactive, engaging subject area learning activities. I could see the SmartBoards potential for music and art classes. With my own eyes, I saw how the SmartBoard engaged students with special needs in counseling activities. (By the way, if you are working with middle school students, PBS Kid's ItsMyLife website activities work great on an interactive whiteboard.)

A few years have passed, and reflecting on all of my fun experiences with interactive whiteboards, with and without students, I now understand that many teachers still have had limited exposure to this technology.

This school year, many teachers are finding themselves teaching in classrooms recently outfitted with interactive whiteboards, scrambling along with educational technology staff development specialists, to figure out how it works best with various groups of students, and what sort of changes need to be made regarding instructional practice.

For the very first time, interactive whiteboards were installed in two classrooms at one of the schools I work at. One of the teachers I know thanked me for telling her about interactive whiteboards and sharing my resources and links.

If I hadn't let her know about this technology, she wouldn't have volunteered to have one installed in her classroom. It has transformed the way she teaches special needs students.

In the few months that she's used the whiteboard, I can see how much it has transformed the way the students learn. They are attentive, more communicative, and engaged. The students don't spend the whole day with the whiteboard - the interactive learning activities are woven into lessons at various times of the day, representing true technology integration.

Now let's see what happens when all-in-one touch-screen PC's are unleashed in our schools!

UPDATE:  Take a look at a post I wrote for Innovative Interactivity just after SMART Technologies acquired NextWindow - the post describe in detail how interactive whiteboards are transforming learning and teaching in a program for students with special needs.
SMARTTechnologies Acquires NextWindow: A "smart window" to the world


There are some interesting changes going on at the intersection of HCI and educational technology research.  I participated in a workshop at CHI 2010 last April and was impressed by what is going on in this area, around the globe:   Next Generation of HCI and Education

Value of ethnographic research:
Ethnographic Research Informed Intel's Classmate PC
"Intel looked closely at how students collaborate and move around in classroom environments. The new tablet feature was implemented so that the device would be more conducive to what Intel calls “micromobility”. Intel wants students to be able to carry around Classmate PCs in much the same way that they currently carry around paper and pencil." -via Putting People First and Ars Technica

The video below is from Intel's YouTube Channel. Information about Intel's approach to ethnographic research in classrooms during the development of the Classroom PC is highlighted. This approach uses participatory design and allows the set of applications developed for the Classmate PC to reflect the needs of local students and teachers. Schools from many different countries were included in this study.




FYI: TOUCH SCREEN DISPLAYS:  NEED FOR IMPROVEMENT!

Touch Screen Coke Machine at the Mall: 90 Seconds to get a Coke


User-Unfriendly Interactive Display in the Rain (Ballantyne Village Shopping Center)

User-Unfriendly Information Kiosk Interactive Map

I encountered this puzzling and frustrating interactive directory/map at the Cleveland Clinic.  When I went to visit a relative at the hospital a year or so later, the map was no longer there.


BETTER EXAMPLES OF INTERACTIVE SCREENS:

Here are some interesting pictures from lm3labs, which are in my interactive usability hall of fame:

Samsung's new Omnia SDG i900 was re-created in a much larger size, using lm3lab's Ubiq'window touchless technology.For more about lm3labs, including several videoclips, take a look at one of my previous posts: Lm3Labs, Nicolas Leoillot, and Multimedia Interaction

Midwife Toad App on a Microsoft Surface, Discovery Place Science Center


TellTable:  Digital Storytelling on the Surface:  Microsoft Research, UK


DECEMBER 31, 2009 -Interactive Soda Machine for Fun

The interactive screen on the Coke machine attracted the attention of this young child. He loved spinning the image of the bottle. So did the dad! He said, "I'd like something like this for my home!". I told him about the HP TouchSmart - both the dad and the mom did not know that there were affordable all-in-one touch screens available, but they knew about SMARTboards, because their children's classrooms had them.  Note:  No one from this family actually purchased a soft drink.  I was hoping to time how long it would take them to do so!

Some resources:

Multimedia, Multitouch, Gesture, and Interaction Resources (Mega-list, work in progress)

lm3labs (catchyoo, ubiq'window)

NUITeqNUI Group (See member's links)

Sparkon (See members links and multi-touch projects)

3M TouchNextWindow

Fingertapps

thirteen23

SmartTechnologies

SMARTTable

Perceptive Pixel - Jeff Hans

Microsoft Surface

InfoStrat

Reactable

RENCI Multi-touch

iPhone

iPad

Stantum

DELL MultiTouch

Vectorform

HP TouchSmart PC website, with demo

HP's TouchSmart YouTube videos

(More information and resources can be found by doing a "multitouch" or related search on this blog or The World Is My Interactive Interface.)

If you have plenty of time, take a look at my Post WIMP Explorers' Club YouTube playlist.
"Natural user interfaces, gesture interaction, multi-touch, natural interaction, post WIMP examples and more..."

FYI: I visited the Ballantyne Village shopping center a couple of months ago to follow up on the interactive displays, including the one I tried to use while it was raining.   The shopping center changed ownership, and the displays were replaced by the old-fashioned kind, pictured below:

Interesting blog roll and links on Christian Zoellner's TUI Blog by FORM+ZWECK; FILE Electronic Arts Festival

Christian Zoellner blogs about "tangible interaction & new interfaces" on the TUI Blog by Form + Zweck, a design magazine.  Christian is a designer and "presearcher" who lives and works in Berlin, Germany.  He teaches at the University of Fine Arts, Berlin. He maintains the Christian Zoellner website.

As I write this post, Christian is attending the FILE electronic arts festival, which focuses on "interactive art in public spaces, game design, and sonification".  It sounds like a fantastic conference! The festival is actually a group of events:  File Machinima, File Documenta, File Media Art, File Hypersonica, File Games, File Symposium, File Prix Lux, and workshops.


Christian Zoellner's Links:

• algorhythmics
• alt N research
• arduino
• bausteln
• cc
• christian zoellner
• createdigitalmotion.com
• createdigitalmusic.com
• de:bug
• doors of perception
• form+zweck
• fritzing
• groovemechanic
• hardhack
• interactiondesign.org
• interactivemultimediatechnology
• intuity.medialab
• M.I.T
• maker faire
• my future me
• nui-group
• openprocessing
• prado media lab
• processing
• salection
• smiles at strangers
• TEI conference
• the kitchen
• tinker
• tui.blog @ youtube

(I just noticed I'm on this list of links.)

RELATED

FILE SAO PAULO 2010:  July 27-August 29

"This year, besides the Centro Cultural FIESP - Ruth Cardoso programme, FILE launches FILE PAI (Paulista Avenue Interactive = Interactive Public Art), a project of digital public art that will occupy several spaces at Paulista Avenue with interactive works of art.

0 - Interactive Projection, 1 - Brigadeiro subway station, 2 - Electronic sound bus, 3 - Nomadic electronic graffiti, 4 - Paraíso subway station, 5 - Fnac store, 6 - FIESP Cultural Center - Ruth Cardoso, 7 - Trianon-Masp subway station, 8 - Conjunto Nacional building, 9 - Consolação subway station, 10 - Cervantes Institute, and 11 - São Paulo Art Museum - MASP compose the FILE PAI set, which intends to highlight the significance of interactive public art in order to understand and to absorb the new social phenomena provided by technology and, thus, to constitute strategies to interconnect with those new mass behaviors."

SOMEWHAT RELATED
Building Music at Spark Festival 09 (opens up to a full screen video of a musical building,from  Play the Magic.

Multi-touch and NUI News: NUITEQ's Multi-touch Snowflake Suite for Windows 7

NUITEQ is a company that has been involved in multi-touch technologies for the past few years, and works diligently at ensuring that it keeps up with the evolution in interactive displays and software.  The company's Snowflake Suite is now available for Windows 7, and is compatible with 3M Touch Systems, N-trig, NextWindow, Lumino, and other third-party platforms.  

Here is the current compatibility list, and also a list of Snowflake Suite for Winodws 7 features:

Compatibility

* NextWindow platforms like HP TouchSmart IQ500, IQ770, IQ800 series and the Dell Studio One (all 2 touch)
* N-trig platforms like HP TouchSmart TX2, Dell Latitude XT/XT2 and others
* 3M Touch Systems multi-touch kit
* Lumio platforms
* dreaMTouch
* PQ Labs
* Wacom

Snowflake Suite for Windows 7 Features

* Detailed user manual included with FAQ

* Developed on fast and reliable C++ platform

* Intuitive

* Customizable backgrounds, logo's, images, 3d models, videos etc

* Gesture recognition library

* TUIO / OSC (Open Sound Control) support (sending and receiving events)

* Low level API

* Hardware accelerated rendering

* Support for wide variety of media types (jpg,jpeg,tga,png,dds,gif,tif, tiff, bmp,mov,avi,wav,ogg)

* Advanced window handler that supports scaling and rotation

* Audio support

* Multi-threaded resource handler (For fast data visualization)

Snowflake Suite is also compatible with the following systems:

* TouchCore 

* Visual Planet

* NextWindow

* NEXIO Plugin

RELATED 

NUITEQ now offers 3M 22" true multi-touch displays bundled with its Snowflake Suite software

"The 3M multi-touch displays support up to 20 finger touches simultaneously and come with a one year warranty. NUITEQ's Snowflake Suite, is included at no additional charge, offering the benefits of a complete bundled product of hardware and software, being the perfect solution to get started with multi-touch technology."

Brochure (pdf) 

NUITEQ Flat 46" Multi-touch LCD 

Photo Credit: NUITEQ

Case Study
"NUITEC freely assisted handicapped children with their education, by applying multi-touch technology for Action For Kids, a UK based national charity working with children and young people with physical and learning disabilities.The children used NUITEQ’s award winning multi-touch software product Snowflake Suite, to enhance their way of learning, by offering a more playful and intuitive solution, in comparison to conventional learning methods."

More Multi-touch: New video from NUITEQ: Snowflake Suite Multi-touch Software on a 46" HD LCD

Here is the promotional information from NUITEQ:


"NUITEQ's Snowflake Suite off the shelf multi-touch software product showcased on a 46" flat full HD multi-touch LCD.  Available for purchase now.


Snowflake Suite is honored with a Stevie Awards finalist recognition for Best Product or Service of the Year 2009 in the category Media and Entertainment for the International Business Awards. Snowflake Suite is available to OEM's, SI's, VAR's, software engineers and end clients. Snowflake Suite comes with hands full of multi-touch applications, an API and a SDK.


Compatible with different multi-touch hardware technology platforms, including: 3M Touch Systems, N-trig, NextWindow, Lumio, Nexio, IR Touch, rear camera based systems, dreaMTouch and others."


"Natural User Interface Technologies AB (NUITEQ) is a Swedish technology company, that offers off the shelf and customized software for interactive single and multi-touch devices. Additionally NUITEQ executes large scale customization projects, concerning multi-touch technology hardware, software and services. In parallel, NUITEQ is working on other innovative emerging technologies within the field of Human Computer Interaction (HCI)."


Today's News:  NUITEQ wins MerlT Award for development through collaboration, growth potential and innovative thinking.


RELATED

NUITEQ Flat 46'' Multi-touch LCD

Key Features Size: 46" Snowflake Suite life time license included Full HD Simultaneous detectionand tracking of up 32 touch points Definite and reliable detection of touch points Real multi-touch function: no specific constraints, like temporal order or position of touch points Particularly suitable for embedded systems: Processing of complex evaluation steps via touch controller, e.g. keyboard emulation Sample rate: 50 frames/s Glass: 4 mm toughened safety glass Slim design: only 3 cm construction depth and 5 cm frame width USB interface and separate power supply No drift, no calibration required Applicable to all display technologies Protocol: TUIO Detection of size and position of objects in the active area 2 years warranty Electrical Power Supply: 19 V DC ± 20 % Power Consumption: 11 W Interface: USB 1.1, full speed   Functional Simultaneous Touch Points: 32 Touch Point Size: > 10 mm Spatial Resolution: < 2 mm Communication Protocol : Propriatary or ·· TUIO 1.0 (2Dcur and 25Dcur profiles) Scan Speed: 55 ms Mechanical Data Frame Dimensions: 1136 x 680 x 28 mm Frame Finish: Black powder coating (RAL 9011) Active Area: 1018.1 x 572.7 mm Window: 4 mm double-sided anti-reflective laminted safety filterglass Weight: 11 kg (including filter glass) Monitor: Direct fit to SHARP PN-465E (35,5 KG) Environmental Operating Temperature: 0 °C to 40 °C Operating Humidity: 20 % to 80 % (no condensation)

Also available as a horizontal solution.









NUITEQ Wiki
Harry van der Veen's NUITEQ Blog
(I've been following Harry van der Veen's journey since he was a university student. At the time, he was a leading member of the NUI-Group, creating a DYI multi-touch table as part of his studies. This was before Microsoft Surface was born.)