LM3LAB's Useful Map of Interactive Gesture-Based Technologies: Tracking fingers, bodies, faces, images, movement, motion, gestures - and more

Nicolas Loeillot, of LM3LABS, has been ahead of the natural user interaction/ interface game for many years as his company has expanded. He's done quite a bit of deep thinking about the work of his company, and has used this wisdom to create a nice concept map that describes how LM3LABS' solutions fit into the world of gesture-based control and interaction:




In my opinion, this chart would make a great template for mapping out other natural interaction applications and products!


Here is the description of the concepts outlined in the chart:


"If all of them belong to the “gesture control” world, the best segmentation is made from 4 categories:

• Finger tracking: precise finger tracking, it can be single touch or multi-touch (this latest not always being a plus). Finger tracking also encompasses hand tracking which comes, for LM3LABS products, as a gestures.
• Body tracking: using one’s body as a pointing device. Body tracking can be associated to “passive” interactivity (users are engaged without their decision to be) or “active” interactivity like 3D Feel where “players” use their body to interact with content.
• Face tracking: using user face as a pointing device. It can be mono user or multiple users. Face tracking is a “passive” interactivity tool for engaging user in an interactive relationship with digital content.
• Image Tracking: Augmented Reality (AR) lets users use images (flyers, real products, t-shirts, faces,…) to interact with digital content. AR can be markerless or marker-based. Markerless technology has advantages but marker-based AR is easier for users to understand. (Please note here that Markerless AR is made in close collaboration with AR leader Total Immersion)."  -LM3LABS

   If you are interested in this subject and want to view some good examples of off-the-desktop interfaces and interactions, take a look at the LM3LABS blog, as well as Nicolas Loeillot's Vimeo channel.  Also take a look at the sample of posts I've written about LM3LABS over the last few years - the links are at the end of this post.

I love LM3LABS' Interactive Balloon:

Interactive balloons from Nicolas Loeillot on Vimeo.


Interactive Balloons v lm3 labs v2 (SlideShare)

Background

I first discovered LM3LABS when I was taking a VR class and researching interactive, immersive large displays in 2005 or 2006.  Back then, there wasn't much information about this sort of technology.  A lot has changed since then!


I've learned quite a bit from watching LM3LABS (and others) grow, given my passion for postWIMP interactive technology and my commitment to blogging about this subject.   Nicolas has really worked hard in this arena.  As early as 2005, LM3LABS was working with Scala to provide "smart" interactive displays, and his company's applications have been supported by computer vision technologies for many years, allowing for gesture-based, or "touch-less" interaction, as demonstrated by the Catchyoo Interactive Table.  This application caught my eye back in early 2007, when I was working on projects for large interactive displays for my HCI and Ubicomp classes, and was thinking about creating a table-top application.


My hunch is that LM3LABS has set the foundation for further growth in the future, given the lessons they've learned by taking risks with postWIMP technologies over the past few years!


Previous Blog Posts Related to LM3LABS:
Interactive Retail Book (Celebrating history of Christian Dior from 1948-2010 (video)
Ubiq Motion Sensor Display at Future Ready Singapore (video)
Interactive Virtual DJ on a Transparent Pane, by LM3LABS and Brief Ad
LM3LABS' Catchyoo Interactive Koi Pond: Release of ubiq'window 2.6 Development Kit and Reader
A Few Things from LM3LABS
LM3LABS, Nicolas Leoillot, and Multi-touch
More from LM3LABS: Ubiq'window and Reactor.cmc's touch screen shopping catalog, Audi's touch-less showroom screen, and the DNP Museum Lab.

About LM3LABS

"Founded in 2003 by a team of passionate researchers, engineers, designers, and marketers from various international backgrounds, focused on fast transformation of innovation into unique products, LM3LABS is a recognized pioneer in computer vision-based interactivity solutions. Keeping a strong customer focus, LM3LABS' team of unique people pioneers new directions, explores new concepts, new technologies and new interactions.  Engaging, playful and magic, LM3LABS' products and solutions are always scalable and reliable"

info@lm3labs.com

Note to readers:

Over the past couple of years there has been an explosion of postWIMP technologies and applications, and with this pace, it has been difficult for me to keep abreast of it all. There is quite a bit I miss, given my full time job and daily life!

I welcome information about postWIMP interactive technologies and applications from my readers.  Due to time constraints, not interest, I am not always able to post about a topic as soon as I'd like.  That is OK, as my intention is not to be the first blogger to spread the latest tech news.  I like to dig in deep when I can and make connections between innovative, interesting technologies and the people and ideas behind them. 

About: Programming for interactive multimedia applications- WPF, Silverlight, EduSim, NeuroVR..

Part I Musings: Learning about application development and programming at mid-life:

I thought I'd write on a more personal level this time.

I'm a school psychologist, so in 2003, my motivation for taking computers at mid-life stemmed from my desire to create engaging interactive multimedia games for learning, games that could be played on hand-held devices as well as on the interactive whiteboards that I noticed were inching into my schools.

It was difficult for me to figure out how to get from Point A to Point B.

I shouldn't have been shocked to learn that most introductory programming classes provide instruction, as well as endless lab assignments, that are geared for people who want to make business forms and manipulate business-related data, build e-commerce websites, or create relational databases for... banks!

I now can make a mortgage calculator forms that adjust for various scenarios and provide cute error messages, in beginning Visual Basic.Net, C#, and Java. I can create a database that will let users look up part numbers for all sorts of widgets, in all sorts of combinations, and ensure that client data can be easily accessed in a nice looking form.

Why should I learn all of the old stuff when there are so many new avenues to explore?

Over the past few years, I've been fortunate to take a variety of classes that were not readily available just 8-10 years ago:

Computer/Internet Multimedia. Computer Music Technology. Game Design/Development. AI for Games. Ubiquitous Computing. Web Development Tools. Virtual Reality for Education and Training. These courses have motivated me to learn more about programming. The traditional programming courses had the opposite effect.


PART II Keeping up

I recently attended a day-long code camp at Central Piedmont Community College to learn more about Windows Presentation Foundation (WPF) and Silverlight. WPF and Silverlight provide the foundation for applications that can run on Microsoft's Surface. WPF and Silverlight provide designers and developers with more efficient ways of developing visualization and interactive multimedia applications.

The architecture behind WPF and Silverlight seems to align more with the way people think and envision, which in my opinion, gives hope for those of us who have toyed with the idea of programming, but were frightened off the first time they opened up a traditional programming textbook.


Part III Visualization and Interactive Multimedia

For more information about WPF, Silverlight, design, etc, read Sam Batterman's (a Microsoft Evangalist) recent blog post: Some thoughts about WPF and Data Visualization

Here are some pictures and text that I lifted from Sam's blog that will give you a picture of what I'm talking about:

"Here's an application that one of our partners built in a few weeks - all WPF and actually, not complicated code...getting that heart rendered was probably less than 100 lines of code. This app is used in a hospital for documenting heart surgery procedures. You can draw and annotate the 3D surface, rotate the heart, etc."





Can you imagine how kids would love to manipulate something like this in a science class?! This would be great on an interactive whiteboard or display.

This focuses on user experience, much more so than applications in the past. For more information about designing for user experience, take a look at the link to Bill Buxton's book, "Sketching User Experiences."

Bill Buxton's webiste, "Multi-touch systems that I have known and loved" is a good resource for those of you who'd like to get a better picture of multi-touch systems and interaction.

(I'll write more about WPF and Silverlight after I get my laptop repaired and have a chance to experiment some more.)

Croquet and EduSim

Right now, I've been experimenting with Croquet, which uses Squeak, and EduSim, which is powered by Croquet, to put together some learning activities for students.The best part is that Croquet is open-source, and all of the EduSim applications are free.

Here are some pictures that link to short video clips from the Greenbush EduSim website:






Neuro VR

Another application that I'm working with is NeuroVR.

NeuroVR is a free virtual environment that was designed for use in clinical settings. If you don't have access to virtual-reality hardware, you can still use NeuroVR on a desktop or large-screen display. Available 3D environments include an office, a supermarket, a park, a classroom, a poolside setting, and a home.

Andrea Gaggioli, Ph.D., is the Chief Technical Officer of the this project. He's also behind the Positive Technology blog, which is a great resource. Andrea is a researcher working at the intersections of psychology, neuroscience, and emerging technologies.

NeuroVR allows the clinician (or educator) to easily insert pictures, objects, and videoclips into each virtual world. Doors open and close, and you can move items around in the environment. For example, fruit set on a table can be moved over to a counter.

In my opinion, NeuroVR has potential not only in the area of rehabilitation and therapy, but in special education as well, particularly for students who have multiple special needs, including severe autism. A variety of "how-to" videos are posted on DaevornLi's YouTube channel.


Here are some videos clips to give you a better picture of the application:

The World is My Web Browser: Interactive Technology in Public Spaces

Off-the desktop Interactive technologies are spreading quickly, across many platforms.   It seems that within the next 3-5 years we will be able to have instant access to a social and dynamic web, with information and entertainment at our fingertips, anywhere we go. 


The "world-as-web-browser" concept is one I've touched upon in the past.  I'd like to focus on this topic in future posts.

I came across some of the information in this post on the Future of Media Blog,maintained by the folks at the Interpublic Emerging Media Lab, based in California. This blog is a must-read!


The Saatchi & Saatchi ad agency in the UK created interactive games that provide people waiting in bus shelters the opportunity to pass the time by bashing digital falling eggs.


A more utilitarian approach to interactive touch screens is the EyeStop bus stop, a concept created by researchers MIT's SENSEable City Lab, along with the city of Florence and the city's transit service:


MIT



MIT researchers unveil the EyeStop: "Bus stop of the future" can sense its surroundings (pdf)



SOMEWHAT RELATED

IPG Future of Media Blog

MindHabits' Happy Games Paired with Pharrell William's "Happy" Music!

We all could use more smiling people and happy music!

If you are looking for a short burst of happiness, try playing the free MindHabits demo games. I recommend the Matrix Trainer for starters. You can uncheck the "email" box if you don't want to sign up for the newsletter.

The games have upbeat music playing in the background, but you can listen to your own music. In the Matrix game, the objective is to tap as many happy faces that you can find, out of a number of frowny or sad faces.   The research shows that this is an effective way of reducing stress.  I have used the on-line version for years with students who have autism, and it is a fun and effective way of "training" them to focus on facial features and expressions.

The desktop version of the suite of games is just under $20.00, and is available for Windows and Macs.   The desktop version tracks data and and allows users to customize the games with their own photos.   

I am about to explore MindHabit's new mobile apps: Psych Me Up PRO! and Happy Cat! 

If you work with young people - special needs or otherwise, try playing the on-line demo with Pharrell William's "Happy" song in the background (see music video embedded below). 

 This is something that wouldn't hurt to try at home!  

You are playing the MindHabits Trainer online demo. Your progress will not be logged beyond this session. Copyright © 2008 MindHabits inc. inc l rights Reserved.

Note: 
I am sharing information about MindHabits because I have been following this company since 2005, when Dr. Mark Baldwin, the lead creator of the suite of games, gave a presentation.   Dr. Baldwin is a psychology professor at McGill University in Montreal, Canada. He has devoted his career to the study of social intelligence, and more recently, how technology can help people reduce stress, build self-confidence, and development, and maintain positive states of mind.



According to the MindHabits website, the Psych Me Up Pro!   ($.99) and PsychMeUp! (free) mobile apps were developed to help people focus attention on positive social feedback. A quiz is included with the application, as well as information about the research that supports the use of the games.  The "pro" version has more options.

The children's version of PsychMeUp! is Happy Cat.  The objective is to find the happy cats and ignore the grumpy cats.  The smiling cat will meow.

HOW MINDHABITS WORKS

RELATED

MindHabits Game Tips

MindHabits FAQ

MindHabits Update
Lynn Marentette, Interactive Multimedia Technology, 2/24/08

McEwan, K., Gilbert, P., Dandeneau, S., Lipka, S., Maratos, F., Paterson, K.B., Baldwin, M. (2014) Facial Expressions Depicting Compassionate and Critical Emotions: The Development and Validation of a New Emotional Face Stimulus Set. PLOS One DOI: 10.1317/journal.pone.0088783

Dandeneau, S. D., Baldwin, M. W. (2009) The buffering effects of rejection-inhibiting training against social and performance threats in adult students. Contemporary Educational Psychology, 34, 42-50

Stephane D. Dandeneau, Mark W. Baldwin, Jodene R. Baccus, and Maya Sakellaropoulo, Jens C. Pruessner (2007), Cutting Stress Off at the Pass: Reducing Vigilance and Responsiveness to Social Threat by Manipulation of Attention (pdf) Journal of Personality and Social Psychology, 2007, Vol. 93, No. 4, 651–666 American Psychological Association 0022-3514/07/$12.00 DOI: 10.1037/0022-3514.93.4.651

Adobe's Resources: Contextual Solutions, Rich internet applications across devices, and Open Screen.

I thought I'd share a few interesting resources from Adobe.  The company is focused on the future, and I plan to watch what unfolds.  Adobe believes that the next generation of the company's products will move us forward to a world of seamless access to content, across screens of all sizes, everywhere we go.

I'm hoping that this will not lead to an environment where we have difficulty escaping "skip intro" ads everywhere we turn.  Hopefully it won't be too bad!

I found a few gems of wisdom in the Adobe white paper,   Designing Contextual Solutions and Applications.  The paper provides a conceptual framework that I think would be helpful to web designers/developers who are interested in off-the-desktop application development.


Here is an excerpt from the introduction of the white paper:
"With its platforms and products that are inherently multichannel and multidevice, Adobe envisions a new class of “contextual solutions” taking hold in the marketplace. These solutions are broader in scope than a device, a campaign, or a single service; rather, they encapsulate the various contexts in which the end user exists, interacts, thinks, consumes, and purchases. And when content owners understand the different types of users and how they differ in their contexts, they are in a much better position to design experiences that are compelling and sticky. At the heart of every contextual solution is at least one or more contextual applications—an emerging type of application that spans multiple devices and uses system, user, and network contexts to provide an optimized user experience and deliver improved business results.

This white paper provides an overview of the types of contexts that compose a contextual solution and application. With examples given along the way, it culminates with a framework for considering all appropriate contexts while designing and developing a new contextual solution or an application that is part of an existing solution."


Another Adobe white paper is quite useful - Rich Internet Applications across devices: The emergence of contextual applications.  It provides examples of RIA's across devices, such as Amazon Video On Demand,  Demandbase, Atlantic Records: Fanbase, Finetune, and Times Reader 2.0.

Additional Adobe Resources
Contextual Applications
Open Screen Project:  "Singular experience, multiple devices"
Adobe Flash Platform

Saturday Quick Links: Digital Signage and Digital Out of Home (DOOH), via The Digital Signage Insider, Digital Signage Today, and more

The links:

Walmart and Microsoft Talk Up Digital Signage at the NRF
Bill Gerba, The Digital Signage Insider, 1/14/11
Wirespring


NRF: Walmart unveils ROI data for in-store digital signage campaigns
James Bickers, Digital Signage Today, 1/10/11


"C-Tailing" (Converged Retailing)  - NCR
NRF: Convergent technologies on the retail horizon
Cherrhy Butler, Digital Signage Today, 1/11/11

RELATED
Connecting Your Business to Devices and Customers with Windows Embedded

View more presentations from Microsoft Windows Embedded.
Barb Edson, Sr. Director of Marketing, Windows Embedded

Here are a few resources related to this topic:

4th Workshop on Pervasive Advertising (topics)

The Web Outside: Exploring place-based social media 

(LocaModa blog)

LocaModa's whitepapers

Blogs:

Digital Signage Today

Daily DOOH

The Digital Signage Insider

COMMENT
Designers and developers need to think about off-the-desktop technologies as a new form of the web/internet.  Information architects who understand interactive media/transmedia, cross-display/device, cross-platform, and interaction design within a broader context are sorely needed in this space.  


The presentation below, by Chris Thorne, Lead Information Architect and User Experience Consultant working for the BBC, provides a good overview about this topic:

Ubiquitous IA (Information Architecture): Building for change and web 3.0 

View more presentations from christhorne.
Chris Thorne

Stay tuned!

Over the past months I've been gathering video and photos of my encounters and interactions with digital signage as a consumer/customer/user during my every-day activities such as shopping, traveling, vacationing, and so forth, which will be included in a post or series of posts in the near future.   


I have content related to interaction with various QR tags,  interactive kiosks, interactive displays and TV on a cruise ship, an interactive touch screen at J.C. Penny, and more.  If you are a regular reader of this blog, you know that my interests include user experience of DOOH, interactive displays in public spaces, and so forth.  


Note:  Despite all of the technological innovations in this converging field, issues related to context, usability, and accessibility are not consistently addressed from a broader systems point of view.

Dance.Draw Project : Exquisite Interaction - Collaboration between Software Information Systems -HCI- and Dance Departments at UNC-Charlotte

DANCE.DRAW: EXQUISITE INTERACTION
(Updated)

"The movement of the visualizations are artifacts in real-time of the movements of the dancers. They draw while they dance, they dance together and they draw together. Every performance generates a new visual imprint." -DanceDraw website

Interactive multimedia technology, blended with the arts!

Dr. Celene LaTulipe, from UNC-Charlotte's Software and Information Systems Department, Professor Sybil Huskey, from the dance department, dance students, and others collaborated to create an amazing performance that I had the opportunity to see performed during the Visualization in the World Symposium in April (2008).

If you look closely, you will see that each dancer holds two wireless mice, one in each hand. The mice trigger the visualization that is projected in the background. Dr. LaTulipe has focused some of her research on two-handed computer interaction. It is interesting to see how her work has been applied to this beautiful "off-the-desktop" application.

Dance.Draw is a work in progress- visit the following links for more information:

Website (Updated)
Movie
Technical Info
Dr. Kosara's Eager Eyes post about Dance.Draw

Note:
Dr. LaTulipe was my HCI professor- Dr. Kosara was my Visualization/Visual Communication professor.

Blast from the 2009 past: News, Videos, and Links about Multi-touch and Screen Technologies

One of the things I like to do is share updates about the world of multimedia, multi-touch, gesture, screen, surface, and interactive technologies, focusing on off-the-desktop applications and systems. When I started this blog, I had to put forth quite a bit of effort just to FIND interesting things to blog about.  


These days, there are so many sources that focus on emerging - and now commonplace- interactive technologies, my main challenge is to filter the noise.  Where do I begin?

My archives are vast.   I randomly picked the year 2009 and came across one of my previous posts, "News, Videos, and Links about Multitouch and Screen Technologies."   The post is long, and contains a number of videos and links that probably will be of value to a future curator of the history of technology.


I welcome comments from readers who might be able to help me update information about various applications and systems I've featured on this blog in the past. 

The pictures are screenshots from the results of an  image search for "interactivemultimediatechnology".  Over the past 6 years, I've posted quite a few!

I want to play with mice! Microsoft's Multiple Multi-touch Mice Preview

Hot off the press from Microsoft's Applied Sciences Group at UIST 2009!
Mouse 2.0: Multi-touch Meets the Mouse

"In this paper we present novel input devices that combine the standard capabilities of a computer mouse with multi-touch sensing. Our goal is to enrich traditional pointer-based desktop interactions with touch and gestures. To chart the design space, we present five different multi-touch mouse implementations. Each explores a different touch sensing strategy, which leads to differing form-factors and hence interactive possibilities. In addition to the detailed description of hardware and software implementa-tions of our prototypes, we discuss the relative strengths, limitations and affordances of these novel input devices as informed by the results of a preliminary user study."

The following video is courtesy of Microsoft's Applied Sciences Group:



Music:  "Motion Blur", by Bjorn Hartman. (Bjorn is an HCI researcher and electronic musician.)

The researchers on the team: 
Nicolas Villar, Shahram Izadi, Dan Rosenfeld, Hrvoje Benko, John Helmes, Jonathan Westhues, Steve Hodges, Eyal Ofek, Alex Butler, Xiang Cao and Billy Chen


Here is a video preview/demo of the multi-touch mice prototypes from Microsoft's Applied Sciences Group Lab, courtesy of CrunchGear:

-CrunchGear

The Future of Interactive Technology: Links to VentureBeat's series of articles

Put your finger on it: The future of interactive technology
Matt Marshall, 8/12/09

"Touch" technology for desktop computing finally taking off
Al Monro (CEO, NextWindow) 8/28/09



Speech, touchscreen--been there, done that. What's the user interface of tomorrow?
9/2/09 Mary Czerwinski (Microsoft)

How phones emerged as main computing devices, and why user interface will improve
Andrew Hsu (Synaptics) 8/24/09

SOMEWHAT RELATED

Playing Tetris could be good for your brain, study says




Mind Research Network
BRAIN IMAGING SHOWS PLAYING TETRIS LEADS TO BOTH BRAIN EFFICIENCY AND THICKER CORTEX

"Over a three-month period, adolescent girls practiced Tetris, a computer game requiring a combination of cognitive skills. The girls who practiced showed greater brain efficiency, consistent with earlier studies. Compared to controls, the girls that practiced also had a thicker cortex, but not in the same brain areas where efficiency occurred."

"We showed that practice on a challenging visuospatial task has an impact on the structure of the cortex, which is in keeping with a growing body of scientific evidence showing that the brain can change with stimulation and is in striking contrast with the pervasive and only-recently outmoded belief that our brain’s structure is fixed,” said Dr. Sherif Karama, a co-investigator at the Montreal Neurological Institute. This study, published in the open-access journal BMC Research Notes, is one of the first to investigate the effects of practice in the brain using two imaging techniques. The girls completed both structural and functional MRI scans before and after the three-month practice period, as did girls in the control group who did not play Tetris. A structural MRI was used to assess cortical thickness, and a functional MRI was used to assess efficient activity."

"Microsoft is Imagining a NUI Future". You can, too!

Microsoft is Imagining a NUI Future
Steve Clayton, Next at Microsoft Blog, 1/26/11


"Our research shows that the vast majority of people polled in both developed and emerging markets see great potential for NUI applications beyond entertainment. This is especially true in China and India, where 9 out of 10 respondents indicate they are likely to use NUI technology across a range of lifestyle areas – from work, education and healthcare, to social connections, entertainment and the environment. We believe that taking technology to the next billion can be aided by NUI – making technology more accessible and more intuitive to a wider audience". - Steve Clayton, Microsoft


The people at Microsoft don't own the concept!  I'm a member of the NUI Group (May, 2007) and SparkOn.  Both are on-line communities where you can find people who live and breathe NUI, learn about their work, and even share designs and code. If you are intrigued by NUI - as a designer, developer, or user, please join us.

Note: 
I've been an evangelist and cheerleader for the NUI cause for many years.  If you search this blog for "post-WIMP", "NUI", "multi-touch", "gesture", "off-the-desktop", "natural user interaction", "natural user interface", or even "DOOH", you'll be provided with an overwhelming number of posts that include videos, photographs, and links to NUI-related resources, including scholarly articles.  There is a small-but-growing number of people from many disciplines, quietly working on NUI-related projects.

RELATED
Microsoft Plans a Natural Interface Future Full of Gestures, Touchscreens, and Haptics
Kit Eaton, Fast Company, 1/26/112
Rethinking Computing (video)
Craig Mundie, Microsoft
Interactive Touch-Screen Technology, Participatory Design, and "Getting It" - Revised
Touch Screen Interaction in Public Spaces:  Room for Improvement, if "every surface is to be a computer".

The World Is My Interface

THE WORLD IS MY INTERFACE  is the new  title of my Technology-Supported Human-World Interaction blog.  The name was just too long!

The first post written under the new title has a variety of links about ubiquitous computing, off-the-desktop interaction design, use of technology in public spaces, and so forth.  There are a few links to scholarly research and related projects, too.

About: Interaction Design (off the desktop)

Interaction design is a relatively new field that combines concepts related to human-computer interaction (HCI), mobile, pervasive, and ubiquitous computing (ubicomp), interface design, service design, user-experience design, interactive media design, and more.

According to Dan Saffer, an interaction designer at Adaptive Path, and author of Interaction Design: Creating Smart Applications and Clever Devices, interaction design is "about people: how people connect with other people through the products and services they use." If you are interested in learning more about designing for interaction, Saffer's book is a great starting point.

Saffer has recently established a wikki about interactive gestures, a site for the "dissemination of gestural interface information such as found on the iPhone and Wii." This is an important resource for those of us who are interested in developing useful interactive applications for emerging technologies.

(Related information can also be found on this blog.)