Gesture Markup Language (GML) for Natural User Interaction and Interfaces

Quick post:

Gesture ML Wiki 

"GML is an extensible markup language used to define gestures that describe interactive object behavior and the relationships between objects in an application.  Gesture Markup Language has been designed to enhance the development of multiuser multi-touch and other HCI device driven applications." -Gesture ML Wiki

GestureML was created and maintained by Ideum. 

More information to come!

Photo credit: Ideum

RELATED
Ideum Blog

OpenExhibits Free multitouch and multiuser software initiative for museums, education, nonprofits, and students

GestureWorks  Multi-touch authoring for Windows 8 & Windows 7

Ideum's GestureWorks Platform Will Let Developers Create Complex Gestures for Ultrabook Apps  -Alex Williams, Tech Crunch, 11/18/12

5 Perspectives On The Future Of The Human Interface -Alex Williams, Tech Crunch, 11/4/12

NUI Group

More to come!

Human Computer Interaction + Informal Science Education Conference (NUI News)

I recently learned of the HCI + ISE conference, funded by the National Science Foundation and organized by Ideum and Independent Exhibitions that will provide the groundwork for the future of the development and design of interactive computer-based science exhibits.

Science museums have a long history of interactivity, well suited to groups of "explorers", such as families or students visiting on a field trip.  

What is really exciting is that new interactive applications and technologies have the power to transform the way people learn and understand science in a collaborative and social way.  Innovations in the field of HCI - Human-Computer Interaction- such as multi-touch and gesture interaction, are  well-suited to meet the goals of science education for all, beyond the school doors and wordy textbooks. 

Below is a screen-shot of the conference website, a description about the conference, quoted from the site, and some related resources.

About the HCI+ISE Conference

"HCI technologies, such as motion capture, multitouch, augmented reality, RFID, and voice recognition are beginning to change the way computer-based science exhibits are designed and developed. Human Computer Interaction in Informal Science Education (HCI+ISE) is a first-of-its-kind gathering to explore and disseminate effective practices in developing a new generation of digital exhibits that are more intuitive, interactive, and social than their predecessors."

"The HCI+ISE Conference, to be held in Albuquerque, New Mexico June 11-14 2013, will bring together 60 museum exhibit designers and developers, learning researchers, and technology industry professionals to share effective practices, and to explore both the enormous potential and possible pitfalls that these new technologies present for exhibit development in informal science education settings."

"HCI+ISE will focus on the practical considerations of implementing new HCI technologies in educational settings with an eye on the future. Along with a survey of how HCI is shaping the museum world, participants will be challenged to envision the museum experience a decade into future. The conference results will provide a concrete starting point for exhibit developers and informal science educators who are just beginning to investigate these emerging technologies and design challenges in creating these new types of exhibits."

Why HCI+ISE?
"Since the mid-1980s informal educational venues have increasingly incorporated computer-based exhibits into their science communication offerings in an effort to keep pace with public expectations and make use of the expanding opportunities these technologies provide. The advent and popularity of once novel HCI technologies are becoming commonplace: the Wii and Microsoft Kinect now allow for motion capture video games, tablet PCs have multitouch interaction, and smart phones and other devices come standard with voice recognition. Yet many museums are still developing single-touch and trackball-driven, single-user computer kiosks."

"Science museums have a long history of championing hands-on, physical, and inquiry-based activities and exhibits. This vast experience has only just begun to be applied to interactive computer interfaces. Along with seasoned science exhibit developers, the Conference will draw upon individuals outside of ISE who will provide fresh insight into the technologies, design issues, and audience expectations that these visitor experiences present."

Involvement and Findings
"HCI+ISE will bring together a diverse group of practitioners and other professionals to discuss (and in some cases share and prototype) new design approaches utilizing emerging HCI technology. Please see our Apply page to learn how you can participate. Conference news and findings will be distributed through a variety of ISE and museum websites, including this one."

"We welcome your questions and comments about the HCI+ISE Conference."

CONTACTS

Kathleen McLean of Independent Exhibitions
& Jim Spadaccini of Ideum
HCI+ISE Co-chairs

RELATED
Independent Exhibitions

Open Exhibits

"Open Exhibits is a multitouch, multi-user tool kit that allows you to create custom interactive exhibits."
CML:  Creative Mark-up Language
GML: Gesture Mark-up Language
GestureWorks
Ideum

Building Interfaces for the New Decade: Conference at the Tech Museum, San Jose, California

Here is information about a day-long conference that I'm sure will be well worth attending:

November 15, 2011 (Tuesday)

Building Interfaces for the New Decade Conference

The Tech Museum

San Jose, California

If you can't make it to San Jose, the conference can be attended "virtually".  You can register for the conference via the EventBright website.


Here is the list of speakers and presentations for the conference:


Tamara Schwarz, Senior Manager of Experience Design, Chabot Space & Science Center
"RFID-enabled Interactive Exhibits, or... What's a warehouse inventory management technology doing in an interactive science center?"
Christopher Stapleton, Creative Venture Catalyst, Simiosys
"The Phydgital InterSpace -The Next Generation of Museum User Experience and Workshop: Experiential Makeover for Learning"
Jim Spadaccini, Director, Ideum; Principal Investigator, Open Exhibits
"Open Exhibits - Community Software for Multi-touch and Motion Recognition"
Rick Ernst, Lead Designer, Ogmento
"How to get People off the Couch and Interact with the World with Augmented Reality Games"
Bob Ketner, Curator, The Tech Test Zone
"The Tech Virtual and The Tech Test Zone Open Platforms"
John MacDuffie Woodburn, Co-Developer, Pixel, and Graduate Student, University of the Arts, PA, & Matt Miller, Co-Developer, Pixel
"Pixel - Using Eye Tracking to Engage and Educate Your Audience"
Ross Smith, Ph.D, Deputy Director, Wearable Computer Laboratory, University of South Australia
"Bringing Fingertip Sculpting Techniques to the Digital Artist"

Below is an excerpt from the conference website:
"Join us on Tuesday, November 15, 2011 for a gathering of the most innovative developers of new interactive technologies at The Tech Museum's Test Zone 2011...Connect with entrepreneurs, developers, and researchers working on cutting-edge interfaces such as:
Augmented Reality
Context Aware 
Deformable Surfaces
Eye Tracking
Feedback
Gestural Interfaces
Gigapixel Images
Motion Sensing
Multi-touch
Natural User Interfaces
RFID
Smart Museum Systems
Tablet Computers
Ubiquitous Media 


Find out how museums and other public-space designers can use these technologies to enhance visitor learning and experience. Experience the interactives firsthand at the opening of The Tech Test Zone Gallery, a new gallery dedicated to temporary exhibitions of prototypes from corporate, academic, and nonprofit partners."

RELATED

About the Tech Museum:

"The Tech Museum is a hands-on technology and science museum for people of all ages and backgrounds. The museum-located in the Capital of Silicon Valley -is a non-profit learning resource established to engage people in exploring and experiencing technologies affecting their lives. Through programs such as The Tech Challenge presented by Cisco, our annual team design competition for youth, and internationally renowned programs such as The Tech Awards presented by Applied Materials, Inc., The Tech Museum celebrates the present and encourages the development of innovative ideas for a more promising future." -Tech Museum YouTube Channel

Hack the Future Day at the Tech Museum

Emerging Interactive Multimedia, New Models of HCI for Museum Exhibits (Course offered by Ideum's Jim Spadaccini, plus info about the MT55 multi-touch table)

Jim Spadaccini, the director and founder of Ideum, will be teaching a course on exhibit development through the University of Victoria, "Emerging Exhibits: Exploring New Models of Human Computer Interaction (HCI).  The excerpt from the course description provides a glimpse of how emerging technologies are beginning to change the museum experience:


"Computer-based interactive exhibits are undergoing a major transformation. The lone, single-user kiosk is now being replaced by multitouch tables and walls, motion-capture spaces, networked installations, and RFID-based exhibits. Advances in augmented reality, voice recognition, eye tracking, and other technologies promise even more radical change for exhibits in the near future."


I've been following Jim's journey with Ideum, a multimedia design firm that collaborates with museums and related non-profits, for many years, and I am impressed with the work of this company.  In addition to his work at Ideum,  Jim serves as the Principal Investigator of a National-Science Foundation sponsored open-source exhibit software project, Open Exhibits, which provides a free software development kit that supports the creation of multi-touch and multi-user software applications for museums and educational settings.


I'm happy to put in a plug for Ideum's latest product, the MT55 Platform Multi-Touch Table. It incorporates a range of features that I'm sure will meet the needs of museum visitors.  In my opinion, this table would be a fantastic resource for all types of libraries, including those in K-12 settings.

The MT55 Platform Multi-touch Table, from Ideum

The MT55 Platform Multitouch Table from Ideum on Vimeo.  (Note: This video features music by Moby, the track "Sevastopol" on his current album, Destroyed. The music was used with the artists' permission. Learn more at: moby.com") - Ideum

"The thinnest, largest, most powerful multitouch table available.The MT55 Platform multitouch table houses a powerful computer and a 55-inch interactive LCD display that responds to 32 touch-points, inside a rugged aluminum body."

"The bright 55″ 1920×1080 HD display has a 5,000,000:1 contrast ratio. A wide 178-degree viewing angle accommodates multiple users around the table. The optical multitouch system supports 32 simultaneous touch points for collaborative interaction. The system is multitouch-enabled from start-up, and runs Windows 7 64-bit professional edition."


"The integrated computer is packed with power. It contains an Intel® Hyper-Threaded DualCore i5® which runs at 2.66 GHz, 8GB of RAM, and a 128 GB solid-state drive (upgradeable to an i7®)."


"The table comes complete with WIFI, Bluetooth, and Ethernet connectivity. It also has multiple HDMI outputs that allow you to easily mirror the table's display, extend the desktop, or connect to and display from another computer or HDMI device."

"Convenient, but secure ports: CAT5, HDMI, and USB 2.0 are available on both the side on bottom the table."

"The MT55 Platform includes blue LED under-lights to illuminate the area beneath the interactive surface (custom LED colors are available). Every MT table includes a sophisticated internal cooling cell to maintain operating conditions that exceed the optimum environmental specifications for the internal components."

"The interactive surface of the MT55 Platform protected by a sheet of hardened, crystal clear, low-iron 5mm tempered glass surface...As an option, we offer Sevasa HapticGlas®, produced exclusively for Ideum. Micro-etched HapticGlas® provides tactile feedback, reduces fingerprints, increases scratch resistance, and directs user focus." -Ideum
 
RELATED
High-res photos of the MT55 Platform

GestureWorks Software
Open Exhibits
Ideum
Open Exhibits Tuio Kinect

Multi-touch and Gesture Interaction News and Updates You Might Have Missed (Part I)

Over the past couple of months, I've come across many interesting links related to multi-touch and gesture interaction, but I haven't had time to devote a thoughtful post to each one.  "Part I",  is a nice collection of experimental, commercial, and non-commercial efforts by a variety of creative technologists, with a smattering of industry news that might be of interest to IMT readers. 


Ideum's MT55 HD Multitouch Table 4/19/11

New MT55 HD Multitouch Table Now Shipping,  Jim Spadaccini, Ideum Blog 4/11/11

Smithsonian American Art Museum to Open Education Center  Sara Beladi, NBC Washington News, 4/4/11 (Rumor has it that the Smithsonian American Art might include touch and multi-touch displays in it's plans for a new education center.  The center was funded by an anonymous $8 million dollar gift.)

Some Milestones in Computer Input Devices: An Informal Timeline

Bill Buxton, Microsoft Research, 4/7/11 - Includes lots of pictures, links to videos, and more information of what might be the first touch-screen.  Also see Bill Buxton's companion website, Multi-Touch Systems that I have Known and Loved, updated on 3/21/11.  Bill Buxton knows all (almost!)

How to make a Multitouch Biggie (Seth Sandler)

"The MTbiggie uses the "Front Diffused Illumination" multitouch technique, with ambient infrared light and a DIY infrared webcam. The MTbiggie is similar to the MTmini, but includes a projected image and infrared webcam (rather than a normal webcam)...The MTbiggie isn’t the most stable and robust setup, but it is the easiest to build. To see other methods of building more stable multitouch displays, view the full multitouch display list." -Seth Sandler

(Also check out NodeBeat, a multi-touch music/audio sequencer/generator app by Seth Sandler and Justin Windle)

IntuiLab announces support for Microsoft Kinect interactive software solutions

Intuilab, 4/13/11

"IntuiLab, a global leader in surface computing software applications, today announced support for the revolutionary Microsoft Kinect device across its full line of IntuiFace products and solutions including IntuiFace Presentation and IntuiFace Commerce...Microsoft Kinect brings distant gesture control to interactive solutions. These gesture controls allow users to interact with displayed digital assets from a distance at their own pace and path – for example, browsing through a large quantity of products in a store catalog or manipulating 3D models (such as a mobile phone) – all without having to actually touch the screen..."  -IntuiLab (Take a look at the IntuiLab team- an interactive page!)

EON Reality Launches EON Interact - Interact with 3D Data by Using Your Bare Hands Animation xPress, 3/31/11

Sparkon:  Videos and links related to multi-touch and gesture-based applications

Official Kinect SDK to be Open Source, Josh Blake, Deconstructing the NUI, 4/18/11  

9 This bit of news excited me, but don't get your hopes up. If anyone knows what will happen with the Kinect SDK, please leave a comment.)

"Update 4/18 7:34pm: Mary Jo Foley picks up this story, but the Microsoft spokesperson she talked to denied that the Kinect SDK will be open source. As she notes, Microsoft has pulled 180’s before regarding Kinect. After spokespeople initially were hostile to the idea of Kinect hacking, Xbox executives later embraced the idea that people are using Kinect for non-gaming purposes on the PC. Let’s hope Microsoft stays open to this idea." -Josh Blake

New Channel 9 Video:  Josh Blake on NaturalShow Multi-touch in WPF, Pete Brown, 10rem.net, 4/22/11

Eden: A Professional Multitouch Tool for Constructing Virtual Environments (pdf)

Kenrick Kin, Tom Miller, Bjoern Bollensdorff, Tony DeRose, Bjoern Hartmann, Manees Agrawala (Pixar Online Library)

Flight Race Game on 3DFeel lm3Labs, 4/18/11


JazzMutant Lemure Version 2 : "The only multi-touch and modular controller for sequencers, synthesizers, virtual instruments, vjing and lights, now even better."


Harry van der Veen's Multitouch Blog (NUITEQ)


Stantum "Unlimited Multi-Touch" Latest News

At Immersive Labs, Ads Watch Who Looks at Them Amy Lee, Huffington Post, 4/26/11 

Immersive Labs

Hard Rock Cafe International Using NextWindow Touch Screens:  "Rock Wall Solo displays enhance music lovers' experience in Seattle, Dallas, Detroit and Berlin" 4/12/11 (Full press release pdf)
Music on Touch Screens (NextWindow)

Razorfish: Thoughts on MIX 11 ,James Ashley, Razorfish Blog, 4/20/11  Also see: Razorfish Lab's Prototypes

"The multitouch microscope brings new dimensions into teaching and research. Researchers at the Institute for Molecular Medicine Finland (FIMM) and Multitouch Ltd have created a hand and finger gesture controlled microscope. The method is a combination between two technologies: web-based virtual microscopy and a giant size multitouch display."
"The result is an entirely new way of performing microscopy: by touching a table- or wall-sized screen the user can navigate and zoom within a microscope sample in the same way as in a conventional microscope. Using the touch control it is possible to move from the natural size of the sample to a 1000-fold magnification, at which cells and even subcellular details can be seen."  -Multitouchfi  Also see the Multitouch website.


Big Size Multitouch Display Turned into a Microscope, Microscopy-News, 3/28/11

Mac OX 10.7 Lion: new multi-touch gestures, Dock integration for Expose, Launchpad, Mission Control Appleinsider, 4/14/11


Vectorform App featured in Royal Caribbean's Video Promotion: James Brolin, Dean Cain get hands-on with Vectorform app Alison Weber, Vectorform Blog, 3/3/11


3M Touch Systems's YouTube Channel

Social Mirror 3D Gestural Display, Now Using Kinnect:  SnibbeInteractive

Designing for Multitouch Tables and Surfaces, by Erin Rose, Open Exhibits Blog

If you are interested in exploring collaborative tabletop applications, take a look at the Open Exhibits blog. Erin Rose's recent post, "Designing for Multi-touch Tables and Surfaces", is a good overview of lessons learned over the past couple of years in design, development, and implementation of multi-user interactive tabletop applications.

Although the focus of Open Exhibits is on applications and systems designed for museum exhibits, many of the design challenges hold true for similar applications in other settings, such as classrooms, libraries, and other public spaces.

Erin's post explores each of the following topics in more detail:

• Don't forget that the table is omni-directional.
• Individual control of objects encourages multi-user interaction.
• Promote collaboration, founded in healthy competition.

(Erin Rose is a developer and community liaison for Open Exhibits.)

RELATED
Exhibit Files
Jim Spadaccini
Visitors Explore L.A. in Google Maps and Flickr Mashup.

Grant from the National Science Foundation for Multi-touch Interactive Museum Exhibits!

This is interesting!

"Open Exhibits is a National Science Foundation-funded initiative to develop a library of free and open multitouch-enabled software modules for exhibit development. Build using the popular Adobe Flash and Flex authoring tools, museum professionals will be able to create innovative floor and web-based exhibits easily and inexpensively" -- Open Exhibits

VIDEO: Introducing Open Exhibits: Open Source Exhibit Software


Open Exhibits Core is based on the commercial GestureWorks software package.


RELATED
Open Exhibits Funded by the National Science Foundation
Jim Spadaccini, Open Exhibits Blog 9/21/10

About Open Exhibits
Jim Spadaccini, of Ideum, is the Principal Investigator of the Open Exhibits project.  Kate Haley Goldman is the co-PI and main researcher.  The three museum partners are the Don Harrington Discovery Center, the Maxwell Museum of Anthropology, and the New Mexico Museum of Natural History and Science. 


Ideum website

BP Oil Spill Multi-touch Map Mashup by Ideum

BP Gulf of Mexico Oil Spill Multi-touch Exhibit

In response to the BP oil spill in the Gulf of Mexico, the team at Ideum has created a free version of their multi-touch-enabled Google Map and Flickr mashup application to educational organizations, including museums and aquariums.  The information included on the map includes oil spill and fishing restriction data from the National Oceanic and Atmospheric Association (NOAA), along with pictures taken at the scene in and around the region of the spill.

Ideum is a company that "designs and creates interactive computer-based exhibits, multitouch installations, rich Internet applications, and social networking sites for museum and Web visitors. We work closely with museums, nonprofit organizations, and socially responsible companies to create memorable visitor experiences."   Ideum uses their own multi-touch framework called GestureWorks to create interactive applications.

Jim Spadaccini is the director and founder of Ideum. He formerly was the Director of Interactive Media at the Exploratorium in San Francisco.

Ideum is working on a version of the program optimized to work on 3M's  22" multi-touch system.  A single-touch version for the web might be available if there is interest.

RELATED

BP Oil Spill in the Gulf of Mexico Flickr Group

Ideum's oil spill-related pictures on Flickr

Update on Ideum: Multitouch, Multiuser Table at the California Science Center

Ideum is a company that has been around since 1999.  It focuses on the design and creation of "computer-based exhibits, multitouch installations, rich Internet applications, and social networking sites for museum and Web visitors."   Gestureworks is the software developed by Ideum, which is an authoring solution for Adobe Flash and Adobe Flex.

Below is some information, including a video clip, of one of Ideum's most recent installations a multiuser, multitouch table at the California Science Center.

Case Study:  Ideum's  L.A. Zone Multi-touch, Multiuser Table (Jim Spadaccini)

Here is an excerpt from Jim Spadaccini's post:
"This multiuser, multitouch table exhibit in the L.A. Zone allows visitors to explore a satellite image and map of the LA Basin and view overlays in thematic areas that explore fire, air, water, and earth. In addition, points of interest are placed on the map in the form of images and videos. All of the content and navigational controls are available in English and Spanish. The exhibit software runs on a custom-built 50" multitouch table that supports 60+ simultaneous points of input."

The Ideum team has spent a great deal of attention to the way groups of museum visitors interact with their multitouch applications. In the past, they've noticed that on a 50" surface, visitors would experience interaction conflicts. In the present application, more than one visitor can manipulate the content without changing the interaction of others.  Spadaccini points out that multi-user control of touch screen exhibits is new, so interface design concepts developed for single-user interaction are no longer useful.  Social interactions around a multiuser display are now something to consider.

In my opinion, what the Ideum team learns through their design process can be quite useful to those of us planning to develop multiuser educational applications for interactive surfaces and displays. What I liked about this exhibit is that it has a monitor that mirrors the display, so people can view the interaction from afar.  This feature would be useful in classroom settings that have an interactive table and an interactive whiteboard. The interactive whiteboard as the secondary display so the teacher and other students could see what the group at the table was exploring.



Multitouch Google Maps and Flickr Mashup Built with Flash



Update: A video showing how Ideum's multi-touch table can withstand all sorts of heavy-duty treatment:




Note:I have been pretty busy over the past couple of months- I have much that I haven't yet blogged about.  More to come!

Ideum's GestureWorks vs Adobe AIR 2 and Flash Player 1.0 comparison of multitouch and gesture support

Jim Spadaccini, of Ideum, shared information about his company's product, Gestureworks, highlighting how it provides better multi-touch and gesture support than Adobe AIR2 and Flash Player 10.1. Gestureworks supports multiple-point drag, rotate, and scale at the same time. In the video, the application is demonstrated on an HP Touchsmart 600 and a 3M multitouch screen.

Adobe AIR 2 and Flash Player 10.1  vs Gestureworks 1.0: A direct comparison of multitouch and gesture support


"A direct comparison between the built-in support for multitouch found in Adobe Flash Player 10.1 beta / Adobe AIR 2 and that of the Gestureworks multitouch framework for Flash. More about this comparison can be found on the Gestureworks website (http://www.gestureworks.com) and the Ideum website (http://www.ideum.com)  There is a blog post with more about this comparison and links to all of the example files at: www.ideum.com/2010/01/true-multitouch-wi th-adobe-flash/ "

True Multitouch with Adobe Flash - Jim Spadaccini


GestureWorks Supported Gestures

Example of Ideum's GestureWorks multi-touch, multi-user design for an exhibit a the Vancouver Aquarium:

Interactive Touch Tables are Multiplying! Ideum's new 100" multi-touch museum display; Ubisoft's Ruse on a Surface....

Photo from the Museum Exhibit and Design News/Ideum Blog

The above table is 100", created for Nashville's Adventure Science Center's Sudekum Planetarium, is based on Ideum's mt2 table: "Built for museum environments, the high-resolution mt2 Table supports intuitive gestures, allowing visitors to interact and explore custom exhibits. With multiuser support, mt2 Table serves as digital gathering place, allowing visitors to interact with museum content and each other."

Two computers power this table. Natural User Interface's Snowflake handles the optical support and tracking, and the second computer will be running the EM spectrum application, which is developed in Adobe Flash. It is a work in progress. For more information, visit the Ideum blog.

Here is a video trailer of Ubisoft's R.U.S.E. played on a multi-touch table. Much of the trailer appears to be a simulation. I liked the part where the battleships are traveling over the water. According to a blog post, the table depicted in the trailer is the Intuiface, by Intuilab.

Photos of Ubisoft's R.U.S.E. game, running on a multi-touch table:




Photo from the Ubisoft Blog

Intuiface Information PDF
Intuilabs Website

If you visit the Intuilabs website, be sure you interact with the photos of the team! I couldn't get the pictures to change when I touched the screen of my HP TouchSmart PC, so I think they are activated by a mouse-over. It would be nice if they could touch-enable their website.)

Ideum's multi-touch, multi user table prototype

The above picture is from Jim Sapadaccini's post about Ideum's mutli-touch table prototype, using NUI (Natural User Interface) software, "Snowflake". Ideum is working on gesture-recognition software for use in future projects.

Ideum will be designing for museum installations, and will include user-testing as they work towards applications that will support collaboration among museum visitors as they interact around the exhibits.

For more information see the Ideum blog, and also Ideum's portfolio page.