Leap Motion: My Dev Kit Arrived - Now What?! Thoughts About "NUI" Child-Computer-Tech-Interaction - and More

My Leap Motion developer kit arrived last week. I carefully unboxed the small device and tried out the demo apps that came with the SDK.  I'm doing more looking than leaping at this point.

I'd like to create a simple cause-and-effect music, art and movement application for my 2-year-old grandson, knowing that he'll be turning three near the end of this year.  It would be nice if my app could provide young children with enough scaffolding to support gameplay and learning over a few years of development.

Now that I'm a grandmother, I've spent some time thinking about what the evolution of NUI will mean for young children like my grandson.   Family and friends captured his first moments after birth with iPhones, and shared across the Internet.  Born into the iWorld, he knows how to use an iPad or smart phone to view his earlier digital self on YouTube, without ever touching a mouse or a physical keyboard.

The little guy is pretty creative in his method of interacting with technology, as I've informally documented on video.   He was seven months old when he first encountered my first iPad.  It was fingers-and-toes interaction from the start.  

In the first picture below,  he's playing with NodeBeat.  In the second picture, he's 27 months old, experimenting with hand and foot interaction, on a variety of apps.

My grandson is new to motion control applications, so I'm just beginning to learn what he likes,  and what he is capable of doing.  A couple of weeks ago, we played River Rush, from the Kinect Adventures game. He loved jumping up and down as he tried to hit the adventure pins. Most of the time, he kept jumping right out of the raft!  (I think next time we'll try Kinect Sesame Street TV, or revisit Kinectimals.)  


One of the steps I'm taking to prepare for my Leap Motion adventure is take a look at what people have done with it so far.  There are at least 12,000 developer kits released, so hopefully there will be some interesting apps to go along with the retail version of Leap Motion when it is released at Best Buy on May 19th of this year.

One app I really like is  Adam Somer's AirHarp, featured in the video clip below:


I also like the idea behing the following app, developed by undergraduate students:

Social Sign: Multi-User sign language gesture translator using the Leap Motion Controller (git.to/socialSign)
 
"Built at the PennApps Spring 2013 hackathon, Social Sign is a friendly tool for learning sign language! By using the Leap Motion device, the BadApples team implemented a rudimentary machine learning algorithm to track and identify American Sign Language from a user's hand gestures."

"Social Sign visualizes these hand gestures and broadcasts them in textual and visual representations to other signers in a signing room. In a standard chat room fashion, the interface permits written communication but with the benefit of enhanced learning in mind. It's all about learning a new way to communicate."-BadApples Team

There are a few NUI-focused tech companies that have experimented with Leap Motion. Today, I received a link to the following videoclip Joanna Taccone, of Intuilab, featuring their most recent work:
Gesture recognition with Leap Motion using IntuiFace Presentation

"Preview of our work with the Leap Motion controller. In the same spirit as our support for Microsoft Kinect, we have encoded true gesture support, not just mouse emulation, for the creation of interactive applications by non-programmers. The goal is to hide complexity from designers using our product, IntuiFace Presentation (IP). Through the use of IP's trigger/action syntax, designers simply select a gesture as a trigger - Swipe Left, Swipe Right, Point, etc. - and associate that gesture with an action like "turn the page" or "rotate the carousel". As you can see in this video, it works quite well. :-) We will offer Leap support as soon as it ships." -IntuiLab



Below is a demonstration of guys playing Drop Cord, a collaboration between Leap Motion and Double Fine.  From the video, you can tell that they had a blast!  

Here is an excerpt from the chatter:  "The thing is that everyone just looks cool..Yeah, I know, it doesn't matter what you are doing...it's got the right amount of speed-up-slow-down stutter-y stuff...it is like a blend of art and science.."

According to the website, Drop Chord is a "A music-driven score challenge game for the Leap Motion controller, coming soon for PC, Mac, & IOS from the creators of Kinect Party.."  

The following video is a demonstration of the use of Leap Motion to control an avatar and other interaction in Second Life:

Below are a few more videos featuring Leap Motion:


Control Your Computer With a Chopstick: Leap Motion Hands On (Mashable)


The Leap Motion Experience at SXSW 2013


LEAP Motion demo: Visualizer, Windows 8, Fruit Ninja, and More...



RELATED
Air Harp for Leap Motion, Responsive Interaction
Leap Motion and Double Fine team on Dropchord, give air guitar skills an outlet
John Fingas, Engadget, 3/7/13
Leap Motion Controller Set To Ship May 13 for Global Pre-Orders, In Best Buy Stores May 19.
Hands on With Leap Motion's Controller
Lance Ulanoff, Mashable, 3/10/13
Leap Motion website
Social Sign
IntuiLab
Leap Motion: Low Cost Gesture Control for Your Computer Display

SOMEWHAT RELATED
Kinect for Windows Academic: Kaplan Early Learning
"3 years & up. Hands-on play with a purpose -- the next generation way. This unique learning tool uses your body as the game controller making it a great opportunity to combine active play and learning all in one. Use any surface to actively engage kinesthetic, visual, and audio learners. Bundle includes the following software: Word Pop, Directions, Patterns, and Shapes."

Comment:
I've been an enthusiastic supporter of natural-user interfaces and interaction for years - back in 2007 I worked on touch-screen applications for large displays as a graduate student, and became an early member of the NUI group.  I'm also a school psychologist, and from my experience, I understand how NUI-based applications and technologies, such as interactive whiteboards and touch-tablets, such as the iPad can support the learning, communication, and leisure needs of students who have significant special needs.   It looks like Leap Motion and similar technologies have the potential to support a wide range of applications that target special populations, of all ages.

Paul Bourke's Experiments in Interactive Visual Immersion in Second Life - and more....

"Preprocessed spherical projections from inside a crystal including the Hershfield surface"
Evaluating Second Life as a Collaborative Tool for Scientific Visualization
http://local.wasp.uwa.edu.au/~pbourke/papers/cgat08/

I thought I'd devote a post to Paul Bourke's exciting work in the area of what I call "interactive visual immersion".

According to a bio on a math art website, Paul's working life "has revolved around high performance computing and computer graphics, in particular scientific visualisation: the use of computer graphics to represent scientific data with the aim of enhancing understanding of the underlying structure and processes." Paul is also interested in the use of "novel display technologies that can further enhance the understanding of large and complicated geometric datasets...These engage capabilities of our visual system not exploited by traditional computer displays..."

Paul recently participated in the the Computer Games, Multimedia & and Allied Technology 08: International Conference & Industry Symposium on Computer Games; Animation, Multimedia, IPTV & Edutainment. (CGAT'08, Singapore, April 28-30)

The above picture is from his paper, Evaluating Second Life as a Tool for Collaborative Scientific Visualization. (pdf), taken from Paul's webpage, where you can find a link to the slides from his talk.

Bourke points out that there are few software tools available that support shared, remote collaborative scientific visualization. What is available is expensive, exploratory, and/or difficult to install and operate. Bourke proposes that Second Life might prove to be a useful tool for collaborative SciViz, despite some current limitations. Second Life is a cross-platform 3D environment that was designed specifically for interaction between many people, and it allows for texture mapping, which is a plus for visualization work. Burke notes that there are limitations, such as Second Life's low geometric complexity.

Paul Bourke's website includes a page that lists all of his papers, articles, and seminars, with impressive visuals alongside each entry. Since I recently completed a visualization class and also gave a short presentation about accessible games for health, I found many of Bourke's papers intriguing:

Vertical dome (iDome): Visualisation and Navigable movies
Visualization for scientists, museums, public outreach, and education
Immersive environments and applications to gaming
Exploiting our sense of touch for scientific visualization.
Tactile Visualization: Feel your data!
Apple technology powering displays that engage the human visual system
iDome and digital projection into hemispherical domes

Take the time to explore Bourke's entire website. Look at his texture library and his "other" section!

FYI: Here is a link to the CGAT'08 list of speakers, complete with bios and abstracts.

More about virtual worlds in education: Edutopia's articles and resources

The recent on-line newsletter from Edutopia has a list of links of articles and media about virtual worlds and games in education. This information compliments the video, "Virtual Worlds Tour", highlighted in my previous post.

Get a Life: Students Collaborate in Simulated Roles (Laila Weir)

The School of Second Life (Wagner James Au)

Simulation Nation: The Promise of Virtual Learning Activities: Inventive computer sims can turn dull lessons into hyperreal experiences, if we can get educators to use them" (Marc Prensky)

Related video: No Gamer Left Behind "Computer simulations area natural learning tools to a generation of video game players"

Let the Games Begin: Entertainment Meets Education (Jenn Shreve)

Related:

Loud and Clear: Students Find Their Voices Through Multimedia (Edutopia Staff)

Edutopia is part of the non-profit George Lucas Educational Foundation

Post a comment if you are using virtual worlds or games in your school!

Post from Jonathan Tarr, HASTAC, about Sharon Burn's Virtual Worlds Tour (8-minute overview video)

If you are interested in interactive multimedia, you might be curious to learn more about virtual 3D worlds. I tried out Second Life for a couple of weeks in 2005, and found that for me, I simply don't have the time. I often don't have enough time in my "first life"!

Since 2005, Second Life and other virtual worlds have exploded in population. Here is a good overview and update, including information and views of children's virtual worlds such as Whyville, Webkins, and Penguin.

FYI: I'm always flattered when I've been "re-blogged", so I hope that Jonathan Tarr doesn't mind that I've reblogged his post from the HASTIC website:

March 4, 2008 - 3:34pm. VIRTUAL WORLDS TOUR FROM SHARON BURNS


"MacArthur Foundation CIO Sharon Burns (whom I met briefly last month in Chicago) has posted the first part of an excellent tour through various virtual worlds on Youtube, for those who are looking for a quick and smart overview. I learned more about it in 8 minutes than I have in most longer presentations about virtual worlds, so there is certainly some benefit to brevity. There's also a useful writeup of the video on Susan Tenby's blog at Netsquared, if you want an even briefer writeup and some additional information."



What is HASTAC?

"A consortium of humanists, artists, scientists, social scientists, and engineers from universities across the country, HASTAC ("Haystack") is committed to new forms of collaboration across institutions, disciplines, and communities to promote creative uses of technology. Since 2003, we have been developing tools for multimedia archiving and social interaction, gaming environments for teaching, innovative educational programs in information science and information studies, virtual museums, and other digital projects. HASTAC leaders have served as consultants to U.S. and international organizations and governments on grid computing and cyberinfrastructure. Our aim is to promote expansive, innovative uses of technology in formal education and lifelong learning."

Stanford University's Humanities Lab; Metaverse U,

On-line interactive virtual worlds such as Second Life, as well as online 3D games have become increasingly popular among adults and teens. It isn't too surprising that virtual worlds and videogames are now topics of formal study at the graduate level!

Researchers from the Stanford Humanities Lab have been involved in some interesting work. For several years, they have been studying virtual worlds, participating in Second Life, documenting the of the history of video games, working towards archiving digital virtual worlds, and participating in variety of other trans-disciplinary projects.

The Stanford Humanities Lab will be hosting a two-day conference in February 2008 that focuses on the topic of virtual worlds:

"The proliferation of virtual worlds has garnered tremendous interest, both in the media and in academia. To explore the cultural, technological, legal, and economic issues surrounding virtual worlds, the Stanford Humanities Lab is excited to announce Metaverse U, a conference to be held on Saturday, February 16th and Sunday, February 17th, 2008. We invite you to participate in this unique event. Metaverse U aims to challenge traditional conference models with more collaborative and inclusive opportunities for participants, both virtual and physical. To ensure that its impact extends well beyond the actual conference, video from Metaverse U will be streamed live to the web, free of charge. After the conference, these videos will be archived and become part of a global conversation on virtual worlds."

If you are interested in learning more about interactive multimedia, 3D virtual worlds, on-line communication and collaboration, and the broader issues surrounding this area, don't miss the opportunity to attend - in person or on the web.

FYI

Metaverse U People

Stanford Humanities Lab's History of Computer Games Collection