In Part I of my "mini-series" about Tangible User Interfaces, I discussed the origins of TUI and provided some examples of Siftables. In this section, I've provided some links to information about Tangible User Interfaces and the abstracts of two articles pertaining to TUI's in educational settings. Zen Waves: A Digital (musical) Zen Garden
Reactable More about the Reactable "The reactable hardware is based on a translucent, round multi-touch surface. A camera situated beneath the table, continuously analyzes the surface, tracking the player's finger tips and the nature, position and orientation of physical objects that are distributed on its surface. These objects represent the components of a classic modular synthesizer, the players interact by moving these objects, changing their distance, orientation and the relation to each other. These actions directly control the topological structure and parameters of the sound synthesizer. A projector, also from underneath the table, draws dynamic animations on its surface, providing a visual feedback of the state, the activity and the main characteristics of the sounds produced by the audio synthesizer." The Bubblegum Sequencer: Making Music with Candy
BodyBeats: Whole-Body, Musical Interfaces for Children (pdf) Telestory is a Siftables application that looks like it would be quite useful for supporting children who have communication disorders or autism spectrum disorders.
"Telestory is an educational, language learning application created by Seth Hunter. In this video, the child is looking at a television screen. He can control onscreen characters, events and objects with the siftables. For example, he has the dog and cat interact by placing the dog and cat siftables next to each other."TeleStory Project Website Here is a video of how Siftables can be used as equation editors:
"Recently, the number of children having autism disorder increases rapidly all over the world. Computer-based training (CBT) has been applied to autism spectrum disorder treatment. Most CBT applications are based on the standard WIMP interface. However, recent study suggests that a Tangible User Interface (TUI) is easier to use for children with autism than the WIMP interface. In this paper, the efficiency of the TUI training system is considered, in comparison with a conventional method of training basic geometric shape classification. A CBT system with TUI was developed using standard computer equipment and a consumer video camera. The experiment was conducted to measure learning efficacy of the new system and the conventional training method. The results show that, under the same time constraint, children with autism who practiced with the new system were able to learn more shapes than those participating in the conventional method."
"External representations have been shown to play a key role in mediating cognition. Tangible environments offer the opportunity for novel representational formats and combinations, potentially increasing representational power for supporting learning. However, we currently know little about the specific learning benefits of tangible environments, and have no established framework within which to analyse the ways that external representations work in tangible environments to support learning. Taking external representation as the central focus, this paper proposes a framework for investigating the effect of tangible technologies on interaction and cognition. Key artefact-action-representation relationships are identified, and classified to form a structure for investigating the differential cognitive effects of these features. An example scenario from our current research is presented to illustrate how the framework can be used as a method for investigating the effectiveness of differential designs for supporting science learning"
"(pdf). According to this vision, "the goal of Tangible Bits is to bridge the gaps between both cyberspace and the physical environment, as well as the foreground and background of human activities." This article isis a must-read for anyone interested in "new" interactive technologies.
The pictures in the article of the metaDesk, transBoard, activeLENS, and ambientRoom, along with the references, are worth taking at least a glance of this seminal work.
Another must-read is Hiroshi Ishii's 2008 article, Tangible Bits: Beyond Pixels(pdf). In this article,Ishii provides a good overview of TUI concepts as well as the contributions of his lab to the field since the first paper was written.
Related tothe Tangible User Interface research is the work of theFluid Interfaces Group at MIT. The Fluid Interfaces Group was formerly known as the Ambient Intelligence Group, and many of the group's projects incorporate concepts related to TUI and ambient intelligence. According to the Fluid Interfaces website, the goal of thisresearch group is to "radically rethink the human-machine interactive experience. By designing interfaces that are more immersive, more intelligent, and more interactive we are changing the human-machine relationship and creating systems that are more responsive to people's needs and actions, and that become true "accessories" for expanding our minds."
The Siftables project is an example of how TUI and fluid interface (FI) interaction can be combined.
Siftables is the work of David Merrill and Pattie Maes, in collaboration with Jeevan Kalanithi, and was brought to popular attention through David Merrill's recent TED talk: David Merrill's TED Talk: Siftables - Making the digital physical -Grasp Information Physically "Siftables aims to enable people to interact with information and media in physical, natural ways that approach interactions with physical objects in our everyday lives. As an interaction platform, Siftables applies technology and methodology from wireless sensor networks to tangible user interfaces. Siftables are independent, compact devices with sensing, graphical display, and wireless communication capabilities. They can be physically manipulated as a group to interact with digital information and media. Siftables can be used to implement any number of gestural interaction languages and HCI applications....Siftables can sense their neighbors, allowing applications to utilize topological arrangement..No special sensing surface or cameras are needed."
"Imagine if all the little programs you had on your iphone were little separate chicklets in your pocket. You'd lose em. Your cat would eat em. You'd vacuum them up. They'd fall down in the sofa. They'd be all over the car floor. You'd throw them away by mistake..."
In my opinion, it is exciting to learn that perhaps some of this technology has the potential of becoming main-stream.
YDreams is doing some interesting things. Watch the delight on this little girl's face as she plays with an avatar in mixed reality, viewed on a large display:
More from YDreams: "...Flapi, YDreams' in-house mascot, and other virtual characters interact in real-time with a little girl and other physical obstacles in a new seamless augmented playground environment."
Photo from YDreams Lab "YLabs’ main focus is on Reality Computing, which uses new technologies such as mobile computing, augmented reality and ubiquitous interactivity to bridge the distance between the user, information and the machine, in a physical, post-browser environment, where the real and the digital come together."
This is a photo of YDream's Architek's yMagic Books. Architek is used to create interactive digital content, including children's storybooks that are manipulated on a touch-screen.
This is a demonstration of Architek's yWalk, an immersive virtual playground that can be vertically projected onto soft mats and floors.
The Architek software provides information about user interaction. yWalk looks like it might be useful for occupational or physical therapists in their work with young children.
Rich White is an educational technologist for Greenbush County, Kansas, has been working with the 3D interactive virtual world, EduSim, for quite a while. He's taking EduSim to the next level.
"The concept is one of an enclosed virtual learning space - with surrounded projection of the virtual learning world the students are exploring - similar to the StarLab Concept (with a rectangular configuration). along the lines of a CAVE - however simpler, mobile, and relatively in-expensive by comparison." The project is at the beginning prototype stage.
Below is a demo of the virtual world as it is projected on two screens that are placed next to each other at a right angle, with the center of the virtual-world view positioned where the two screens meet:
This might be a great way of reaching students who have autism!
More about EduSim:
EduSim, for those of you haven't seen my previous posts on the topic, is a multi-user 3D interactive environment used in classrooms with interactive whiteboards:
Wikipedia entry: "Edusim is a Cave Automatic Virtual Environment based concept of lesson driven 3D virtual worlds on the classroom interactive whiteboard or classroom interactive surface. The Edusim concept is demonstrated by the Edusim free and open source multi-user 3DOpen Cobalt virtual world platform and authoring tool kit modified for the classroom interactive whiteboard or surface. The Edusim application is a modified edition of the open source Open Cobalt Project and relies heavily on the affordances of direct manipulation of 3D virtual learning models and Constructionist Learning Principles." History of Edusim: "The Edusim project began in September 2007 at the Greenbush Education Service Center in Southeast Kansas as an effort to bring an engaging 3D experience to the classroom interactive whiteboard. Pilot groups were established with 6th and 7th grade middle school students throughout Southeast Kansas to observe how students would be engaged through the software, and how the user interface would need to be augmented to account for the affordances of the whiteboard, and the usability of the students."
Here is a virtual world in Edusim in COBALT, showing how a drag and drop function is used for in-world VNC application sharing:
The Cobalt 3D metaverse browser has been modified for multi-touch interaction by some of the members of RENCI, a collaborative venture of Duke University and several other North Carolina universities. The video below is Dr. Xunlei Wu, demonstrating how gesture and touch is used to manipulate items and navigate through two Cobalt virtual worlds:
Some of the members of RENCI built a multi-touch table in addition to the collaborative multi-touch wall. For more information:
Jonathan Jarvis created a series of animated shorts as a project for his work as a graduate student in the Media Design Program at Art Center College of Design. He started exploring the concept of system diagrams and integrated them into motion interactions.
Jonathan worked on concept development, interface design & content development with a team for a multi-touch project, Interactive Oracles, for Acura: (Some of the information is cross-posted on the Economic Sounds and Sights blog.)
The concept of multi-touch/gesture/surface computing is spreading.
Here's more evidence:
Panasonic Touch Air Hockey
The game was demonstrated at ISE 2009 (Integrated Systems) Amsterdam. The interface was developed by UI Centric, a Soho, London company.
Microsoft's SurfaceWare at the Tangible Embedded Interactions Conference (TEI 2009): SurfaceWare is a level-sensing software that alerts waitstaff when glasses need refilling. Photos from Nachiket Apte, via Ru Zarin
