One of my interests is how the power and potential post-WIMP interactive technologies can be harnessed for use for formal and informal education purposes, including life-long collaborative learning.
In May, I had a chance to meet with a number of like-minded people during the CHI 2011 conference at the 2nd Workshop on UI Technologies and Impact on Educational Pedagogy. I was impressed with the depth and breadth of the presentations at the workshop. Since then, I've been looking through other papers and videos from CHI 2011 to find interesting applications that hold potential for use in educational settings.
I've come across a good number of interesting applications and prototypes, so be sure to check back for future posts on this topic. For now, here are a few applications that I'd like to share.
Below are a few videos from Phillip Chi-Wing Fu. (He doesn't know it yet, but I've admitted his videos into the Post-WIMP Explorers' Club.)
Interactive Multi-touch Sketching Interface for Diffusion Curves
"A novel multi-touch sketching interface enabling interactive and practical design with 2D diffusion curves is proposed; featured interaction techniques include simultaneous sketching of multiple diffusion curves and at-the-spot colors tuning."
Distinguishing Multiple Smart-Phone Interactions on a Multi-touch Wall Display using Tilt Correlation
"This paper proposes a novel matching technique, called tilt correlation, which employs the built-in tilt sensor on smart-phones to identify their concurrent contacts on a common multi-touch wall display."
WYSIWYF: Exploring and Annotating Volume Data with a Tangible Handheld Device (CHI 2011)
"Integration of a multi-touch wall display with a tangible handheld device with multi-touch and tilt sensing capabilities to provide intuitive what-you-see-is-what-you-feel visual exploration and annotation of volume data."
The following videos were uploaded by alucero:
Pass-Them-Around: Collaborative Use of Mobile Phones for Photo Sharing (CHI 2011)
"Pass-Them-Around is a phone-based application that allows a small group of collocated people to share photos using the metaphor of passing paper photos around. The prototype encourages people to share their devices and use them interchangeably while discussing photos face-to-face. The prototype supports ad-hoc photo sharing in different contexts by taking into account the spatial arrangement of users around a table, measured with sensors embedded in their mobile phones."
The next video was part of MobileHCI '10:
MindMap: Collaborative Use of Mobile Phones for Brainstorming
Showing posts sorted by relevance for query post-WIMP. Sort by date Show all posts
Showing posts sorted by relevance for query post-WIMP. Sort by date Show all posts
Jul 6, 2011
Revisiting CHI 2011: Videos of Interactive Touch, Gesture, Large Surface, and Mobile Apps with Potential for Use in Education (CHI = Computer Human Interaction)
Labels:
#CHI2011,
CHI 2011,
education,
gesture,
HCI,
interaction,
learning,
mobile,
multitouch,
NUI,
pedagogy,
post-WIMP,
teaching,
UI technology
No comments:
No comments:
Feb 13, 2011
Wii Just Dance2 and Kinect Dance Central: UI and Usability Approaches; Challenges for Developing Accessible Games (revised)
I love to dance- I studied dance through college, and off and on as an adult. I have a DDR (Dance Dance Revolution) game-floor pad somewhere in my attic gathering dust. I'm ready for new challenges.
I'm planning on buying a couple new dance games for the Wii and the Kinect. There is more to this story, given my interest off-the-desktop, post-WIMP HCI (human-computer interaction), interactive multimedia and games, and a career as a school psychologist dedicated to young people with disabilities, I'm excited to see where new technologies, interfaces, and interactions will take us.
So what do the wise men of usability have to say about new ways of interacting with games and other applications?
"Kinect has many great design elements that clearly show that the team (a) knows usability, (b) did user testing, and (c) had management support to prioritize usability improvements, even when they required extra development work." -Jakob Nielsen
Jakob Nielsen, one of the godfathers of usability, shared a few words of wisdom about the Kinect in his 12/27/10 Alertbox post: Kinect Gestural UI: First Impressions. Although he did not review Dance Central, he concludes that the game he reviewed, Kinect Adventures, was fun to play, despite usability problems.
If this is a topic that interests you, I recommend you read Neilsen's post, and also take a look at which are outlined in the post. Also take a look at recent essay Neilsen co-authored with Don Norman, another godfather of usability: Gestural Interfaces: A Step Backwards In Usability
The screen shots below show how the movement icons are used in the game:
I thought it would be useful to learn more about the story behind the making of JustDance2.
At 2:22, Alexia, the project's usability expert, makes her presence known. From what I can tell, she focused on aspects of the game that would make it more usable for non-dancers, including those with "two left feet", to play the game. (I don't know if there was anyone consulted about accessibility concerns for the game.)
Kinect Dance Central
Dance Central uses a different approach when it comes to "teaching" people how to dance along through the game. It would be interesting to test out Dance Central and JustDance 2 with the same set of people to get a better feel for what works and what doesn't. Below is a video that previews, in split-screen, the interaction that takes place in Dance Central:
Dance Central Full Motion Preview
In Dance Central, gamers are provided with information about the moves through icons that cycle up the right hand side of the screen. The level of dance-coordination to keep up with the moves is challenging at times, even for people who are OK at dancing. Players can select dances according to level of difficulty.
Kinect Usability with Regular People
Below are screen shots that provide examples of how the movement icons are displayed in Dance Central:
RELATED
Just Dance 2 Review: Get your body moving. No, really. Give it a shot.
I'm planning on buying a couple new dance games for the Wii and the Kinect. There is more to this story, given my interest off-the-desktop, post-WIMP HCI (human-computer interaction), interactive multimedia and games, and a career as a school psychologist dedicated to young people with disabilities, I'm excited to see where new technologies, interfaces, and interactions will take us.
So what do the wise men of usability have to say about new ways of interacting with games and other applications?
"Kinect has many great design elements that clearly show that the team (a) knows usability, (b) did user testing, and (c) had management support to prioritize usability improvements, even when they required extra development work." -Jakob Nielsen
Jakob Nielsen, one of the godfathers of usability, shared a few words of wisdom about the Kinect in his 12/27/10 Alertbox post: Kinect Gestural UI: First Impressions. Although he did not review Dance Central, he concludes that the game he reviewed, Kinect Adventures, was fun to play, despite usability problems.
If this is a topic that interests you, I recommend you read Neilsen's post, and also take a look at which are outlined in the post. Also take a look at recent essay Neilsen co-authored with Don Norman, another godfather of usability: Gestural Interfaces: A Step Backwards In Usability
Why is this topic important to me?
I have been involved in the Games for Health and Game Accessibility movement for many years. Lately I've been exploring the OpenKinect project with an aim to create ways of making movement-oriented games accessible for young people with more complex disabilities. For example, there is a need to have dance and movement games modified for students (and adults!) who need wheelchairs or walkers. There are students who have milder mobility challenges who love to dance, and the current games don't address their needs. Some of my students have vision or hearing impairments, too. They deserve a chance to play things designed for the Kinect.
"OpenKinect is an open community of people interested in making use of the amazing Xbox Kinect hardware with our PCs and other devices. We are working on free, open source libraries that will enable the Kinect to be used with Windows, Linux, and Mac."
Note: I currently work as a school psychologist with students up to age 22. My main office is adjacent to a large OT and PT room at Wolfe, a program for students who have special needs. We just had a large interactive whiteboard installed in the room that is begging for us to connect it with the school's Wii, and soon (we hope), a Kinect. If we are going to use dance games to help promote healthy activities among our special students, the games need to be accessible for students with cognitive, motor, and other limitations.
FIRST STEPS
Although I can dance, I understand what the world is like through the eyes of many of the young people I work with who have motor coordination and sensory integration problems that interfere with their ability to move and dance, let alone access fast-paced dance games on the Wii or Kinect.
My initial plan is to look at what the new dance games might be like from the view of someone who doesn't know how to dance, and admits that they have "two left feet" - an perhaps, no sense of rhythm. Where would I start?
Wii's Just Dance2 seems to offer some support for learning how to dance through the use of simple movement icons, in the form of outlined figures, that provide information about how to move with the dancer on the screen. As you can see from the video below, the gamer is provided with information about upcoming moves throughout the game.
I decided to take a look at Just Dance2's MIKA "Big Girl" (You Are Beautiful) because some of the adolescent females I work with have weight concerns that interfere with their health. During the teen years, this can become a vicious cycle, resulting in less movement, and less participation with peers in physical activities, such as playing dance games. If a teen has depression as part of this mix, we know that exercise can help, and a fun dance game might be a life-saver, in more ways than one.
Although I can dance, I understand what the world is like through the eyes of many of the young people I work with who have motor coordination and sensory integration problems that interfere with their ability to move and dance, let alone access fast-paced dance games on the Wii or Kinect.
My initial plan is to look at what the new dance games might be like from the view of someone who doesn't know how to dance, and admits that they have "two left feet" - an perhaps, no sense of rhythm. Where would I start?
Wii's Just Dance2 seems to offer some support for learning how to dance through the use of simple movement icons, in the form of outlined figures, that provide information about how to move with the dancer on the screen. As you can see from the video below, the gamer is provided with information about upcoming moves throughout the game.
I decided to take a look at Just Dance2's MIKA "Big Girl" (You Are Beautiful) because some of the adolescent females I work with have weight concerns that interfere with their health. During the teen years, this can become a vicious cycle, resulting in less movement, and less participation with peers in physical activities, such as playing dance games. If a teen has depression as part of this mix, we know that exercise can help, and a fun dance game might be a life-saver, in more ways than one.
The screen shots below show how the movement icons are used in the game:
I thought it would be useful to learn more about the story behind the making of JustDance2.
At 2:22, Alexia, the project's usability expert, makes her presence known. From what I can tell, she focused on aspects of the game that would make it more usable for non-dancers, including those with "two left feet", to play the game. (I don't know if there was anyone consulted about accessibility concerns for the game.)
Kinect Dance Central
Dance Central uses a different approach when it comes to "teaching" people how to dance along through the game. It would be interesting to test out Dance Central and JustDance 2 with the same set of people to get a better feel for what works and what doesn't. Below is a video that previews, in split-screen, the interaction that takes place in Dance Central:
Dance Central Full Motion Preview
In Dance Central, gamers are provided with information about the moves through icons that cycle up the right hand side of the screen. The level of dance-coordination to keep up with the moves is challenging at times, even for people who are OK at dancing. Players can select dances according to level of difficulty.
Kinect Usability with Regular People
Steve Cable (CX Partmers) shared his team's look at usability issues related to the Kinect by testing several games, including Dance Central, with groups of people in his article, "Designing for XBox Kinect - a usability study". The quote below is from the Steve's article:
"We’ve loved playing with the Kinect. There’s no doubt that the game play is lots of fun. In-game menus are a barrier to that fun. Kinect should allow players to move through menus quickly and compensate for inaccuracy.
We felt the Kinect would benefit from some standardised global controls – much like a controller uses the A button to select and the B button to move backwards. We also think it needs a more responsive pause gesture – one that doesn’t interfere with the user’s game play.
Most of our participants found the Dance Central menu to be more effective, more efficient and more satisfying to use. Here are our recommendations for designing a Kinect menu interface:
- Allow users to make selections through positive gestures, rather than timed positions
- Place options on a single axis to make them easier and quicker to select
- Allow users to control menus with the game pad if they prefer
- Use large easy to read text
- Don’t make users scroll through options unnecessarily – it takes too long
- Users will be distracted if used in a social setting – test your menus in a social context to see if they are prone to errors
- Avoid the cursor metaphor, it’s not what gamers are used to seeing in game menus, and makes it harder to implement alternative joypad controls"
Below are screen shots that provide examples of how the movement icons are displayed in Dance Central:
RELATED
Just Dance 2 Review: Get your body moving. No, really. Give it a shot.
Kexa MacDonald, 10/19/10
Labels:
accessibility,
Dance Central,
Games for Health,
Just Dance2,
kinect,
NUI design,
openkinect,
quora,
review,
usability,
video,
Wii
No comments:
No comments:
Jun 26, 2010
A few links: GizmoWatch's 10 Interactive User Interfaces for the Future, CNN's Eatocracy, EVA 2010 and More!
Here's a quick link to a recent Gizmowatch post, Ten Interactive User Interfaces for the Future. Bharat, the authro, reviews a variety of interfaces, input methods, and interaction techniques, such as Skinput, a water-based touch screen, a muscle-computer interface, air gestures, brain-computer systems, and even a mud-tub interface.
I was fortunate to see some of these interaction techniques and interfaces when I attended CHI 2010 this past April, and plan to share some of my photos and video clips from the conference on this blog soon.
Totally Unrelated
Online connection for foodies
Eatocracy is a new website within the CNN pages that provides news- and more- about all things related to food. The categories on the site include "main", "news", "bite", "sip", "make", "think", and "buzz". The best part, in my opinion, is the heirloom recipe collection index, where people can upload and share family recipies and the stories behind them.
Here is the description of Eatocracy from the website:
Eatocracy "is your online home for smart, passionate conversation and information about food news, politics, culture. We'll highlight regional and family recipes, dive into restaurants and food shopping, chat with celebrity and local chefs, and show you what's for dinner around the world tonight. Grab a place at the table and read with your mouth full."
Enjoy!
(The above is a repost from The World is My Interactive Interface)
I was fortunate to see some of these interaction techniques and interfaces when I attended CHI 2010 this past April, and plan to share some of my photos and video clips from the conference on this blog soon.
Totally Unrelated
Online connection for foodies
Eatocracy is a new website within the CNN pages that provides news- and more- about all things related to food. The categories on the site include "main", "news", "bite", "sip", "make", "think", and "buzz". The best part, in my opinion, is the heirloom recipe collection index, where people can upload and share family recipies and the stories behind them.
Here is the description of Eatocracy from the website:
Eatocracy "is your online home for smart, passionate conversation and information about food news, politics, culture. We'll highlight regional and family recipes, dive into restaurants and food shopping, chat with celebrity and local chefs, and show you what's for dinner around the world tonight. Grab a place at the table and read with your mouth full."
Enjoy!
(The above is a repost from The World is My Interactive Interface)
Coming Soon
--More about 3D TV and Interactive TV
--Highlights from CHI 2010 (better late than never!)
--My experiments- SMARTTable, a game, interactive timeline prototype pictures...
--A post about Lieven van Velthoven's interesting Post-WIMP explorations - here are some links that he recently sent me:
- 3D birthday invitation: http://www.youtube.com/watch?
v=pNOP5aIoS9o - Virtual overgrowth in the real world: http://www.youtube.com/watch?
v=nuJPoUpBIKU - Make race tracks for virtual cars with any real stuff you can find: http://www.youtube.com/watch?
v=e1K3og8qLcA - Smoke Pong: Pong, ONLY for smokers: http://www.youtube.com/watch?
v=zwrsVo5p2_E - Webcam Airmouse: http://www.youtube.com/watch?
v=57kTrKJHOjM - MusicalNodes: the visual music library: www.musicalnodes.com, http://www.youtube.com/watch?
v=czQdhfA__UY (will be published at EVA2010)
As I took a peek at Lieven's video links, I noticed an interesting video mash-up Lieven created from the open-source code from the RadioHead's House of Cards music video and his One Million Particles app. I'll post them soon.
I'll try to get video, pictures, and commentary about EVA 2010. EVA stands for Electronic Visualization and the Arts. "Electronic Information, the Visual Arts, and Beyond.
FYI
I'm in the process of sorting through and re-organizing my blogs, which have been around for over four years! During this time, my blogs have attracted a growing number of readers. Because of this, I'd like to make things a bit user-centered. So expect to see little changes here and there. I promise I'll give my readers warnings in advance if I make any serious changes!
If you are new to this blog, you should know that my blogs started out as on-line filing cabinets, open to the world. Although there is a bit of overlap of material and some cross-posting between the blogs, they are arranged to serve as a paper-less way of keeping track of things that I've learned through my coursework, conference attendance, readings, and research. Since emerging technologies are high on my list of interests, I also use my blogs to share interesting things that cross my path.
I changed the name of my World Is My Interface blog to The World Is My Interactive Interface. "Off-the-desktop natural user interfaces, interaction, and user experience" are the main topics of the blog. It sometimes includes information about ubiquitous computing and DOOH, otherwise known as Digital Out Of Home.
I plan to tinker with my TechPsych blog later on. It focuses on topics that are useful to psychologists, educators, special education teachers, speech and language therapists, health and wellness professionals, and parents.
Feel free to leave comments, as I welcome your input.
Dec 31, 2009
Josh Blake's' Nice Multi-touch and Natural User Interface Applications for Surface (Cross-Post )
Information from Josh's YouTube channel:
"This is a video of some of the cool multi-touch and Natural User Interface (NUI) applications I designed and developed for Surface and Windows 7." The InfoStrat.VE map control for WPF and Surface is available for free at http://virtualearthwpf.codeplex.com.
I especially like the moving ring-menu concept, as it facilitates smoother collaboration between people on an interactive table or surface, where flexible orientation control is important.
At 3:15, the demonstration of Josh's ink-shape recognition begins. This is a feature that would be great to incorporate in my applications for children with disabilities who have some fine-motor limitations.
Josh's Blog: Deconstructing the NUI
Josh's Recent Post about post-WIMP concepts:
Metaphors and OCGM
Josh works at InfoStrat
Feb 27, 2009
Tangible User Interfaces Part II: More Examples, Resources, and Use for TUI's in Education
In Part I of my "mini-series" about Tangible User Interfaces, I discussed the origins of TUI and provided some examples of Siftables. In this section, I've provided some links to information about Tangible User Interfaces and the abstracts of two articles pertaining to TUI's in educational settings.
Zen Waves: A Digital (musical) Zen Garden
reactable from Nick M. on Vimeo.
Reactable
More about the Reactable
"The reactable hardware is based on a translucent, round multi-touch surface. A camera situated beneath the table, continuously analyzes the surface, tracking the player's finger tips and the nature, position and orientation of physical objects that are distributed on its surface. These objects represent the components of a classic modular synthesizer, the players interact by moving these objects, changing their distance, orientation and the relation to each other. These actions directly control the topological structure and parameters of the sound synthesizer. A projector, also from underneath the table, draws dynamic animations on its surface, providing a visual feedback of the state, the activity and the main characteristics of the sounds produced by the audio synthesizer."
The Bubblegum Sequencer: Making Music with Candy
Jabberstamp: Embedding Sound and Voice in Children's Drawings(pdf)
(A TUI application to support literacy development in children)
Affective TouchCasting (pdf)
TapTap: A Haptic Wearable for Asynchronous Distributed Touch Therapy (pdf)
BodyBeats: Whole-Body, Musical Interfaces for Children (pdf)
Telestory is a Siftables application that looks like it would be quite useful for supporting children who have communication disorders or autism spectrum disorders.
Telestory Siftables application from Jeevan Kalanithi on Vimeo.
"Telestory is an educational, language learning application created by Seth Hunter. In this video, the child is looking at a television screen. He can control onscreen characters, events and objects with the siftables. For example, he has the dog and cat interact by placing the dog and cat siftables next to each other." TeleStory Project Website
Here is a video of how Siftables can be used as equation editors:
Siftables Equation Editor from Jeevan Kalanithi on Vimeo.
RESOURCES ABOUT TUI'S:
5 lessons about tangible interfaces, GDC Lyon, December 2007(pdf) Nicolas Nova
Special Issue on Tangible and Embedded Interaction (Guest Editors: Eva Hornecker, Albrecht Schmidt, Brygg Ullmer) Journal of Arts and Technology (IJART) Volume 1 Issue 3/4 - 2008
Reality-Based Interaction: A Framework for Post-WIMP Interfaces (pdf)
Here are a couple of abstracts of articles related to the use of TUI's in education:
Evaluation of the Efficacy of Computer-Based Training Using Tangible User Interface for Low-Functioning Children with Autism Proceedings of the 2008 IEEE International Conference on Digital Games and Intelligent Toys
"Recently, the number of children having autism disorder increases rapidly all over the world. Computer-based training (CBT) has been applied to autism spectrum disorder treatment. Most CBT applications are based on the standard WIMP interface. However, recent study suggests that a Tangible User Interface (TUI) is easier to use for children with autism than the WIMP interface. In this paper, the efficiency of the TUI training system is considered, in comparison with a conventional method of training basic geometric shape classification. A CBT system with TUI was developed using standard computer equipment and a consumer video camera. The experiment was conducted to measure learning efficacy of the new system and the conventional training method. The results show that, under the same time constraint, children with autism who practiced with the new system were able to learn more shapes than those participating in the conventional method."
Towards a framework for investigating tangible environments for learning Sara Price, Jennifer G. Sheridan, Taciana Pontual Falcao, George Roussos, London Knowledge Lab, 2008
"External representations have been shown to play a key role in mediating cognition. Tangible environments offer the opportunity for novel representational formats and combinations, potentially increasing representational power for supporting learning. However, we currently know little about the specific learning benefits of tangible environments, and have no established framework within which to analyse the ways that external representations work in tangible environments to support learning. Taking external representation as the central focus, this paper proposes a framework for investigating the effect of tangible technologies on interaction and cognition. Key artefact-action-representation relationships are identified, and classified to form a structure for investigating the differential cognitive effects of these features. An example scenario from our current research is presented to illustrate how the framework can be used as a method for investigating the effectiveness of differential designs for supporting science learning"
Zen Waves: A Digital (musical) Zen Garden
reactable from Nick M. on Vimeo.
Reactable
More about the Reactable
"The reactable hardware is based on a translucent, round multi-touch surface. A camera situated beneath the table, continuously analyzes the surface, tracking the player's finger tips and the nature, position and orientation of physical objects that are distributed on its surface. These objects represent the components of a classic modular synthesizer, the players interact by moving these objects, changing their distance, orientation and the relation to each other. These actions directly control the topological structure and parameters of the sound synthesizer. A projector, also from underneath the table, draws dynamic animations on its surface, providing a visual feedback of the state, the activity and the main characteristics of the sounds produced by the audio synthesizer."
The Bubblegum Sequencer: Making Music with Candy
Jabberstamp: Embedding Sound and Voice in Children's Drawings(pdf)
(A TUI application to support literacy development in children)
Affective TouchCasting (pdf)
TapTap: A Haptic Wearable for Asynchronous Distributed Touch Therapy (pdf)
BodyBeats: Whole-Body, Musical Interfaces for Children (pdf)
Telestory is a Siftables application that looks like it would be quite useful for supporting children who have communication disorders or autism spectrum disorders.
Telestory Siftables application from Jeevan Kalanithi on Vimeo.
"Telestory is an educational, language learning application created by Seth Hunter. In this video, the child is looking at a television screen. He can control onscreen characters, events and objects with the siftables. For example, he has the dog and cat interact by placing the dog and cat siftables next to each other." TeleStory Project Website
Here is a video of how Siftables can be used as equation editors:
Siftables Equation Editor from Jeevan Kalanithi on Vimeo.
RESOURCES ABOUT TUI'S:
5 lessons about tangible interfaces, GDC Lyon, December 2007(pdf) Nicolas Nova
Special Issue on Tangible and Embedded Interaction (Guest Editors: Eva Hornecker, Albrecht Schmidt, Brygg Ullmer) Journal of Arts and Technology (IJART) Volume 1 Issue 3/4 - 2008
Reality-Based Interaction: A Framework for Post-WIMP Interfaces (pdf)
Here are a couple of abstracts of articles related to the use of TUI's in education:
Evaluation of the Efficacy of Computer-Based Training Using Tangible User Interface for Low-Functioning Children with Autism Proceedings of the 2008 IEEE International Conference on Digital Games and Intelligent Toys
"Recently, the number of children having autism disorder increases rapidly all over the world. Computer-based training (CBT) has been applied to autism spectrum disorder treatment. Most CBT applications are based on the standard WIMP interface. However, recent study suggests that a Tangible User Interface (TUI) is easier to use for children with autism than the WIMP interface. In this paper, the efficiency of the TUI training system is considered, in comparison with a conventional method of training basic geometric shape classification. A CBT system with TUI was developed using standard computer equipment and a consumer video camera. The experiment was conducted to measure learning efficacy of the new system and the conventional training method. The results show that, under the same time constraint, children with autism who practiced with the new system were able to learn more shapes than those participating in the conventional method."
Towards a framework for investigating tangible environments for learning Sara Price, Jennifer G. Sheridan, Taciana Pontual Falcao, George Roussos, London Knowledge Lab, 2008
"External representations have been shown to play a key role in mediating cognition. Tangible environments offer the opportunity for novel representational formats and combinations, potentially increasing representational power for supporting learning. However, we currently know little about the specific learning benefits of tangible environments, and have no established framework within which to analyse the ways that external representations work in tangible environments to support learning. Taking external representation as the central focus, this paper proposes a framework for investigating the effect of tangible technologies on interaction and cognition. Key artefact-action-representation relationships are identified, and classified to form a structure for investigating the differential cognitive effects of these features. An example scenario from our current research is presented to illustrate how the framework can be used as a method for investigating the effectiveness of differential designs for supporting science learning"
May 29, 2010
Preview: Update on Touch & Multitouch Technologies, Websites, and Touch-Interactive Multimedia Apps
It is about time for an update about touch/gesture- interactive technologies.
I've been researching the latest in "touch" screens and new developments in interactive multi-media content. In just one year, a multitude of websites have been transformed from static to interactive.
Although the initial objective for some of these websites was to optimize the interface and navigation for people accessing websites via touch-screen cell phones, some are ideal for use on touch-enabled slates, the iPad, and even larger touch screen displays and surfaces.
Convergence seems to be the buzz word of the day. Interactive TV. Game sets with Internet access. Movies on your cell phone. Touch screen Coke machines displaying movie trailers. What's happening now, and what is next?
I welcome input from my readers in the form of links to websites, university labs with grad students and professors who are obsessed with emerging interactive technologies, proof-of-concept video clips, video clips of related technologies that are new-to-market, etc.
I will add video clips to the following playlist:
FYI: I'm also in the middle of writing a series of posts about 3D television technologies for the Innovative Interactivity blog, and welcome input from my readers about this topic.
RELATED (Previous posts)
(the above post includes links to various multi-touch developer kits and resources)
Multi-touch Linux on a Stantum Slate PC & More (links to a nice overview about multi-touch interaction from ENAC)
Multimedia, Multi-touch, Gesture, and Interaction Resources (needs a little updating)
Labels:
display,
interactive,
interactive website,
multi-touch,
post-WIMP,
research,
resources,
technology,
touch,
update
No comments:
No comments:
Jan 1, 2010
Apple iSlate, iTablet , MacBook Touch: Will it support gesture interaction & haptic feedback?
Soldier Knows Best produces great tech-oriented videos. Here's his spin on all of the rumors about the possibility of the Apple iSlate.
I just inherited a 10 month-old Mac Book, installed Snow Leopard and upgraded to iLife 2009. I'm so used to touching the screen on my HP TouchSmart PC that I found myself touching my Mac Book screen from time to time, especially when I was editing video clips in iMovie. I think the latest version of iMovie was designed with touch/gesture interaction in mind!
From what I can tell, Snow Leopard and iLife 2009 will be able to support a range of touch interactions, if not gesture input as well.
Here are some rumors that have been conjured up and distributed on the web:
The Exhaustive Guide to Applet Tablet Rumors (Matt Buchanan, Gizmodo, 12/26/09)
Apple Expects to Sell 10 Million Tablets in First Year (Pete Cashmore, Mashable, 1/1/10)
iGuide Emerges as Another Potential Apple Tablet Name (Adam Ostrow, Mashable, 12/29/09)
The Tablet (John Gruber, Daring Fireball, 12/31/09)
"And so in answer to my central question, regarding why buy The Tablet if you already have an iPhone and a MacBook, my best guess is that ultimately, The Tablet is something you’ll buy instead of a MacBook."
Apple Owns iSlate.com Domain: The Mystery Deepens (Dan Nosowitz, Gismodo, 12/25/09)
What is the Ultimate Role of the Apple Tablet? (Arnold Kim, MacRumors, 12/31/09)
iPad, iTablet, iSlate, or MacTab (Cruz Miranda, 8/31/09)
Why am I excited about this?
I want to see if the iSlate would be good for collaborative educational games, assisted technology, augmentative communication, and alternative assessment for students who have multiple/severe disabilities.
That is a huge goal, so I'm going to start simple. I am not giving up on Windows 7 multi-touch programming. I just have an urge to find out for myself what works, what doesn't, and what platform works best for specific "personas" and "scenarios".
I plan to make a little app for the iPhone/iPod Touch, based on a game I made several years ago, "Shoes Your Battles" for a game class. I think I'd like to make this game for the Apple iTablet!
The first version of Shoes Your Battles created with Game Maker, and the second version was in Flash, back in the days of ActionScript 2.0. I started on third version, one that could be used as an advergame for people to play while shopping for shoes during shoe sales, but it never got past the planning stage.
The idea for the third version came to me when I my elderly aunt came to visit from out-of-town and just had to go shoe shopping on the day after Thanksgiving. It was extremely difficult to figure out what was on sale, how much it cost, after taking off the previous mark-downs and what was on sale that had a price that was not yet marked down.
Adding to the confusion was the fact that there were few salespeople and herds of women. It was madness. There were pairs of shoes in the wrong boxes, boxes of shoes and no way to quickly find out the true prices! We were in the shoe department for hours, and it wasn't as fun as you'd think. If you've been in a crowded women's shoe department to buy that special pair of shoes during a fantastic shoe sale, you'll know what I mean.
At any rate, I wanted my little "Shoes Your Battles" game to help with this dreadful scenario, by somehow incorporating a shoe shopping advisor and a means to figure out the REAL sales prices of those awesome, to-die-for shoes. Unfortunately, the technology wasn't where it needed to be at the time- I am always dreaming up things that are too d--- futuristic!
4 years later, we have iPhones and SmartPhones and 3G internet and RFID and ubiquitous WiFi and the Wii and more women who like to play games and...and... The time is ripe.
Apple better come up with the iSlate!
SOMEWHAT RELATED
Thinking about post-WIMP HCI
It is always important to re-visit wisdom from the past when thinking about new interfaces and means of technology-supported human interaction. Here are a few resources from the field of Human-Computer Interaction found on the HCI Vistas website:
The Prism of User Experience -A nice graphic metaphor to help the conceptualization process. (Denish Katre, 2007)
Journal of HCI Vistas: Multi-disciplinary Perspective of Usability and HCI
Personas as part of a user-centered innovation process Lene Nielsen, 1/08 HCI Vistas Vol-IV
10 Steps to Personas (Lene Nielsen, 7/07, HCI Vistas Vol-III)
I just inherited a 10 month-old Mac Book, installed Snow Leopard and upgraded to iLife 2009. I'm so used to touching the screen on my HP TouchSmart PC that I found myself touching my Mac Book screen from time to time, especially when I was editing video clips in iMovie. I think the latest version of iMovie was designed with touch/gesture interaction in mind!
From what I can tell, Snow Leopard and iLife 2009 will be able to support a range of touch interactions, if not gesture input as well.
Here are some rumors that have been conjured up and distributed on the web:
The Exhaustive Guide to Applet Tablet Rumors (Matt Buchanan, Gizmodo, 12/26/09)
Apple Expects to Sell 10 Million Tablets in First Year (Pete Cashmore, Mashable, 1/1/10)
iGuide Emerges as Another Potential Apple Tablet Name (Adam Ostrow, Mashable, 12/29/09)
The Tablet (John Gruber, Daring Fireball, 12/31/09)
"And so in answer to my central question, regarding why buy The Tablet if you already have an iPhone and a MacBook, my best guess is that ultimately, The Tablet is something you’ll buy instead of a MacBook."
Apple Owns iSlate.com Domain: The Mystery Deepens (Dan Nosowitz, Gismodo, 12/25/09)
What is the Ultimate Role of the Apple Tablet? (Arnold Kim, MacRumors, 12/31/09)
iPad, iTablet, iSlate, or MacTab (Cruz Miranda, 8/31/09)
Why am I excited about this?
I want to see if the iSlate would be good for collaborative educational games, assisted technology, augmentative communication, and alternative assessment for students who have multiple/severe disabilities.
That is a huge goal, so I'm going to start simple. I am not giving up on Windows 7 multi-touch programming. I just have an urge to find out for myself what works, what doesn't, and what platform works best for specific "personas" and "scenarios".
I plan to make a little app for the iPhone/iPod Touch, based on a game I made several years ago, "Shoes Your Battles" for a game class. I think I'd like to make this game for the Apple iTablet!
The first version of Shoes Your Battles created with Game Maker, and the second version was in Flash, back in the days of ActionScript 2.0. I started on third version, one that could be used as an advergame for people to play while shopping for shoes during shoe sales, but it never got past the planning stage.
The idea for the third version came to me when I my elderly aunt came to visit from out-of-town and just had to go shoe shopping on the day after Thanksgiving. It was extremely difficult to figure out what was on sale, how much it cost, after taking off the previous mark-downs and what was on sale that had a price that was not yet marked down.
Adding to the confusion was the fact that there were few salespeople and herds of women. It was madness. There were pairs of shoes in the wrong boxes, boxes of shoes and no way to quickly find out the true prices! We were in the shoe department for hours, and it wasn't as fun as you'd think. If you've been in a crowded women's shoe department to buy that special pair of shoes during a fantastic shoe sale, you'll know what I mean.
At any rate, I wanted my little "Shoes Your Battles" game to help with this dreadful scenario, by somehow incorporating a shoe shopping advisor and a means to figure out the REAL sales prices of those awesome, to-die-for shoes. Unfortunately, the technology wasn't where it needed to be at the time- I am always dreaming up things that are too d--- futuristic!
4 years later, we have iPhones and SmartPhones and 3G internet and RFID and ubiquitous WiFi and the Wii and more women who like to play games and...and... The time is ripe.
Apple better come up with the iSlate!
SOMEWHAT RELATED
Thinking about post-WIMP HCI
It is always important to re-visit wisdom from the past when thinking about new interfaces and means of technology-supported human interaction. Here are a few resources from the field of Human-Computer Interaction found on the HCI Vistas website:
The Prism of User Experience -A nice graphic metaphor to help the conceptualization process. (Denish Katre, 2007)
Journal of HCI Vistas: Multi-disciplinary Perspective of Usability and HCI
Personas as part of a user-centered innovation process Lene Nielsen, 1/08 HCI Vistas Vol-IV
10 Steps to Personas (Lene Nielsen, 7/07, HCI Vistas Vol-III)
Labels:
accessible games,
Apple,
apps,
creative programming,
design,
games,
gizmodo,
iGuide,
iTablet,
mac,
Macbook Touch,
multi-touch,
NUI,
post-WIMP,
product,
rumors,
Soldier Knows Best,
touch
No comments:
No comments:
Nov 21, 2009
Want to make some multi-touch? Try PyMT- Python Multitouch. Featured in Make. (via Sharath Patali)
Sharath Patali, a member of the NUI-Group, has been working with Python Multitouch, otherwise known as PyMT, to create multi-touch applications. He shared a link to a recent post in Make, featuring PyMT. Sharath is the author of the UI Addict blog, and is currently doing his internship at NUITEQ (Natural User Interface Technologies).
I've been told that the beauty of PyMT is that it makes it "easy" to create multi-touch prototype applications using very few lines of code, which is great for trying out different ideas in a short period of time. It helps if you already know Python!
PyMT - A post-WIMP Multi-Touch UI Toolkit from Thomas Hansen on Vimeo.
"PyMT is a python module for developing multi-touch enabled media rich applications. Currently the aim is to allow for quick and easy interaction design and rapid prototype development. PyMT is written in Python, based on pyglet toolkit."
PyMT Programming Guide
PyMT Website
Note:
Christopher, author of The Space Station blog, is a member of the NUI-Group, and is building his own multi-touch table running his PyMT-based applications. Christopher is a student in Koblenz, Germany, studying computational visualistics, known as information visualization in the US.
I've been told that the beauty of PyMT is that it makes it "easy" to create multi-touch prototype applications using very few lines of code, which is great for trying out different ideas in a short period of time. It helps if you already know Python!
PyMT - A post-WIMP Multi-Touch UI Toolkit from Thomas Hansen on Vimeo.
"PyMT is a python module for developing multi-touch enabled media rich applications. Currently the aim is to allow for quick and easy interaction design and rapid prototype development. PyMT is written in Python, based on pyglet toolkit."
PyMT Programming Guide
PyMT Website
Note:
Christopher, author of The Space Station blog, is a member of the NUI-Group, and is building his own multi-touch table running his PyMT-based applications. Christopher is a student in Koblenz, Germany, studying computational visualistics, known as information visualization in the US.
Jul 23, 2013
Monkeying Around with Autism Assessments: Kinect-based game by Vectorform and Kaiser Permanente therapists offers a barrel of possibilities!
"The goal was to build a game that is extremely accessible, non-complex and includes simple mechanics that children with autism can quickly understand to retain their attention, prevent over-stimulation, encourage play, and prevent frustration." -Vectorform "Monkey Business" Team
I recently had the chance to visit the team at Vectorform's Royal Oak (MI) headquarters and experience a demo of one of their newest applications, a Kinect-based game developed to assist in the assessment of children with autism. The game, known as Monkey Business, was designed in collaboration with physical, occupational, and speech therapists from Kaiser Permanente, a large health care service provider in California. The project was supported by Kaiser Permanente's Innovation Fund for Technology group.
I was impressed!
As I watched the demo, it was clear that much time and attention was given to the input of the therapists at Kaiser Permanente. The avatar in the game is a friendly monkey who embodies the mannerisms of a friendly, curious child. The artwork is visually appealing and not too busy. The sound effects relate to the interaction at hand and provide feedback to the child as they engage in various game-like tasks designed to assess visual-motor, gross motor, and basic receptive language skills. Another good feature of this application is the simplicity of the screens designed for input of the child's information and viewing data regarding progress.
In my opinion, the Monkey Business assessment game has the potential for use in school settings, and would be enhanced if additional assessment activities are provided in the application. Expanded activities within the Monkey Business environment could support a range of treatment, education, and related intervention efforts. In addition to supporting assessment and intervention of children with autism, the Monkey Business concept would be especially useful in working with children who have experienced a traumatic brain injury.
Below are screen shots and descriptions of a few activities from the Monkey Business game:
Lily Pad Jumping Test
In the jumping test, the child is asked to jump across a pond, from one lily pad to another. The child is provided with a set of toy lily pads on the floor. As the child jumps, the Kinect sensor picks up movement and displays the jumps on the screen in the form of the monkey.
Balance Beam Test
The following screen shot depicts the monkey avatar crossing a stream on a balance beam. The monkey's movements reflect the child's movements across a real balance beam in front of the Kinect sensor and a large screen display.
Block Building Test
In the block building test, the monkey avatar encourages the child to build a tower of blocks to match the one displayed on the large screen. The child uses real blocks, placed on a table, to build a tower while the monkey provides encouragements. Objects other than blocks can be used, as the Kinect sensor is capable of identifying a variety of objects that can be use for building and stacking.
Clinician Screens
The screens designed for inputting and reviewing data regarding the child are designed with simplicity and ease-of use in mind, as shown in the pictures below:
In the present version of Monkey Business, a TV remote is used to control the application. It is possible that future versions of Monkey Business could integrated with a Smartwatch to input student information and control the activities in the application.
Comment: As a school psychologist who works with children, teens, and young adults with autism and other significant disabilities, I know how difficult it can be to conduct assessments using traditional test materials. If a student has difficulty interacting with unfamiliar adults, has communication challenges, or has motor difficulties, it may not be possible to administer an assessment that generates meaningful or valid results. Even if the student is capable of completing some of the traditional test tasks, much time and effort is required on the part of the evaluator to sustain their interest, effort, and attention.
To get a better picture of a student's emerging skills, I use interactive multimedia applications during my assessments of children with autism. Newer technologies such as interactive whiteboards, larger touch-screen monitors, and tablets have proven to be useful tools in assessment. A significant drawback to my approach is that it is cobbled together and might be difficult to replicate by other evaluation teams. I see a growing need for a range of technologically-enhanced tools for assessment, including applications that offer opportunities for playful engagement. Monkey Business holds potential to fill the bill.
What next?
I'd like to share some stories about the people who make things come to life at Vectorform. During my visit, I learned that they are involved in a number of projects related to health care and are always brainstorming new ways to harness technology for their various clients.
Over the past several years, I've followed Vectorform's journey as they've jumped off of the desktop and transformed from web-developers to Post-WIMP explorers of natural user interfaces and interaction. The most recent area of exploration is Google Glass.
Intrigued by Google Glass? So am I.
During my visit to Vectorform, I had a chance to see the world - or rather Vectorform's basement conference room - through Google Glass. As soon as I put them on I was flooded with ideas, and will expand on my ideas, and those of others, in future posts.
In the meantime, take the time to read an excellent post by Kevin Foreman, Vectorform's Director of Product Vision. In "The Glass Experience", Kevin provides an in-depth reflection of his experience wearing Google Glass throughout his daily routine. He also explains the inner workings of the hardware, the strengths and limitations of the Glass system, details about the user interface, the "on-board" accessories, and a few comments about what it is like to be a new Glass-wearing celebrity.
RELATED
Here are just three of the innovators I met during my recent visit to Vectorform, left to right: Jennifer Tonio, Marketing Manager, Kevin Foreman, Director of Product Vision, and Patric Samona, Director of Health Solutions.
Below are a few links related to the use of games for the assessment and intervention of autism, along with links to information about the use of games and emerging technologies for rehabilitation, health care and health promotion:
I recently had the chance to visit the team at Vectorform's Royal Oak (MI) headquarters and experience a demo of one of their newest applications, a Kinect-based game developed to assist in the assessment of children with autism. The game, known as Monkey Business, was designed in collaboration with physical, occupational, and speech therapists from Kaiser Permanente, a large health care service provider in California. The project was supported by Kaiser Permanente's Innovation Fund for Technology group.
I was impressed!
As I watched the demo, it was clear that much time and attention was given to the input of the therapists at Kaiser Permanente. The avatar in the game is a friendly monkey who embodies the mannerisms of a friendly, curious child. The artwork is visually appealing and not too busy. The sound effects relate to the interaction at hand and provide feedback to the child as they engage in various game-like tasks designed to assess visual-motor, gross motor, and basic receptive language skills. Another good feature of this application is the simplicity of the screens designed for input of the child's information and viewing data regarding progress.
In my opinion, the Monkey Business assessment game has the potential for use in school settings, and would be enhanced if additional assessment activities are provided in the application. Expanded activities within the Monkey Business environment could support a range of treatment, education, and related intervention efforts. In addition to supporting assessment and intervention of children with autism, the Monkey Business concept would be especially useful in working with children who have experienced a traumatic brain injury.
Below are screen shots and descriptions of a few activities from the Monkey Business game:
Lily Pad Jumping Test
In the jumping test, the child is asked to jump across a pond, from one lily pad to another. The child is provided with a set of toy lily pads on the floor. As the child jumps, the Kinect sensor picks up movement and displays the jumps on the screen in the form of the monkey.
Balance Beam Test
The following screen shot depicts the monkey avatar crossing a stream on a balance beam. The monkey's movements reflect the child's movements across a real balance beam in front of the Kinect sensor and a large screen display.
Block Building Test
In the block building test, the monkey avatar encourages the child to build a tower of blocks to match the one displayed on the large screen. The child uses real blocks, placed on a table, to build a tower while the monkey provides encouragements. Objects other than blocks can be used, as the Kinect sensor is capable of identifying a variety of objects that can be use for building and stacking.
Clinician Screens
The screens designed for inputting and reviewing data regarding the child are designed with simplicity and ease-of use in mind, as shown in the pictures below:
In the present version of Monkey Business, a TV remote is used to control the application. It is possible that future versions of Monkey Business could integrated with a Smartwatch to input student information and control the activities in the application.
Comment: As a school psychologist who works with children, teens, and young adults with autism and other significant disabilities, I know how difficult it can be to conduct assessments using traditional test materials. If a student has difficulty interacting with unfamiliar adults, has communication challenges, or has motor difficulties, it may not be possible to administer an assessment that generates meaningful or valid results. Even if the student is capable of completing some of the traditional test tasks, much time and effort is required on the part of the evaluator to sustain their interest, effort, and attention.
To get a better picture of a student's emerging skills, I use interactive multimedia applications during my assessments of children with autism. Newer technologies such as interactive whiteboards, larger touch-screen monitors, and tablets have proven to be useful tools in assessment. A significant drawback to my approach is that it is cobbled together and might be difficult to replicate by other evaluation teams. I see a growing need for a range of technologically-enhanced tools for assessment, including applications that offer opportunities for playful engagement. Monkey Business holds potential to fill the bill.
What next?
I'd like to share some stories about the people who make things come to life at Vectorform. During my visit, I learned that they are involved in a number of projects related to health care and are always brainstorming new ways to harness technology for their various clients.
Over the past several years, I've followed Vectorform's journey as they've jumped off of the desktop and transformed from web-developers to Post-WIMP explorers of natural user interfaces and interaction. The most recent area of exploration is Google Glass.
Intrigued by Google Glass? So am I.
During my visit to Vectorform, I had a chance to see the world - or rather Vectorform's basement conference room - through Google Glass. As soon as I put them on I was flooded with ideas, and will expand on my ideas, and those of others, in future posts.
In the meantime, take the time to read an excellent post by Kevin Foreman, Vectorform's Director of Product Vision. In "The Glass Experience", Kevin provides an in-depth reflection of his experience wearing Google Glass throughout his daily routine. He also explains the inner workings of the hardware, the strengths and limitations of the Glass system, details about the user interface, the "on-board" accessories, and a few comments about what it is like to be a new Glass-wearing celebrity.
RELATED
Here are just three of the innovators I met during my recent visit to Vectorform, left to right: Jennifer Tonio, Marketing Manager, Kevin Foreman, Director of Product Vision, and Patric Samona, Director of Health Solutions.
Below are a few links related to the use of games for the assessment and intervention of autism, along with links to information about the use of games and emerging technologies for rehabilitation, health care and health promotion:
Vectorform Partners with Kaiser
Permanente to Assess Autism Spectrum Disorder Through Innovation Patrick
Samona, Vectorform Blog, 7/10/13
Using Kinect and Happy Action Theater
as therapy: how one school is reaching autistic children Ben Kuchera, The Penny Arcade Report,
11/19/12
Hacking for autism: Apps to help
everyone on the spectrum Donna Tam,
CNET, 2/22/13
Italian Team Uses the Kinect to Treat
Autistic Children Andrea
Lorini, Epoch Times, 12/13/12
Microsoft Surface Multi-touch
Application for Pediatric Neuropsychology Assessment (Featuring Vectorform) Lynn
Marentette, TechPsych Blog, 8/26/09
Video: http://vimeo.com/4141208
Microsoft Gaming Technology Helps
Children with Cerebral Palsy eHealthNews,
8/16/12
Should We Integrate Video Games Into
Home-Based Rehabilitation Therapies for Cerebral Palsy? Elaine
Biddiss, Future Neurology, Medscape Today, 7/2012
Xbox One, Kinect 2.0 and the future of
health technology Marcelo
Calbucci, Mobihealth News, 5/26/13
Accessible Games for Health and K-12
Education: Lessons from the Classroom (SlideShare) Lynn
Marentette, 5/9/08, Games for Health Conference Presentation
Researchers: Microsoft Kinect is a
Money-Saving Telemedicine Device Gabriel
Perna, Healthcare Informatics, 2/15/13
Lowes,
LP., Alfano LN, Yetter BA, Worthen-CHaudhari, L, Hinchman W, Samona P, Flanigan
KM, Mendell JR Proof of Concept of the Ability of the Kinect to Quantify
Upper Extremity Function in Dystrophinopathy
PLoS Curr.
2013 Mar 14; 5 doi: 10.1371/currents.md.9ab5d872bbb944c6035c9f9bfd314ee2
Subscribe to:
Posts (Atom)