Kinect Interaction to support people with disabilities: DoubleFine's Happy Action Theater/Kinect Party; OAK Air Switch and Face Switch, resources

I've been experimenting with gesture and touch-based applications for many years and I'm excited to see how things have unfolded over the past couple of years, especially in the field of special education.

Last week I downloaded DoubleFine's Kinect-based Happy Action Theater/Kinect Party,  to use during some group activities with students I work with who have significant disabilities (including severe autism).  I wish I had discovered this suite of games sooner!

I had loads of fun with students and colleagues as we explored some of the 36 creative, and sometimes zany, minigames.  I had heard that DoubleFine had launched something special, but didn't realize how awesome it was until I spent some serious playful time with it at home last weekend. I then tried it out at work this past week.  

If you are planning to explore Happy Action Theater/Kinect party, keep in mind that it plays best when there are at least two people and an audience to cheer everything along.  Through the use of blob detection algorithms, the games can handle up to 6 players at a time, which is perfect for small-group special classes.

The following trailer gives just a little hint of what this suite of mini-games is all about!


I noted that many of the games were effective in helping students become more aware of their peers. They began to play and interact with one-another in ways I hadn't previously imagined.  I especially liked the fact that many of the mini-games made it possible for students in wheelchairs to participate.  

I look forward to exploring more of the games over the next few months and will follow up with a future post after I get more input from my colleagues (and students).

I learned about Kinect Party through my contact with people involved with the GestureSEN wiki. The wiki was created as part of a Professional Learning Community (PLC) for people who work with students in specialized schools, similar to the school where I work, and contains a wealth of information about the use of newer and emerging technologies, such as the iPad, Leap Motion, the Kinect, and eye-gaze systems to support young people with significant disabilities  including autism  Some members of the GestureSEN wiki have learned to code or are in the process of doing so, motivated by what they've experienced so far with their students.  (More information and links are listed in the "RELATED" section of this post.)

OAK

OAK was developed by RCAST at the University of Tokyo in collaboration with Microsoft Japan Co., Ltd. It uses the motion-tracking capabilities of Microsoft's Kinect sensor to create non-contact switches for people with limited mobility, enabling them access to computers and other electronic devices and systems.  The video below provides a nice overview of the OAK system.

The OAK Pro bundle includes the following applications:

The Air Switch software uses the distance/depth capabilities of the Kinect sensor to support gestures of the head, hands, or larger body part to turn things off or on. The infrared from the Kinect also supports the use of the Air Switch in the dark.   The color mode function captures movements from smaller parts of the body, such as a fingertip.

The Face Switch software uses facial recognition software that can track the movements of the face, mouth, tongue, and eyes.  It can identify facial parts that have moved significantly, and records motion data 

The Motion History software observes  the movement of a person's body using the video component of the Kinect sensor.   This customizes the system to the individual and ensures accuracy of the switch.   Movements are color coded and provide the person who is setting up the system a means to fit the system to the specific capabilities and needs of the user.  

The OAK system can be enhanced by the sue of peripherals, such as a USB 4 channel relay box, an IR remote control device or outlet, or other on/off switches/outlets.

The Assist-i corporation has made the OAK system and peripherals available on Amazon Japan.  From what I can tell from the company's website, the OAK software can be downloaded free for a 30-day trial.   I'd love to see how it would work with some of the students I work with who have difficulty accessing conventional switches!  It would be wonderful to come up with ways for these students to access a wider range of digital media activities and games.

RELATED
University adapting videogame technology to help physically disabled computer users
Philip Kendall, Japan Today, 10/10/12
OAK Air Switch (PC Kinect)
OneSwitch.org.uk 4/30/13
OAK Air Switch, Face Switch, Motion History Pro Bundle (pdf)
Assist-i Corporation
Amazon Ai store: Assist-i Corporation (Prices are in Yen.)

Below is a partial list of links to resources related to using or creating engaging interactive applications and games for people with special needs: 

Using Kinect in Special Ed Classrooms: Advice from Loudoun County, Virginia Teachers
Microsoft in Education Team, Microsoft in Education Blog, 6/1/12

KinectSEN-Kinect and Special Educational Needs round-up
Greg Dunan, Microsoft Coding4Fun, 10/11/12

Monkeying Around with Autism Assessments: Kinect-based game by Vectorform andKaiser Permanente therapists offers a barrel of possibilities!
Lynn Marentette, Interactive Multimedia Technology, 7/23/13

Behind the Scenes: Creating Marty the Monkey (The character from Vectorform's autism assessment app) John Einselen, Vectorform Blog, 7/24/13


Kinect Party Review: More Fun from the Fun Kings
Casey Lynch, IGN, 12/20/12


The Power of Kinect in Special Needs Education
Willemijn de Lint,  Hans Smeele, mytylschool De Ruimte

Sign Language Recognition and Translation with Kinect (pdf)
Ming Zhou, et. al.

Cool Kinect move: Reading sign language in real time
Christopher MacManus, CNET, 7/18/13

Anthony Rhys, Trinity Fields ICT

James Winchester, SENClassroom blog

PMLD Eyegaze Project at Trinity Fields

Kinect hacking using Processing

Kinect SEN and Processing Resources
Keith Manville, Oak Grove College OpenSEN

Mat's Classroom Blog

GestureSEN Wiki
KinectSEN Wiki; KinectSEN News
ProcessingSEN wiki
LeapSEN Wiki
EyegazeSEN Wiki

SEN Students and Coding
OpenSEN, 3/5/13

Processing2

Kinect for Windows Blog

Kinect For Windows
DoubleFine

Understanding Engagement, Module 3.2, Training materials for teachers of learners with severe, profound and complex learning difficulties, UK Dept. for Education


Subscribe by e-mail

Quick Link: iPads and Autism- 60 Minutes Segment

Take a few minutes to watch the 60 Minutes segment about iPad apps and autism!

http://www.cbsnews.com/video/watch/?id=7385686n

Update, plus iGaze app by Dunedin Multimedia for use during social skills group activities

So what am I up to now?  


I'd like to share with my readers that I've decided to continue in my present position as a school psychologist, while still devoting a portion of my free time to technology. From time-to-time I think deep thoughts about usability, accessibility,  and UX/Interaction related to off-the-desktop interactive multimedia applications running on screens of all sizes.  I'm hoping to create a few multimedia experiments using HTML5 and JavaScript, and explore jQuery if and when I can find the time!  


For the present school year, my main school is a program for students with more significant disabilities, including autism spectrum disorders.  My second school is a magnet high school for technology and the arts,  located on the same campus.  I also consult throughout the district on cases involving students who have suffered traumatic brain injuries, as well as students who have multiple disabilities.  I am thankful that I have a job in a school district that values 21st Century technology.  


I'm looking forward to another technology-rich school year.  I've spent some of the time I usually devote to blogging devoted exploring iPad apps instead.  Since I'm new to the world of iPads, I'm still in discovery mode.  What an adventure!   


There are plenty of educational apps out there, and many of them are suitable for students with special needs.  On the other hand, there is much room for improvement - across all iPad app categories.  Since there is very little research about what makes up a killer app- or suite of apps- for students with special needs, experimenting with  iPad apps is uncharted territory. 

I made the decision to bring my personal iPad2 to work after I discovered a number of apps that I thought would be useful in my work as a school psychologist with students who have special needs, including autism spectrum disorders.  

One of my intervention themes this year focuses on social skills.   This is especially important for students who participate in our schools community-based job training program.  I'm using some content from Unique Learning's transition materials,  as well as on-line activities from Do2Learn's JobTips website, because my aim is to facilitate social skills that will be useful in a variety of job and community settings.   


Although my main technology tool for working with groups is the SMARTBoard,  I've found that using a combination of interactive whiteboard and iPad activities to be especially effective.  I'm paving the way for more role-play activities in the future, and attempting to use technology to my advantage.


This past week, I used the iGaze app, created by Dunedin Multimedia, to help a group of high-school level students practice establishing and maintaining eye gaze, something that is difficult for most of them to demonstrate "in-person".  I was amazed.  Each student was excited to take his or her turn.  Even more amazing?  When each student took a turn, the other students looked at their eyes and faces.  No one rocked or "stimmed".  No one made noises.  I observed several instances of joint attention, much to my delight.   

Below is a video from Dunedin Multimedia's YouTube channel that is similar to what the students viewed during their group activity:

Here is some information from Dunedin Multimedia about the iGaze app:

"Eye contact is important to communication and social development, and yet the impaired ability to make and maintain eye contact is one of the most striking aspects of autism. iGaze is an eye contact simulator that can help to build confidence in using this important means of nonverbal social communication.  The app also contains information on eye contact and eye gaze, with links to relevant research."

During the social skills activity involving the iGaze app,  I used the SMART Board to display a large picture of a boss and a worker standing face to face, making eye contact, engaged in conversation.  The picture served as an anchor to remind the students of pictures and videos they'd previously viewed that illustrated the concept of face-to-face interaction and the importance of establishing eye-contact with others from time-to-time.


I'm hoping I will be able to access the YouTube videos from Dunedin so I can use them on the SMART Board. It will be interesting to see how this plays out!   I'm also planning to take a closer look at Dunedin Multimedia's emotion x app for the iPad.


RELATED 
Screen-shot of iGaze for the iPad Dunedin:

SOMEWHAT RELATED
The SMART Table at my school was updated today - I'm looking forward to using it for some group activities, now that it is back in working order and has new applications loaded up and ready to go!


If you are interested in learning more about technology related to students with special needs, be sure to check out Kate Ahern's blog, Teaching Learners with Multiple Special Needs

Kate's post about the features of Unique Learning Systems.


Upcoming:  more about tablets, interview with folks from Stantum, social-skills game-in-progress.....large displays in public spaces update....

Words of Wisdom from Ira David Socal: "Writing without the blocks" using free/low cost technologies.

"Liberate students from the cognitive waste going to mechanical issues which have nothing to do with effective communication. Help them to become communicators and storytellers, and let your teaching focus on construction of effective writing, and what separates "writing" from "talking" in our culture." -Ira David Socal


You don't need to be a student to appreciate this approach to writing and communication!


WRITING WITHOUT THE BLOCKS
Ira David Socal, SpeEdChange 3/10/11


Ira Socal starts out his thoughtful blog post by mentioning that he dictated it using a Jawbone bluetooth headset and Windows 7 Speech Recognition, which is a free component of the operating system.  He's found that this is a great approach to use with students who struggle with the writing process.   He points out that there are many barriers that students face when attempting to write, especially for those who have difficulty holding a pen or using a keyboard efficiently.  He also points out that "keyboards injure more people each year than any other workplace tool."


Below are two videos Ira Socal shared on his blog:

RELATED/SOMEWHAT RELATED
How to set up Windows 7 Speech Recognition
MIT Freedom Stick (Michigan Integrated Technologies Supports)
Freedom Stick and Firefox Accessibility
"The MITS Freedom Stick is designed to provide students with information and communication access on any computer using a Windows or Linux operating system. When you insert the USB Flash Drive into the computer the LearnApps software should load giving you a menu in the toolbar on the lower-right corner of the screen. Please note, however, that when using the software on some computers you may need to open the drive and click directly on the LearnApps icon to open."

Cross posted: TechPsych

Usability, utility, user-centered design, data security, Section 508 Accessibility, interoperability: Issues related to Electronic Health Records (and student information systems?) Link to info and resources

Although this topic is not directly related to the focus of this blog, I think some of my readers would be interested in my recent TechPsych post about the topic of electronic health and education records:


Electronic Health Records, Electronic Special Education Records: What are the issues?  (Usability, utility, user-centered design, data security, interoperability, Section 508 accessibility....)


In the post, I've included direct links to in-depth resources that apply to health and education information systems, including information from the U.S. government regarding information systems and data ethics.

HP TouchSmart PC Supports Communication for Students with Autism

Autism Tech | Consumer Health Video Library - Heath Video

Posted using ShareThis

Hope Technology School

Link to post: Assisting young people with autism spectrum disorders through the use of digital media

If you have an interest in digital media and interventions for young people with autism disorders, take a look at my recent TechPsych post:.

Cognitive Bursts; Sense of Self: Assisting young people with autism spectrum disorders through the use of digital media.

iSocial: A Virtual Social Space for Youth with ASD, built with Croquet

iSocial is a 3D environment developed to support conversation and positive social skills for young people who have autism spectrum disorders. iSocial is a work in progress, designed by a team of researchers at the University of Missouri.

iSocial runs on Croquet, an open-source 3D SDK/tool designed to build 3D virtual collaborative environments. (EduSim applications also run on Croquet.)

iSocial Video

Cross Post: Children of the Code

I came across the Children of the Code website today and thought I'd share it- if you are familiar with Children of the Code project, please leave a comment, since I haven't yet explored the entire site.

Childrenof thecode.org is an on-line multi-media resource that is part of a public/social education project to that aims to spread information and education about "The Code and the Challenge of Learning to Read it"

The project has four main components:

1. A three hour Public Television, DVD and Web documentary series;
2. A ten-hour college, university, and professional development DVD series;
3. A series of teacher and parent presentations and seminars;
4. A cross-indexed website/database containing audio, video and transcripts with the world's leading experts in fields related to reading.

The website includes over 100 interviews of people from a wide range of disciplines who are committed to promotion of health, education, and well-being of children, youth, and in turn, communities and society.

Here is the project's abstract:

"Abstract: Our children's cognitive and emotional development, self-esteem, academic, and later social and economic success, all depend on how well they learn - on the health of their learning. Whether we are involved in parenting, teaching, cognitive science, psychology, pedagogy, curriculum design, instructional design, direct instruction, constructivism, assessment, multiple intelligences, learning styles, learning differences, learning disabilities, learning theory, learning communities, organizational learning, preschool, elementary school, middle school, high school, home school, unschooling, college, university... we all share the responsibility of stewarding the health of our children's learning."


I'd recommend starting with the on-line video tour of the project:
http://www.childrenofthecode.org/Tour/index.htm

The following is a quote from an interview with David Boulton, the director of the Children of the Code project:

"The mission of the Children of the Code project is to catalyze and resource a transformation in how our society thinks about the "code" of our written language and the "challenges involved in learning to read it.". I think we're living in the "Stone Age of Literacy." Our lack of understanding of what is involved and what is at stake in acquiring literacy is wreaking havoc on the lives of our population, including children"

"..the first think I hope i that it (the project) changes the mental lens through which parents and teachers see struggling learners. I want them to see someone who is struggling as somebody who is struggling with an artificially confusing technology (written language) and somebody who is in significant emotional and cognitive danger.

"What I hope is that people realize that if children and adults struggle too long with the process of acquiring literacy, it can seriously affect how they develop and grow and learn. Struggling to read causes many, many people to grow up feeling ashamed of their mind.."

Links to the list of some of the interviews of "name" researchers and educators are posted on the TechPsych blog, or you can visit the site's interview list.

Link to Mechelle De Craene's Blog

Mechelle De Craen is a special education teacher who integrates technology into her work with students. Some EduSpaces e-Learning 2.0 Blog. She is a member of the following on-line communities: #
Community memberships

* MirandaNet
* Blogs, wikis, and social software for teaching and learning.
* Pedagogical impact
* Digital Storytelling
* eportfolios
* nanoHUB for Kids
* K-12 Education
* Multimedia Development

Link to info about a "super touch screen" for Google Earth - it has multiple uses.

Watch this video about a "super touch screen" for Google Earth from Perceptive Pixel! More information about this can be found on the TechPsych blog, and from the Google Earth Blog. I think this application would be great for visual learners.

Applications like this are immersive and engaging. If you are an educator, think about the ways you could use this application in your classroom!

Link to related article.

-Lynn