Drawing with Sounds, an Interactive Musical Drawing App Created with Processing

This is my first creation using the Drawing with Sounds application. This cause and effect application was created with Processing 1.5.1 and produces musical sounds and patterns, along with random abstract shapes that are triggered through movements of a mouse. The application has been adapted for switch users, making it accessible to those who have motor impairments.

Processing is an open-source programming and integrated development environment that was build on the Java programming that has been easy to learn by people who have limited technical backgrounds. It is used by people from from a range of fields, such as art, music, journalism, and data visualization.

The fact that Processing is easy to use makes it great for people with just a little bit of coding knowledge to adopt a variety of open-source applications and tweak a few of the variables to according to the need.   Using the Drawing with Sounds app as an example, I wanted to change the size of the interactive canvas.  By typing in a few keystrokes,  I changed the original size of the screen from 800 x 800 to something larger, as shown below:

I then typed in another variable for the background color, and was pleased with the results:

The application was developed from Andrew R. Brown's tutorial on "SoundCipher", based on the SoundCipher library for Processing.  According to the description of the application, "The sketch is intended for use....as a sensory stimulus using sound, shape and color to create engagement". 

Below is a videoclip created by Keith Manville, demonstrating what happens when you interact with Drawing with Sounds: 

I look forward to sharing more on this topic in future blog posts.  I'll be sure to include basic "how-to" information for my "low-tech" readers and colleagues who desire to learn a bit more about using and creating basic interactive applications that appeal to young people with special needs.  

RELATED

Note:  SEN is the UK acronym for "Special Educational Needs"

Drawing with Sounds Processing sketch

James Winchester, SEN Classroom Blog, 10/31/13

Drawing with Sound

Keith Manville, opensen blog, 10/27/12

"Open-source, Sensory and Interactive Technology in SEN"

ProcessingSEN Wiki

ProcessingSEN Drawing with Sounds Download Page

(An accessible switch-enabled version of Drawing with Sounds can be downloaded from the above link.)

Free sensory applications built in Processing 1.5.1"Download links for sensory applications built in Processing 1.5.1, designed to assist SEN students in engaging in learning through the the use of ICT, interactive “Smart” boards, touchscreen or tablet technologies.  Our experience is that using applications on these devices or with natural user interfaces such as the Microsoft Kinect can increase the opportunities for engagement and social communication with many students."

I will be posting more information about emerging interactive technologies for special needs in the future. In the meantime, take a look at the following resources:

The SHAPE Project, technologies to enhance learning for young people on the Autism Spectrum

Keith Manville, opensen blog, 11/10/12SEN Classroom "Ideas and Tech used in a SEN class"

iPadSEN

GestureSEN Wiki

LeapSEN

KinectSEN Wiki
"Using Kinect in Special Schools for Pupils with Severe Learning Difficulties"
iGazeSEN
OPENSEN Videos

Monkeying Around with Autism Assessments: Kinect-based game by Vectorform and Kaiser Permanente therapists offers a barrel of possibilities!

"The goal was to build a game that is extremely accessible, non-complex and includes simple mechanics that children with autism can quickly understand to retain their attention, prevent over-stimulation, encourage play, and prevent frustration." -Vectorform "Monkey Business" Team


I recently had the chance to visit the team at Vectorform's Royal Oak (MI) headquarters and experience a demo of one of their newest applications, a Kinect-based game developed to assist in the assessment of children with autism. The game, known as Monkey Business, was designed in collaboration with physical, occupational, and speech therapists from Kaiser Permanente, a large health care service provider in California. The project was supported by Kaiser Permanente's Innovation Fund for Technology group.

I was impressed!

As I watched the demo, it was clear that much time and attention was given to the input of the therapists at Kaiser Permanente. The avatar in the game is a friendly monkey who embodies the mannerisms of a friendly, curious child.  The artwork is visually appealing and not too busy. The sound effects relate to the interaction at hand and provide feedback to the child as they engage in various game-like tasks designed to assess visual-motor, gross motor, and basic receptive language skills.  Another good feature of this application is the simplicity of the screens designed for input of the child's information and viewing data regarding progress.

In my opinion, the Monkey Business assessment game has the potential for use in school settings, and would be enhanced if additional assessment activities are provided in the application.  Expanded activities within the Monkey Business environment could support a range of treatment, education, and related intervention efforts. In addition to supporting assessment and intervention of children with autism, the Monkey Business concept would be especially useful in working with children who have experienced a traumatic brain injury.


Below are screen shots and descriptions of a few activities from the Monkey Business game:

Lily Pad Jumping Test
In the jumping test, the child is asked to jump across a pond, from one lily pad to another.  The child is provided with a set of toy lily pads on the floor.  As the child jumps, the Kinect sensor picks up movement and displays the jumps on the screen in the form of the monkey.

Balance Beam Test
The following screen shot depicts the monkey avatar crossing a stream on a balance beam. The monkey's movements reflect the child's movements across a real balance beam in front of the Kinect sensor and a large screen display.

Block Building Test
In the block building test, the monkey avatar encourages the child to build a tower of blocks to match the one displayed on the large screen.  The child uses real blocks, placed on a table, to build a tower while the monkey provides encouragements.  Objects other than blocks can be used, as the Kinect sensor is capable of identifying a variety of objects that can be use for building and stacking.

Clinician Screens
The screens designed for inputting and reviewing data regarding the child are designed with simplicity and ease-of use in mind, as shown in the pictures below:

In the present version of Monkey Business, a TV remote is used to control the application. It is possible that future versions of Monkey Business could integrated with a Smartwatch to input student information and control the activities in the application.  


Comment: As a school psychologist who works with children, teens, and young adults with autism and other significant disabilities, I know how difficult it can be to conduct assessments using traditional test materials.  If a student has difficulty interacting with unfamiliar adults, has communication challenges, or has motor difficulties, it may not be possible to administer an assessment that generates meaningful or valid results.  Even if the student is capable of completing some of the traditional test tasks, much time and effort is required on the part of the evaluator to sustain their interest, effort, and attention.   

To get a better picture of a student's emerging skills, I use interactive multimedia applications during my assessments of children with autism.  Newer technologies such as interactive whiteboards, larger touch-screen monitors, and tablets have proven to be useful tools in assessment.   A significant drawback to my approach is that it is cobbled together and might be difficult to replicate by other evaluation teams.  I see a growing need for a range of technologically-enhanced tools for assessment, including applications that offer opportunities for playful engagement.  Monkey Business holds potential to fill the bill.

What next?

I'd like to share some stories about the people who make things come to life at Vectorform. During my visit, I learned that they are involved in a number of projects related to health care and are always brainstorming new ways to harness technology for their various clients.  

Over the past several years, I've followed Vectorform's journey as they've jumped off of the desktop and transformed from web-developers to Post-WIMP explorers of natural user interfaces and interaction.  The most recent area of exploration is Google Glass. 

Intrigued by Google Glass?  So am I.  

During my visit to Vectorform, I had a chance to see the world - or rather Vectorform's basement conference room - through Google Glass.  As soon as I put them on I was flooded with ideas, and will expand on my ideas, and those of others, in future posts.

In the meantime, take the time to read an excellent post by Kevin Foreman, Vectorform's Director of Product Vision.  In "The Glass Experience", Kevin provides an in-depth reflection of his experience wearing Google Glass throughout his daily routine.  He also explains the inner workings of the hardware, the strengths and limitations of the Glass system, details about the user interface, the "on-board" accessories, and a few comments about what it is like to be a new Glass-wearing celebrity.


RELATED
Here are just three of the innovators I met during my recent visit to Vectorform, left to right: Jennifer Tonio, Marketing Manager, Kevin Foreman, Director of Product Vision, and Patric Samona, Director of Health Solutions.

Below are a few links related to the use of games for the assessment and intervention of autism, along with links to information about the use of games and emerging technologies for rehabilitation, health care and health promotion:

Vectorform Partners with Kaiser Permanente to Assess Autism Spectrum Disorder Through Innovation  Patrick Samona, Vectorform Blog, 7/10/13


Kaiser talks up monkey game, talks down EMR integration Jonah Comstock, Mobihealth News, 7/9/13

Using Kinect and Happy Action Theater as therapy: how one school is reaching autistic children Ben Kuchera, The Penny Arcade Report, 11/19/12

Microsoft Kinect and Autism (SlideShare) Susan McCarthy, Little Angel's School 2/09/13

Hacking for autism: Apps to help everyone on the spectrum Donna Tam, CNET, 2/22/13

Italian Team Uses the Kinect to Treat Autistic Children Andrea Lorini, Epoch Times, 12/13/12

Microsoft Surface Multi-touch Application for Pediatric Neuropsychology Assessment (Featuring Vectorform) Lynn Marentette, TechPsych Blog, 8/26/09   

Video: http://vimeo.com/4141208

Microsoft Gaming Technology Helps Children with Cerebral Palsy eHealthNews, 8/16/12

Should We Integrate Video Games Into Home-Based Rehabilitation Therapies for Cerebral Palsy? Elaine Biddiss, Future Neurology, Medscape Today, 7/2012

Xbox One, Kinect 2.0 and the future of health technology Marcelo Calbucci, Mobihealth News, 5/26/13

Games for Health Website

Kinect: Interactive Multimedia Technology Posts

Accessible Games for Health and K-12 Education: Lessons from the Classroom (SlideShare) Lynn Marentette, 5/9/08, Games for Health Conference Presentation

Researchers: Microsoft Kinect is a Money-Saving Telemedicine Device Gabriel Perna, Healthcare Informatics, 2/15/13

Lowes, LP., Alfano LN, Yetter BA, Worthen-CHaudhari, L, Hinchman W, Samona P, Flanigan KM, Mendell JR Proof of Concept of the Ability of the Kinect to Quantify Upper Extremity Function in Dystrophinopathy

PLoS Curr. 2013 Mar 14; 5   doi:  10.1371/currents.md.9ab5d872bbb944c6035c9f9bfd314ee2

iPad3 and iPad Mini: Hands-on Side-by Side Comparison Video, by Eric Sailers (quick post)

Here is a good side-by-side "hands-on" comparison of the new iPad3 and the new iPad Mini by Eric Sailers:

Eric Sailers is a speech and language pathologist who has co-created apps for iOS devices since 2009. His website has a wealth of information iOS devices and apps for education, especially for children with special needs.  

60-Minutes Segment about iPads and Autism; James Winchester's Tech and Special Needs Blog

Tonight's episode of 60 Minutes included a repeat of a segment about the use of iPad apps with young people who have autism spectrum disorders.   I missed it, but I found it on the CBS website. 


Along with the segment, I found several related videos and transcripts. If you have a moment, take the time to look!


Apps for Autism (60 Minutes Video)

RELATED
Interview of Temple Grandin about autism
Temple Grandin's Unique Brain
SEN Classroom: Ideas and Tech in a SEN Classroom
(James Winchester's blog)
James is a special educator who has a wealth of  "how-to" knowledge about technology and special needs. If you are interested, take a look at his blog's archive. He writes about iPad apps,  the use of the Kinect with students at his school, and more.  He specializes in a Life Skills curriculum, which focuses on social, communication, and vocational skills that the students will need as they become members of the community.  



I recently wrote a post about Po-Motion, an interactive tech start-up based in Winnipeg, Canada, and learned that the system is used as an interactive wall display in a sensory room at a school for children who have severe disabilities, including autism. More information about the use of this system, including a video, can be found on James Winchester's blog post, Po-Motion Interactive Wall in the Sensory Room.  



Comment:
In my work as a school psychologist, I use technology with students who have severe autism several days a week, along with my colleagues.  I plan to share more information on this topic from time-to-time in future posts. 


I am putting together a web page with resources about autism and technology. My resources include descriptions of systems and applications, videos, and presentation slides from a variety of researchers, developers, and practitioners.  Suggestions are welcome!

Update, plus iGaze app by Dunedin Multimedia for use during social skills group activities

So what am I up to now?  


I'd like to share with my readers that I've decided to continue in my present position as a school psychologist, while still devoting a portion of my free time to technology. From time-to-time I think deep thoughts about usability, accessibility,  and UX/Interaction related to off-the-desktop interactive multimedia applications running on screens of all sizes.  I'm hoping to create a few multimedia experiments using HTML5 and JavaScript, and explore jQuery if and when I can find the time!  


For the present school year, my main school is a program for students with more significant disabilities, including autism spectrum disorders.  My second school is a magnet high school for technology and the arts,  located on the same campus.  I also consult throughout the district on cases involving students who have suffered traumatic brain injuries, as well as students who have multiple disabilities.  I am thankful that I have a job in a school district that values 21st Century technology.  


I'm looking forward to another technology-rich school year.  I've spent some of the time I usually devote to blogging devoted exploring iPad apps instead.  Since I'm new to the world of iPads, I'm still in discovery mode.  What an adventure!   


There are plenty of educational apps out there, and many of them are suitable for students with special needs.  On the other hand, there is much room for improvement - across all iPad app categories.  Since there is very little research about what makes up a killer app- or suite of apps- for students with special needs, experimenting with  iPad apps is uncharted territory. 

I made the decision to bring my personal iPad2 to work after I discovered a number of apps that I thought would be useful in my work as a school psychologist with students who have special needs, including autism spectrum disorders.  

One of my intervention themes this year focuses on social skills.   This is especially important for students who participate in our schools community-based job training program.  I'm using some content from Unique Learning's transition materials,  as well as on-line activities from Do2Learn's JobTips website, because my aim is to facilitate social skills that will be useful in a variety of job and community settings.   


Although my main technology tool for working with groups is the SMARTBoard,  I've found that using a combination of interactive whiteboard and iPad activities to be especially effective.  I'm paving the way for more role-play activities in the future, and attempting to use technology to my advantage.


This past week, I used the iGaze app, created by Dunedin Multimedia, to help a group of high-school level students practice establishing and maintaining eye gaze, something that is difficult for most of them to demonstrate "in-person".  I was amazed.  Each student was excited to take his or her turn.  Even more amazing?  When each student took a turn, the other students looked at their eyes and faces.  No one rocked or "stimmed".  No one made noises.  I observed several instances of joint attention, much to my delight.   

Below is a video from Dunedin Multimedia's YouTube channel that is similar to what the students viewed during their group activity:

Here is some information from Dunedin Multimedia about the iGaze app:

"Eye contact is important to communication and social development, and yet the impaired ability to make and maintain eye contact is one of the most striking aspects of autism. iGaze is an eye contact simulator that can help to build confidence in using this important means of nonverbal social communication.  The app also contains information on eye contact and eye gaze, with links to relevant research."

During the social skills activity involving the iGaze app,  I used the SMART Board to display a large picture of a boss and a worker standing face to face, making eye contact, engaged in conversation.  The picture served as an anchor to remind the students of pictures and videos they'd previously viewed that illustrated the concept of face-to-face interaction and the importance of establishing eye-contact with others from time-to-time.


I'm hoping I will be able to access the YouTube videos from Dunedin so I can use them on the SMART Board. It will be interesting to see how this plays out!   I'm also planning to take a closer look at Dunedin Multimedia's emotion x app for the iPad.


RELATED 
Screen-shot of iGaze for the iPad Dunedin:

SOMEWHAT RELATED
The SMART Table at my school was updated today - I'm looking forward to using it for some group activities, now that it is back in working order and has new applications loaded up and ready to go!


If you are interested in learning more about technology related to students with special needs, be sure to check out Kate Ahern's blog, Teaching Learners with Multiple Special Needs

Kate's post about the features of Unique Learning Systems.


Upcoming:  more about tablets, interview with folks from Stantum, social-skills game-in-progress.....large displays in public spaces update....

Link: Songify for Instant Musical Social Stories?!

Here is the link to the post on the TechPsych blog:


Songify for Instant Musical Social Stories?!

AAC Technology at Camp (augmentative and alternative communication) -via Kate Ahern

The following video clip is an awesome example of how AAC technology (augmentative and alternative communication) can be integrated into a range of activities- learning, social, leisure, and creative, when everyone makes an effort to make it work- and not give up.  Thanks to Kate Ahern for sharing this! (Cross-posted on the TechPsych blog.)

The song in the background is "Talk", by Coldplay, a perfect fit for the theme. 

 

"This year's AAC Summer Camp students taught us a lot. This video highlights some important things to think about when it comes to augmentative and alternative communication." -Communicare LLC

RELATED 

Let's Go To AAC Camp! (Includes a list of AAC camps around the U.S.)

Kate Ahern, Teaching Learners with Multiple Special Needs, 4/25/11 

Communicare: Speech-language pathologist specializing in Augmentative and Alternative Communication (AAC) Communicare's resource page

Lyrics to Coldplay's "Talk"

COMMENT
Most of the devices highlighted in the video are very expensive and are difficult to integrate seamlessly with other technologies that are emerging in classrooms- and in homes.  I am in the process of writing a post (or two) about this topic. The ultimate goal of AAC technology is to provide people with a means to communicate more seamlessly with people beyond the "sheltered" circle of therapists, special educators, parents and teachers.

Interactive technologists, from a range of disciplines, can help make this happen.

Eric Sailer's Lists of iPad, iPod, and iPhone Apps for (special) education

Update: Here is a link to Eric Sailer's resources on Scribd:
http://www.scribd.com/doc/24470331/iPhone-iPad-and-iPod-touch-Apps-for-Special-Education



Update:  I think Eric Sailer's blog has moved.  For now, here is a link to his website:
http://ericsailers.com/iresources.html

Eric Sailers is a speech/language pathologist who has an extensive amount of experience using "apps" in educational settings.  He's compiled a great list of apps and related resources on his blog:  http://slpsharing.com/app-resources/

Eric spends some of his time giving presentations about the use of Apple's mobile devices in education. He's co-created a few apps, too!