This is my first creation using the Drawing with Sounds application. This cause and effect application was created with Processing 1.5.1 and produces musical sounds and patterns, along with random abstract shapes that are triggered through movements of a mouse. The application has been adapted for switch users, making it accessible to those who have motor impairments.
Processing is an open-source programming and integrated development environment that was build on the Java programming that has been easy to learn by people who have limited technical backgrounds. It is used by people from from a range of fields, such as art, music, journalism, and data visualization.
The fact that Processing is easy to use makes it great for people with just a little bit of coding knowledge to adopt a variety of open-source applications and tweak a few of the variables to according to the need. Using the Drawing with Sounds app as an example, I wanted to change the size of the interactive canvas. By typing in a few keystrokes, I changed the original size of the screen from 800 x 800 to something larger, as shown below:
I then typed in another variable for the background color, and was pleased with the results:
The application was developed from Andrew R. Brown's tutorial on "SoundCipher", based on the SoundCipher library for Processing. According to the description of the application, "The sketch is intended for use....as a sensory stimulus using sound, shape and color to create engagement".
Below is a videoclip created by Keith Manville, demonstrating what happens when you interact with Drawing with Sounds:
I look forward to sharing more on this topic in future blog posts. I'll be sure to include basic "how-to" information for my "low-tech" readers and colleagues who desire to learn a bit more about using and creating basic interactive applications that appeal to young people with special needs.
RELATED
Note: SEN is the UK acronym for "Special Educational Needs"
(An accessible switch-enabled version of Drawing with Sounds can be downloaded from the above link.)
Free sensory applications built in Processing 1.5.1"Download links for sensory applications built in Processing 1.5.1, designed to assist SEN students in engaging in learning through the the use of ICT, interactive “Smart” boards, touchscreen or tablet technologies. Our experience is that using applications on these devices or with natural user interfaces such as the Microsoft Kinect can increase the opportunities for engagement and social communication with many students."
I will be posting more information about emerging interactive technologies for special needs in the future. In the meantime, take a look at the following resources:
"The goal was to build a game that is extremely accessible, non-complex and includes simple mechanics that children with autism can quickly understand to retain their attention, prevent over-stimulation, encourage play, and prevent frustration." -Vectorform "Monkey Business" Team I recently had the chance to visit the team at Vectorform's Royal Oak (MI) headquarters and experience a demo of one of their newest applications, a Kinect-based game developed to assist in the assessment of children with autism. The game, known as Monkey Business, was designed in collaboration with physical, occupational, and speech therapists from Kaiser Permanente, a large health care service provider in California. The project was supported by Kaiser Permanente's Innovation Fund for Technology group. I was impressed! As I watched the demo, it was clear that much time and attention was given to the input of the therapists at Kaiser Permanente. The avatar in the game is a friendly monkey who embodies the mannerisms of a friendly, curious child. The artwork is visually appealing and not too busy. The sound effects relate to the interaction at hand and provide feedback to the child as they engage in various game-like tasks designed to assess visual-motor, gross motor, and basic receptive language skills. Another good feature of this application is the simplicity of the screens designed for input of the child's information and viewing data regarding progress. In my opinion, the Monkey Business assessment game has the potential for use in school settings, and would be enhanced if additional assessment activities are provided in the application. Expanded activities within the Monkey Business environment could support a range of treatment, education, and related intervention efforts. In addition to supporting assessment and intervention of children with autism, the Monkey Business concept would be especially useful in working with children who have experienced a traumatic brain injury. Below are screen shots and descriptions of a few activities from the Monkey Business game: Lily Pad Jumping Test In the jumping test, the child is asked to jump across a pond, from one lily pad to another. The child is provided with a set of toy lily pads on the floor. As the child jumps, the Kinect sensor picks up movement and displays the jumps on the screen in the form of the monkey.
Balance Beam Test The following screen shot depicts the monkey avatar crossing a stream on a balance beam. The monkey's movements reflect the child's movements across a real balance beam in front of the Kinect sensor and a large screen display.
Block Building Test In the block building test, the monkey avatar encourages the child to build a tower of blocks to match the one displayed on the large screen. The child uses real blocks, placed on a table, to build a tower while the monkey provides encouragements. Objects other than blocks can be used, as the Kinect sensor is capable of identifying a variety of objects that can be use for building and stacking.
Clinician Screens The screens designed for inputting and reviewing data regarding the child are designed with simplicity and ease-of use in mind, as shown in the pictures below:
In the present version of Monkey Business, a TV remote is used to control the application. It is possible that future versions of Monkey Business could integrated with a Smartwatch to input student information and control the activities in the application. Comment: As a school psychologist who works with children, teens, and young adults with autism and other significant disabilities, I know how difficult it can be to conduct assessments using traditional test materials. If a student has difficulty interacting with unfamiliar adults, has communication challenges, or has motor difficulties, it may not be possible to administer an assessment that generates meaningful or valid results. Even if the student is capable of completing some of the traditional test tasks, much time and effort is required on the part of the evaluator to sustain their interest, effort, and attention. To get a better picture of a student's emerging skills, I use interactive multimedia applications during my assessments of children with autism. Newer technologies such as interactive whiteboards, larger touch-screen monitors, and tablets have proven to be useful tools in assessment. A significant drawback to my approach is that it is cobbled together and might be difficult to replicate by other evaluation teams. I see a growing need for a range of technologically-enhanced tools for assessment, including applications that offer opportunities for playful engagement. Monkey Business holds potential to fill the bill. What next? I'd like to share some stories about the people who make things come to life at Vectorform. During my visit, I learned that they are involved in a number of projects related to health care and are always brainstorming new ways to harness technology for their various clients.
Over the past several years, I've followed Vectorform's journey as they've jumped off of the desktop and transformed from web-developers to Post-WIMP explorers of natural user interfaces and interaction. The most recent area of exploration is Google Glass. Intrigued by Google Glass? So am I. During my visit to Vectorform, I had a chance to see the world - or rather Vectorform's basement conference room - through Google Glass. As soon as I put them on I was flooded with ideas, and will expand on my ideas, and those of others, in future posts. In the meantime, take the time to read an excellent post by Kevin Foreman, Vectorform's Director of Product Vision. In "The Glass Experience", Kevin provides an in-depth reflection of his experience wearing Google Glass throughout his daily routine. He also explains the inner workings of the hardware, the strengths and limitations of the Glass system, details about the user interface, the "on-board" accessories, and a few comments about what it is like to be a new Glass-wearing celebrity.
RELATED Here are just three of the innovators I met during my recent visit to Vectorform, left to right: Jennifer Tonio, Marketing Manager, Kevin Foreman, Director of Product Vision, and Patric Samona, Director of Health Solutions.
Below are a few links related to the use of games for the assessment and intervention of autism, along with links to information about the use of games and emerging technologies for rehabilitation, health care and health promotion:
I first heard the term 'uncanny valley' about eight or nine years ago when I was taking a 3D-modeling class. At that time, the technology available was not close to reaching this valley - where robots or computer-generated characters are so real that they are almost repulsive. A lot has changed over the years. The following video, recently featured by Activision Blizzard during the 2013 Game Developer Conference, has attracted much attention in just a few days, partly because it is so real.
Although I noticed that a little more work needs to be done with the teeth, I was impressed. I liked the quality of the eye shaders that were used in the creation of this demo. Examples of faces created with this feature turned on and off can be found on Jorge Jimenez' blog. Jorge's slides from a 2012 course offered during SIGGRAPH provide additional information. Computer processers have become powerful enough to handle quite a lot of processing, and the tech world has been spreading the word. Below is a presentation by Jen-Hsun Huang, CEO of Nvidia, about the company's work simulate the human face, touching on the 'uncanny valley':
Although the use of this technology to create characters, realistic on many levels, seems to be a bit creepy, it might be OK after some refinement. There are a few questions that remain unanswered. What would be the impact on children or teens who might spend many hours each week playing games with such realistic characters? I'd hate to have a nightmare featuring one of these guys! I think that this technology might have some potential for use in serious games and simulations, such as preparing emergency workers to handle a variety of realistic scenarios. Games with realistic digital characters, capable of generating a range of facial expressions, might be useful to support the learning of social interaction skills among young people with autism spectrum disorders.
I have spent much of my adult life as a school psychologist, working with children, teens, young adults, and those who care about them. This Sandy Hill Elementary tragedy hit close to home, because among those killed was a fellow school psychologist, Mary Sherlach. My heart goes out to everyone who has been touched by this event in some way, especially the families of those who lost their lives. Before going on to the rest of my post, I encourage you to take a few moments to listen to the children singing Silent Night in honor of those who lost their lives on Friday, December 14, 2012.
RESOURCES AND LINKS
Over the past few days, members of the National Association of School Psychologists (NASP)have been providing information and resources for the public to assist with coping with the tragedy, and Twitter and Facebook have proven to be great ways to spread the word. Some of the information below is appropriate for those who are directly involved with the children who attend Sandy Hook Elementary or those who attend other elementary schools in the community. Other articles have suggestions that would be appropriate for parents, teachers, and support staff in schools around the nation (and world). There are other articles below that are good for those responsible for planning longer-term safe school strategies
REFLECTIONS Violence prevention is a complex problem. From my point of view, preventing or minimizing tragedies such as the one experienced at Sandy Hook Elementary will take a systems approach, and involve people from a number of disciplines. I am convinced that it will take a larger number of people, who can commit to sustaining their efforts over a long period of time. What troubles me in this case is that the perpetrator was homicidal, suicidal. and matricidal. He had the wherewithal to know that the school most likely had a school safety plan in place. He did the unthinkable - he used a powerful weapon to gain access by blasting through the entry doors. And he would not stop until he had killed 20 young children and a good handful of caring adults. The problem of protecting schools from intruders can't be solved by making schools more prison-like. Sandy Hook school reportedly had a video surveillance system, locked front doors, a policy for visitors to be buzzed in, and a safe schools policy that required that all teachers (and students) receive training in what to do in an emergency requiring a school lock-down. If these things were not in place, many more lives would have been lost.
Firearms: The Elephant in Our Nation's Living-Room The elephant that is taking up a huge space in our country's living room? Weapons that are appropriate for military and law enforcement use, such as the semi-automatic rifle that was in the hands of this young man, are readily available in most communities. We know that the perpetrator did not own the weapons he brought to Sandy Hook Elementary School. We know that he had easy access to it because it was purchased legally by his mother, and apparently was kept in his home.
This topic is a political hot potato, but worthy of serious study. With "Big Data", advanced analytics, and some engaging information visualizations, I am sure something positive can emerge from the 'debate'. (Nate Silver, author of the New York Times FiveThirtyEight blog, might have a few words to say on this topic, judging from his December 14th post, "In Public 'Conversation' on Guns, a Rhetorical Shift".)
Need for Research - Co-Morbidity - Identification, Evidence-based Treatment, and Prevention At the time of this post, it was not certain if the perpetrator had a disability, a personality disorder, or a mental health disorder that may have contributed to his violent acts. We do not know if he was receiving treatment, or if he had ever been hospitalized or prescribed medication. Whatever the case, I am sure that the tragedy that happened at the Sandy Hook elementary school has caused additional worry for a number of parents of teens and young adults who struggle from mental health problems and other disorders. Having worked with a number of young people with challenging mental health concerns over the years, including psychiatric patients with a history of serious violence, my heart goes out to these parents. I am sure some of them wonder if the next crisis that flashes on the news is something that was initiated by their child, no matter what the child's age. Not everyone is aware that serious mental health disorders can crop insidiously during the early teens. The "red flags" might be attributed to puberty, and not be interpreted as precursors to something more serious. Some mental illnesses become full -blown during the later teen years or during young adulthood, often past the age where a school psychologist or similar professional in the education system can step in to intervene, or provide resources and information for parents. Many families have struggled to obtain an appropriate level of care for their teen or young adult child, only to find that these services have been reduced due to budget cuts to mental health and transition-to-adulthood programs in their communities. Many of these young people have good potential, but their mood states, unusual mannerisms, or periods of erratic thought processes make it difficult to interview for jobs and keep them unless they receive consistent support in the form of counseling, job coaching, and/or medical management. There is a need for more research and support that focuses on the needs of teen to young adult age group. This includes research in psychological treatment,group counseling approaches, psychopharmacology, mental health education, and more. Update: For a graphic description of what this might be like for a parent of a teen who struggles with emotional/behavioral issues at time, see theblog postthat has been circulating around the internet. The mother, Liza Long, wrote the post immediately after she learned of the Sandy Hook shootings. Her post triggered hundreds of comments, some of them critical. In response, an anonymous school psychologist reflected on some experiences with a student in need of support. I do not think the school psychologist's story exaggerates reality in any way.
Role of technology in Intervention and Prevention: Interactive digital media, such as serious games, might play a role in this effort. For example, the Australian National University developed the MoodGym training program, an interactive, free web-based application that use a cognitive-behavior approach to cope with anxiety and depression, and E-Couch, an online program for preventing and coping with depression, generalized anxiety disorder, and social anxiety disorder.
For middle-school children, the PBS ItsMyLife website provides a number of activities, games, and videos that explore topics such as emotions, school, friendships, bullying, and more. The website includes lesson plans for teachers and tips for parents. More research is underway to support the use of interactive digital technology to support mental health. One example is the work of Stanford University's Calming Technology lab. Another example is the use virtual reality for treating post-traumatic stress disorder. The USC Institute of Creative Technologies has been involved in this area for a number of years. MindHabits, a suite of games developed by psychologists in Canada, was based on research and aims to help people reduce stress.
Coincidentally, Connecticut's Southwest Regional Mental Health Board is developing a web-based resource for young adults with mental health concerns, if the information from a job posting for the project is correct: "The goal of this project is to design or adopt an electronic/virtual system to engage young adults in mental health services. This system would assist young adults in enhancing their own mental health, and addressing mental health issues as they arise or develop via an interactive, multi-media electronic platform. This platform would integrate social networking and information media which may include website, facebook, You-Tube, Skype, blog, chat room, texting and phone apps. Development of this platform will be informed by Connecticut youth and young adults and advised by an advisory group of statewide young adult leaders in the mental health and addiction fields." It is my hope that the power of the "social" web will support efforts to collaborate and tackle this problem, on many levels. RELATED
Mental Health Issues - Co-morbidities After I learned that it was speculated (but not confirmed) that the perpetrator of the Sandy Hook Elementary School shootings may have had Asperger's Syndrome, I thought it would be helpful to include additional on the topic of Asperger's Syndrome and co-morbid mental health disorders.
Mazzone, Luigi; Ruta, Liliana; Reale, Laura. Psychiatric comorbidities in Asperger syndrome and high functioning autism: diagnostic challenges. Annals of General Psychiatry. 2012; 11: 16.
