Designing for Touch & Gesture: Tips for Apps and the Web (Updated)

In the past, our fingers did the walking, sifting through files, papers, pamphlets, and phonebooks, and then by point-click-clicking with a mouse to interact with images and text, in essence, electronic imitations of the paper-based world. Traditional forms, brochures, ad inserts, and posters informed much of the design. 

How much have things change?   It is 2013, but you'd think it was 1997 from the PowerPoint look and feel of many apps and web sites!   Touch is everywhere, but from what I can tell, not enough designers and developers have stepped up to the plate to think more deeply about ways their applications can support human endeavors though touch and gesture interactions.  

For an overview of this topic, take a look at my 2011 post, written after a number of ugly encounters with user-unfriendly applications:  Why bother switching from GUI to NUI?  

For an in-depth look into the history of multi-touch, the wisdom of Bill Buxton is well-worth absorbing.  He's worked with all sorts of interfaces, and has been curating the history of multi-touch and gesture systems since 2007:


Multi-Touch Systems that I have Known and Loved
Bill Buxton, Microsoft Research, Updated 8/30/12



Even if you are not a designer or developer, I encourage you to explore some of the links below:

Touch Gestures for Application Design
Luke Wroblewski, 10/9/12

Common Misconceptions About Touch
Steven Hoober, 3/18/13

Designing With Tablets in Mind:  Six Tips to Remember
Connor Turnbull, Webdesign tuts+, 9/27/11

Finger-Friendly Design: IDeal Mobile Touchscreen Target Sizes
Anthony T, Smashing Magazine, 2/21/12

Best Practices: Designing Touch Tablet Experiences for Preschoolers (pdf)
Sesame Street Workshop


Are Touch Screens Accessible?
AcessIT, National center on Accessible Information Technology in Education

iOS Human Interface Guidelines
Apple

Android User Interface Guidelines
Using Touch Gestures
Handling Multi-Touch Gestures
Android

Designing for Tablets?  We're Here to Help!
Roman Nurik, Android Developers Blog 11/26/12

Touch interaction design (Windows Store apps)
Microsoft - MSDN

Multi-Touch Systems that I have Known and Loved
Bill Buxton, Microsoft Research, Updated 8/30/12

Keepin' It Real: Organic, Natural, or Reality-Based Interactions? Video of TEI'12 panel discussion: Bill Buxton, Rob Jacob, Mili John Tharakan, and Roel Vertegaal (and more from TEI '12)

TEI '12 is finishing up in Kingston, Ontario.  If you missed it, like I did, don't worry.  Many of the presentations were streamed live, and are available to watch at your convenience.  
(The panel gets started at about 1: 48 on the timeline)


Keepin' It Real: Organic, Natural or Reality-Based Interactions?
Video streaming by Ustream Moderator: Amanda Parkes, Columbia University 
Panelists
William Buxton, Microsoft Research
Rob Jacob, Tufts University 
Mili John Tharakan, The Swedish School of Textiles 
Roel Vertegaal, Queen's University


RELATED
Here are some TEI '12 presentations interest me!
Process Pad: A Low-Cost Multi-Touch Platform to Facilitate Multimodal Documentation of Complex Learning
Shima Salehi, Stanford University
Jain Kim, Stanford University
Colin Meltzer, Stanford University
Paulo Blikstein, Stanford University
Process Pad project website


Exploring peripheral interaction design for primary school teachers
Saskia Bakker Eindhoven University of Technology, the Netherlands 
Elise van den Hoven Eindhoven University of Technology, the Netherlands 
Berry Eggen Eindhoven University of Technology, the Netherlands 
Kees Overbeeke Eindhoven University of Technology, the Netherlands 


Others:
Exploring Tabletops as an Effective Tool to Foster Creativity Traits
Mobile Lorm Glove - Introducing a communication device for deaf-blind people
Mobile Lorm Glove project website
The HapticTouch Toolkit: Enabling Exploration of Haptic Interactions

HapticTouch Toolkit from Haptic Touch on Vimeo.

CHI 2011, Bill Buxton, and the Buxton Collection: Explore 35 years of interactive devices, online!

Bill Buxton is a researcher at Microsoft who focuses on Human-Computer Interaction and is known for his work in user experience design and natural user interfaces such as multi-touch surfaces.   His talk at the recent CHI 2011 conference, held in Vancouver, Canada, was an overview of the Buxton Collection, an on-line historical archive of interactive input devices spanning over the past 35 years.  


It was interesting to note that at the time of the presentation, the Vancouver Conference Center, where the conference was taking place, was having serious problems with the network/internet connections, and as a consequence, Buxton was not able to demonstrate the on-line version of his collection as planned.  

Not to worry.  The physical version of the Buxton's archive was on display during the conference, along with Buxton, who was happy to tell the story behind every device and gadget in the archive, with much enthusiasm. The slideshow below provides a glimpse of the Bill Buxton archive displayed at CHI 2011:

My Buxton Collection Slideshow, CHI 2011, Vancouver, Canada

Buxton's archive of gadgets comes with a rich history, accumulated over the years. Much of this history, until now, has resided in Buxton's head.   Holding and touching the items in the archive while listening to Buxton's passionate stories about each one, was unlike anything I had ever experienced.  His archive is a labor of love, and the interactive, on-line version of the Buxton Collection is his way of sharing his knowledge with the world.


During his talk, Buxton pointed out that in computer science programs, students are not required to have much exposure to the "history of ideas" in the field.  Huge chunks of work are often ignored in the literature,  and in some cases, the wheel is unknowingly reinvented, and this is something that must be addressed within the CHI community, according to Buxton. 


I agree.

RELATED
Previous IMT posts about Bill Buxton
Bill Buxton's Presentation Video: "A Little Tale about Touch" (Microsoft Worldwide Partner Conference, 2010)
Two good articles by Bill Buxton: The Mad Dash Towards Touch Technology; The Long Nose of Innovation
Buxton Collection
Buxton Collection, PivotViewer
Back to the Past: Bill Buxton Shows Off 36 Years of Tech Devices
Microsoft News Center, 5/9/11
Microsoft's Bill Buxton exhibits gadget collection 35 years in the making
Donald Melanson, engadget, 5/9/11
Bill Buxton's Haptic Input References (pdf)
Bill Buxton's website
Multi-touch Systems that I Have Known and Loved (Bill Buxton)
CES 2010: NUI with Bill Buxton

On Engineering and Design: An Open Letter Microsoft Research Principal Scientist Bill Buxton calls for engineers and user experience designers to learn to appreciate one another

Bill Buxton, Bloomberg Businessweek, 4/29/09

Update from #CHI 2011 - much more to come.

I've been attending CHI 2011 for the past few days and have been impressed by the depth and breadth of the conference.  In addition to attending a number of interesting, high-quality presentations, I had the chance to interview several key contributors to the HCI field, explore a variety of awesome interactive applications,  and participate in an interactive musical multimedia performance.  Below is a video taken of part of the performance, taken by someone at the conference:


I also had a chance to explore Bill Buxton's exhibit of his historical archive of input and interactive devices.  To see the interactive online version of the exhibit, take a look at  Buxton Collection.  You won't be disappointed.

I will be sharing my "take aways" from the conference in a series of posts on this blog over the next couple of weeks. I meantime,  take a look at some of the CHI "Madness" videos below. The video clips are about 20-seconds long and are created by CHI presenters as previews to help conference attendees plan their schedules, since there are many sessions that are held concurrently. 

Gamification: Using Game Design Elements in Non-Gaming Contexts


CHI 2011 Accessible Games SIG




Games for your cat:


Don't forget Grandma!

CHI 2011! - Today's Highlights from the ACM CHI Conference on Human Factors in Computing Systems

I'm happy to share that I was approved for a press pass for the CHI 2011  conference I'm attending in Vancouver, Canada.   (I was one of the organizers for a pre-conference workshop held on Saturday, "Child Computer Interaction: Workshop on UI Technologies and Educational Pedagogy" and will share more information from the workshop in future blog posts.)


The press pass will provide me the opportunity to meet a number of key thinkers/researchers related to the field of Human Factors/Human Computer Interaction.  Here are a few of today's press highlights:


A conversation with ...Howard Rheingold
CHI 2011’s opening session features Howard Rheingold, an acknowledged authority on mobile communications, discussing the evolution and future of social media in teaching and learning. Rheingold, the author of Smart Mobs, innovator, and Stanford University lecturer, explains, "use of social media in higher education teaching sessions can provide opportunities for innovative and meaningful interactions that extend far beyond the traditional face-to-face classroom experience." -CHI 2011


A conversation with...the "PaperPhone Team"
Hang up the iPhone - here comes the paper phone: Bending the device will allow users to make calls and play music Gillian Shaw, Vancouver Sun, 5/5/11
UltraThin 'PaperPhone' bends to user's will Emily Chung, CBC News, 5/5/11


An Interview with Conference Organizers and Interaction Researchers ...
Desney Tan, Conference Chair and Microsoft Research
Bo Begole, Technical Program Co-Char and XEROX Palo Alto Research Center (PARC)
Wendy Kellogg, Technical Program Co-Chair and IBM


I can't wait until 4:00 PM - a special tour of the interactive demonstrations - here is the information I received from the CHI 2011 PR coordinator:



SPECIAL PRESS PREVIEW of the Interactive Demonstrations 4:00 - 5:30 PM  INTERACTIVITY PRESS TOUR
"This event will be highly visual & interactive http://chi2011.org/program/interactivity.html Press Release: Interactive Technologies Showcased at ACM Conference on Computer-Human Interaction http://chi2011.org/press/presskit/interactivity.html"


I'm sure this will be interesting:
Collecting the History of Interactive Devices
"An exhibit of interactive devices from the past 35 years that point to ways of interacting with computers in the future. First shown at the Vancouver Art Gallery in 2004 (and updated for 2011), the collection documents the history of pen computing, pointing devices, touch technologies, and illustrates the nature of how new technologies emerge." - CHI 2011


Stay updated!
To keep up with my blog posts during the CHI 2011 conference and afterwards, sign up for the RSS feed.  Another option is to follow me on Twitter,  and you'll receive an automatic "tweet" with a link whenever I post something on this blog. 

Bill Buxton's Presentation Video: "A Little Tale about Touch" (Microsoft Worldwide Partner Conference 2010)

I am always inspired by Bill Buxton's words of wisdom.  If you haven't heard of him, know that he's been around for a very long time, coming to the world of computer science and IT through his passion for music. According to his on-line biography, "Bill Buxton is a relentless advocate for innovation, design, and - especially - the appropriate consideration of human values, capacity, and culture in the conception, implementation, and use of new products and technologies. This is reflected in his research, teaching, talks, and writing - including his column on design and innovation for BusinessWeek.com, and his 2007 book, Sketching User Experiences."

A Little Tale about Touch "It's about the physical and the social context where it is happening" (Bill Buxton)


RELATED
“The only true voyage of discovery is not to go to new places, but to see the world through different eyes”.  - Bill Buxton, quoting Proust (1913, Remembrance of Things Past)

Bill Buxton: After the Show (Interviewed by Caroline Goles, Worldwide Partner Group Business Manager)


Bill Buxton's Website
Buxton, William. (1994) The three mirrors of interaction: a holistic approach to user interfaces. In L.W. MacDonald & J. Vince (Eds.) Interacting with virtual environments. New York: Wiley.

Here is a quote from the above reference:

"The thesis of this chapter is that we should consider technology in terms of the fidelity with which it reflects human capabilities on three levels:


* physical: how we are built and what motor/sensory skills we possess;

* cognitive: how we think, learn, solve problems and what cognitive skills we possess;

* social: how we relate to our social milieu, including group structure and dynamics, power, politics, and what social skills we possess.

Our metaphor is one of three separate mirrors, each reflecting one of these levels. In order to be judged acceptable, designs must provide an acceptable degree of fidelity in how they reflect each of these three aspects of human makeup and activity. The benefit is in how the model can provide a simple but valuable test that can be used during the design process. We now look at each of these mirrors in turn."

Two good articles by Bill Buxton: The Mad Dash Towards Touch Technology; The Long Nose of Innovation

I came a couple of interesting links to a couple of articles from the Putting People First blog. The links are articles written by Microsoft Research principal scientist, Bill Buxton.  If you've never heard of Bill Buxton, he's the guy that was doing multi-touch research way back in the 1980's. 

The Mad Dash Toward Touch Technology
Bill Buxton, Business Week, 10/21/09
"True innovators need to know as much about when, why, and how not to use trendy technology as when to use i."

The Long Nose of Innovation
Bill Buxton,  Business Week, 1/2/08
"The bulk of innovation is low-amplitude and takes place over a long period. Companies should focus on refining existing technologies as much as on creation." 

RELATED
Updated!

Multi-Touch Systems that I Have Known and Loved
(Bill Buxton)


I came across Bill Buxton's Multi-Touch website in early 2007 when I was taking HCI and Ubicomp.  I was searching for information about large touch-screen displays and applications for a couple of class projects.  The website was the answer to my graduate student prayers.  On the site, you'll find a fantastic overview of the history of "multi-touch", including gesture recognition and related surface technologies. 

The website has interesting links.  If you have the time, take a look at Buxton's main websitehttp://www.billbuxton.com/. You'll find loads of interesting links. I especially like the links to his Business Week articles.



Bill Buxton is the author of "Sketching User Experiences:  Getting the design right and the right design", a book that I own and recommend.

Good news about the San Jose Interactive Displays Conference (that I couldn't attend)

Sadly, I was unable to attend the Interactive Displays conference that was held last week in San Jose, California. Jeff Han, the guy in the video clip in my sidebar, was one of the keynote presenters. I really wish I could have seen - and touched - the action!

Update (4/27/09) from Thomas Hansen, a member of NUI Group, who attended IDC:

"..It was certainly an interesting event. I got to talk (albeit briefly) to both Andy Wilson and Jeff Han. Both of whom produce very inspiring and world class UI/HCI research on a consistent basis. Further, it was a great honor to meet some of the other members of the nuigroup community in person, some of whom where inspiring not only because of their amazing intellect and artistic talents, but especially due to their friendliness, benevolence, and maturity."

Here are a few excerpts from people who were fortunate to attend and then write about the experience:

Putting our arms around the future of touch Ina Fried , CNET 4/23/09

"..if you used one of the interactive displays here to show a heat map of this industry, it would glow red hot. That's because touch displays, for years relegated to kiosks and industrial uses, are quickly becoming mainstream. Hewlett-Packard and Dell already have touch-capable machines, while Microsoft is set to make gesture input standard with Windows 7...For his own part, Han said he was inspired by seeing a PBS documentary in the early 1980s that showed Microsoft researcher Bill Buxton, then at the University of Toronto, using multitouch to compose music on a computer. The computer itself was a green screen with an ancient processor and little memory, but the key underlying concept was already there..

"Sometimes it takes that long for these things to marinate and gestate," Han said....And while things are now taking off, Han urged the crowd not to forsake quality in the rush to take advantage of a hot market. "That will ruin it and mess it up for all of us, and that would be a real shame," Han said."

Interactive Displays Conference Highlights Kevin Arthur, Touch Usability 4/22/09

"...He (Jeff Han) showed a great clip from an early 80s TV show called Bits and Bytes that featured a young Bill Buxton demonstrating some of his tablet work at the University of Toronto...Jeff Han's point in showing this clip was not just to share what inspired him as a kid to pursue computer science as a career. He also wanted to make the point that none of this stuff is really new. He urged the audience not to jump on whatever tech is cool this week, but to be aware of the history and to do the research. Be thoughtful and careful about what you're making. He said one of his fears with the multitouch craze is that the waters will be poisoned by bad and poorly conceived implementations. He said "don't add noise" to the ecosystem by using terms sloppily -- like "multitouch"...

The importance of being more thoughtful and mindful of prior work are not exactly new to most of us with design, HCI, or CS backgrounds, but the audience here is largely made up of marketing or other business types, I believe, who sometimes tend to get a bit carried away, you might say. I mean no disrespect to my friends in marketing..."

Interactive Displays Conference San Jose Harry van der Veen, 4/25/09
" Big thumbs up for Pira tech for managing to get so many multi-touch industry professionals (Wacom, Mindstorm, NextWindow, 3M, Jeff Han (Perceptive Pixel), LG, Tyco Electronics, Stantum and more) and hobbyists together."

Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics

Put-That-There: Voice and Gesture at the Graphics Interface and more Blasts from the 1980's HCI Past

bigkif's information about "Put-That-There" about Put-That-There gives a good description of this video:

Put-That-There at CHI '84

"In 1980, Richard A. Bolt from MIT wrote Put-that-there : voice and gesture at the graphics interface. It was a pioneering multimodal application that combined speech and gesture recognition.

This demo shows users commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference."

Richard A. Bolt "Put-That-There": Voice and Gesture at the Graphics Interface (pdf) SIGGRAPH '80

Here is another blast from the '80's:

Kankaanpaa A. FIDS- AFlat-Panel Interactive Display System IEEE March 1988 IEEE Computer Graphics Applications(Nokia Information Systems)

"Although the needs and expectations of these various users are very diverse, they all have a common requirement: more natural and easier methods for communicating with the computer than are available today. Furthermore, they do not want to interact with the computer; they want to communicate with the application they are using. They do not want to use computer jargon; they want to use the same natural methods that they use when they perform the same tasks without a computer."

“We believe that only three of the flat-panel technologies described above, namely LCD, EL, and plasma, will be sufficiently advanced for mass production within this decade.”

Bill Buxton was working on multi-touch and gesture interaction in the 1980's, but his dreams did not become a reality until this century, for a variety of reasons. He shared his thoughts about the paradox of the speed of technology in a presentation at the 2008 IEEE International Solid-State Circuits Conference:Surface and Tangible Computing, and the “Small” Matter of People and Design”(pdf)

‘Carrying on from an earlier thesis in our department (Mehta , 1982) , we built a tablet that was sensitive to simultaneous touches at multiple locations, and with the ability to sense the degree of each touch independently (Lee, Buxton & Smith, 1984). We stopped the work in late 1984 when I saw a much better implementation at Bell Labs – one that was transparent and mounted over a CRT. The problem was that they never released the technology, so, the whole multi-touch venture went dormant for 20 years. But, I never stopped dreaming about it. (Lesson: don’t stop your research just because someone else is way ahead of you. It might be transitory, and anyhow, remember the story of the tortoise and the hare.)

“I spoke earlier about the paradox in the speed of technology development it goes at rocket speed, but that of a glacier as well; Simultaneously! In the perfect world, this would be ideal: we could go through several iterations of ideas so that by the time the new paradigms of interaction, such as Surface and Tangible computing are ready for prime time, everything will be in place. But, the rapid iteration is more directed at supporting the old paradigms faster and cheaper, rather then helping shape the new ones. The reasons are not hard to understand. From the perspective of circuit design, the problems are really hard. So, one has to have one’s head down working flat out to get anything done. But, there is a side of me that motivated this paper that asks, If it is so hard, then isn’t it worth making sure that the things one is working on are things that are worthy of one’s hard-earned skills?”

SOMEWHAT RELATED

Bill Buxton's Haptic Input References (pdf)

About: Programming for interactive multimedia applications- WPF, Silverlight, EduSim, NeuroVR..

Part I Musings: Learning about application development and programming at mid-life:

I thought I'd write on a more personal level this time.

I'm a school psychologist, so in 2003, my motivation for taking computers at mid-life stemmed from my desire to create engaging interactive multimedia games for learning, games that could be played on hand-held devices as well as on the interactive whiteboards that I noticed were inching into my schools.

It was difficult for me to figure out how to get from Point A to Point B.

I shouldn't have been shocked to learn that most introductory programming classes provide instruction, as well as endless lab assignments, that are geared for people who want to make business forms and manipulate business-related data, build e-commerce websites, or create relational databases for... banks!

I now can make a mortgage calculator forms that adjust for various scenarios and provide cute error messages, in beginning Visual Basic.Net, C#, and Java. I can create a database that will let users look up part numbers for all sorts of widgets, in all sorts of combinations, and ensure that client data can be easily accessed in a nice looking form.

Why should I learn all of the old stuff when there are so many new avenues to explore?

Over the past few years, I've been fortunate to take a variety of classes that were not readily available just 8-10 years ago:

Computer/Internet Multimedia. Computer Music Technology. Game Design/Development. AI for Games. Ubiquitous Computing. Web Development Tools. Virtual Reality for Education and Training. These courses have motivated me to learn more about programming. The traditional programming courses had the opposite effect.


PART II Keeping up

I recently attended a day-long code camp at Central Piedmont Community College to learn more about Windows Presentation Foundation (WPF) and Silverlight. WPF and Silverlight provide the foundation for applications that can run on Microsoft's Surface. WPF and Silverlight provide designers and developers with more efficient ways of developing visualization and interactive multimedia applications.

The architecture behind WPF and Silverlight seems to align more with the way people think and envision, which in my opinion, gives hope for those of us who have toyed with the idea of programming, but were frightened off the first time they opened up a traditional programming textbook.


Part III Visualization and Interactive Multimedia

For more information about WPF, Silverlight, design, etc, read Sam Batterman's (a Microsoft Evangalist) recent blog post: Some thoughts about WPF and Data Visualization

Here are some pictures and text that I lifted from Sam's blog that will give you a picture of what I'm talking about:

"Here's an application that one of our partners built in a few weeks - all WPF and actually, not complicated code...getting that heart rendered was probably less than 100 lines of code. This app is used in a hospital for documenting heart surgery procedures. You can draw and annotate the 3D surface, rotate the heart, etc."





Can you imagine how kids would love to manipulate something like this in a science class?! This would be great on an interactive whiteboard or display.

This focuses on user experience, much more so than applications in the past. For more information about designing for user experience, take a look at the link to Bill Buxton's book, "Sketching User Experiences."

Bill Buxton's webiste, "Multi-touch systems that I have known and loved" is a good resource for those of you who'd like to get a better picture of multi-touch systems and interaction.

(I'll write more about WPF and Silverlight after I get my laptop repaired and have a chance to experiment some more.)

Croquet and EduSim

Right now, I've been experimenting with Croquet, which uses Squeak, and EduSim, which is powered by Croquet, to put together some learning activities for students.The best part is that Croquet is open-source, and all of the EduSim applications are free.

Here are some pictures that link to short video clips from the Greenbush EduSim website:






Neuro VR

Another application that I'm working with is NeuroVR.

NeuroVR is a free virtual environment that was designed for use in clinical settings. If you don't have access to virtual-reality hardware, you can still use NeuroVR on a desktop or large-screen display. Available 3D environments include an office, a supermarket, a park, a classroom, a poolside setting, and a home.

Andrea Gaggioli, Ph.D., is the Chief Technical Officer of the this project. He's also behind the Positive Technology blog, which is a great resource. Andrea is a researcher working at the intersections of psychology, neuroscience, and emerging technologies.

NeuroVR allows the clinician (or educator) to easily insert pictures, objects, and videoclips into each virtual world. Doors open and close, and you can move items around in the environment. For example, fruit set on a table can be moved over to a counter.

In my opinion, NeuroVR has potential not only in the area of rehabilitation and therapy, but in special education as well, particularly for students who have multiple special needs, including severe autism. A variety of "how-to" videos are posted on DaevornLi's YouTube channel.


Here are some videos clips to give you a better picture of the application:

About Interaction: Bill Buxton and Multi-Touch Systems

Based on data from SiteMeter, there are a number of my readers who are searching for information about touch screen interaction design and multi-touch systems. A great resource for anyone interested in these topics is Bill Buxton's website, "Multi-Touch Systems that I have Known and Loved".

Bill provides a good overview of the various flavors of multi-touch interaction and interfaces, along with a an annotated chronology of multi-touch and related technologies, starting before 1982. The chronology includes some interesting pictures.

Bill Buxton works for Microsoft Research. The following information is from his bio:

"Bill Buxton is a designer and a researcher concerned with human aspects of technology. His work reflects a particular interest in the use of technology to support creative activities such as design, film making and music. Buxton's research specialties include technologies, techniques and theories of input to computers, technology mediated human-human collaboration, and ubiquitous computing."