Pervasive Checkers on Microsoft's Surface: The Gamepack Video

In early 2007, before we knew of the existence of Microsoft's multi-touch surface, I worked on a "Pervasive Checkers" project with Johnny Hopkins, a classmate in my Ubiquitous Computing class. I'd previously worked with XNA Game Studio Express in an AI for Games class, and thought that it would be cool to make a casual checkers game on a multi-touch table that could be played in gathering spots such as coffee houses and neighborhood cafes.

Below is a screen shot of what we created using Inspiration software - (in the application, you can click on an item and it expands to reveal additional information.)

Two years later, and the Pervasive Checkers idea is reality - but I wasn't involved in the process. Checkers is one of the games that is included in a game-pack created specifically for the Surface.

Take a look at the video:



From Surface Computer News:

"The Microsoft Surface Games Pack is a clear illustration of where the Natural User Interface of Windows 7 has the potential to take games. Windows is traditionally the number 1 gaming platform around the world. With the introduction of the NUI, allowing players to literally have titles at their fingertips via touch, Windows 7 can kick the gaming experience up a notch. Provided that developers rise up to the challenge."

Interactive Displays in Public Spaces

Daniel Michelis recently completed his Ph.D. dissertation on a topic that is dear to my heart. Information about his research can be found on his Interactive Displays in Public Spaces blog.

(Note: This was cross posted on the Technology-Supported Human World Interaction blog.)

Here are links to a few of his posts:

Interactive Displays: Perception, Awareness, and Interaction

Evaluating Interaction with Display Applications in Public Space

I especially like the diagrams Daniel uses to depict zones of interaction:

Figure 3: Four-phase Model
(Source: Daniel Michelis (2009), according to: Vogel and Balakrishnan, 2004)

(Author: Daniel Michelis, Institute for Media and Communications Management, University of St. Gallen)

Figure 1: Perception and Usage of Interactive Displays
(Source: Daniel Michelis (2009), according to: Brignull & Rogers, 2003)

Interactive Displays 2009: Listen to a podcast interview each day leading up to the conference.

The following people were on the podcast on 4/13/09:
Steven Bathiche- Microsoft
Jerry Bertrand- Microscent, LLC
Henry Kaufman- Tactable

Interactive Displays 2009

Note: Jeff Han will be one of the presenters.

More info to come...

Dreams about haptic overlays: Part Two- SLAP: Bridging the Gap Between Virtual and Physical Controls on Tabletops

In response to my previous post about my dream about haptic touch-screen overlays,
Martin Kaltenbrunner pointed me to an interesting link SLAP: Silicone Illuminated Active Peripherals "Bridging the Gap Between Virtual and Physical Controls on Tabletops". SLAP is a project of the Media Computing Group at RWTH Aachen University, headed by Dr. Jan Borchers.



FYI - Information from the Media Computing Group website:

• Grounded in Computer Science, we develop and study new interaction theories, techniques, and systems in the areas of interaction with multimedia, ubiquitous computing environments, tangible user interfaces, and HCI design patterns. Our goal is to make the Brave New World of ubiquitous multimedia technologies useful by making it usable. A recent overview of our research approach and research directions can be found here.

SLAP: The Movies




Here are some pictures and related information from the SLAP website:

• CHI 2009 Video on YouTube or as QuickTime video (21 MB).
• Presentation of our SLAP demo at TEI 2009 (YouTube)

Publications

2009

	Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In CHI '09: Proceeding of the twenty-seventh annual SIGCHI conference on Human factors in computing systems, New York, NY, USA, 2009. ACM. .
	Malte Weiss, Julie Wagner, Roger Jennings, Yvonne Jansen, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In CHI '09: Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2009. ACM Press. .
	Malte Weiss, Julie Wagner, Roger Jennings, Yvonne Jansen, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAPbook: Tangible Widgets on Multi-touch Tables in Groupware Environments. In TEI '09: Proceedings of the 3rd international conference on Tangible and embedded interaction, New York, NY, USA, 2009. ACM. .

2008

Malte Weiss, Roger Jennings, Julie Wagner, James D. Hollan, Ramsin Khoshabeh, and Jan Borchers. SLAP: Silicone Illuminated Active Peripherals. In Extended Abstracts of Tabletop '08, 2008. .

MORE FYI:

(Dr. Jan Borchers, the head of the Media Computing Group, recommends the following list of books, and I agree with his list, 100%.)

Dr. Jan Borchers' (Annotated) Top Ten List of Books on Human-Computer Interaction:

Note: I have read many of these books and I agree that they should be required reading anyone who plans to design, develop, program, or implement anything related to technology and people.

1. Alan Dix, Janet Finlay, Gregory D. Abowd, and Russell Beale: Human-Computer Interaction, 3rd ed., Prentice Hall, 2004. Currently the best, most well-rounded book I know to teach introductory HCI if you need to limit yourself to a single title. Technical enough, good breadth, not too fuzzy for a CS curriculum, very current, with a web site that includes resources such as sample programs, slides, etc.
   
   
2. Ben Shneiderman and Catherine Plaisant: Designing The User Interface, 4th ed., Pearson Addison-Wesley, 2004. Best overall reference book for all areas of HCI, providing an introduction and great up-to-date pointers to most sub-fields of HCI research and practice, especially different interaction techniques. His Golden Rules of User Interface Design and sample questionnaires for user testing are very useful in an introductory class. Unfortunately, the companion web site costs money after an initial trial period.
   
   
3. Donald A. Norman, The Design Of Everyday Things, Basic Books, 2002. A classic text from 1988 with an updated introduction that, while some of the technologies described or envisioned seem somewhat outdated now, still provides the best introduction to the spirit of good human-centered design. A not too technical read with hilarious stories of badly designed everyday technology, it provides some very useful basic models for human cognition, such as the Seven Stages of Action. This book also introduced the fundamental concept of affordances to HCI. Changed my view of the world of technology around me, and is probably the best initial brainwash for engineering students to "get" user-centered design.
   
   
4. Jenny Preece, Yvonne Rogers, and Helen Sharp: Interaction Design, 2nd ed., Wiley, 2007. This title focuses more on the process of designing good user interfaces, and is less technical, but excellent and up-to-date in the area it addresses. The companion web site has slides, case studies, and other materials.
   
   
5. Bill Moggridge, Designing Interactions, MIT Press, 2008. A truly beautiful "coffee-table style" book on interaction design, also covering product and industrial design of digital technology (Moggridge is a founder of IDEO). It has wonderful short essays about seminal digial product designs, from Engelbart's mouse, to the Mac and Palm, to Google and other internet services, as well as articles on digital product design theory. My own Sweet Sports and Baroque Technology article was based on one of the theory articles. Special treat: video interviews and chapters are available for free, on a weekly rotation, at http://www.designinginteractions.com/.
   
   
6. Bill Buxton, Sketching User Experiences, Elsevier, 2007. Similar to Moggridge's book in style, this book focuses on the early stages of product design. It also includes very interesting stories of key interactive products, such as Apple's iPod. And of course it's written by one of the long-time key players in HCI. More at http://www.billbuxton.com/.
   
   
7. Terry Winograd (ed.): Bringing Design to Software, Addison-Wesley, 1996. An excellent and very well edited collection of contributions from key players in HCI, from Kapor's Software Design Manifesto to Rheinfrank's Design Languages. Its particular value also comes from the profiles that link chapters and give an insider's view of how some of the most seminal UI designs came to be, from the Xerox Star to VisiCalc and HyperCard. Terry has some information about his book at http://hci.stanford.edu/bds/, and I used it with great success when I had the fortunate opportunity to teach an introductory HCI class in his program at Stanford in 2002.
   
   
8. Brenda Laurel (ed.): The Art of Human-Computer Interaction, Addison-Wesley, 1990. While ancient by today's standards, this book is another carefully compiled and very coherent collection of highly relevant articles on HCI by some of the most influential people in the field. I particularly like the article by Scott Kim on interdisciplinary design, and Tom Erickson's chapter.
   
   
9. Apple Computer: The Apple Software Design Guidelines, latest edition 2005. OK, I'm a Mac head, but then many HCI people are because Apple has such an excellent sense of doing the right thing when it comes to user interface design. These guidelines have been around since the 90's, with several new editions since then, and especially Part I ("Application Design Fundamentals") contains excellent, system-independent, hands-on advice for anybody developing interactive software, especially desktop applications. And it's free! Apple's developer website has the latest version both online and as downloadable PDF. I often recommend this as a quick read for engineering types that just want the bare essentials to help avoid major UI design catastrophes.
   
   
10. Jef Raskin, The Humane Interface, Addison-Wesley, 2000. Similar to Norman's book above, but more recent and more technical, this is another good first read to start thinking about user interface design, written by the father of the original Apple Macintosh. Some of the ideas presented here are quite unusual, and that's intended. Some related materials, such as demos of his Zoomable User Interface and The Humane Environment are at http://www.jefraskin.com/.
    
    

"So that's my top 10 list. I may add some more in the future. But I figure it's more important to restrict myself to those books I think are really outstanding than bother you with additional titles that don't really have that special something....For a good current PhD-level HCI reading list that is based more on papers and individual chapters than single books, see Terry Winograd's HCI reading list at Stanford University." -Dr. Jan Borchers

Last night I dreamt about haptic touch-screen overlays...

Technology Dreams
I just haven't had enough time during the day to focus on my passion for interactive technology. I wish I could grab a picture from my dream about haptic overlays and share it here on my blog!   To make sure this technology wasn't only in my dreams, I decided to google "haptic overlays", and here are some of the results:

Future Directions for Tactile Feedback (Peter Odum, Idlemode, 1/23/09)
"Current tactile solutions fall short either in reconfigurability or in pre-interaction feedback...And it’s easy to provide a physical sensation after the user interacts, but not to provide buttons that can be physically felt *before* the interaction is committed."

Some of the work regarding haptic or tactile feedback has focused on smaller screens and virtual reality:
Vinyl overlay for tactile feel on iPhone NES emulator
More Details on Nokia Haptikos Tactile Touchscreens
Haptic Overlay Device for Flat Panel Touch Displays (pdf)
Touchscreen Feedback Overview
Team Daimler-Chrysler Touch (pdf)
The Twelfth Haptic Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems 2004
Patent: Method and apparatus for multi-touch tactile touch panel actuator mechanisms
Benali-Khoudja, M., Hafez, M., Alexandre, J., Kheddar, A. Tactile Interfaces: A State-of-the-Art Survey (pdf) ISR 2004, 35th International Symposium on Robotics

Here is the concept in the form of a "wearable" design:

The surface of this design looks very much like the overlay material that I was working with in my dream, except that the grid-sensors were the same translucent color as the overlay. (I was trying to program this overlay in my dream, but that is another story!)


Here is some technical information about these little haptic finger overlays- with a little tweaking, they could be transformed into something really cute...

"The key material to the display is an electroactive polymer that can stimulate the skin without using any additional electromechanical transmission. The polymer consists of eight layers of dielectric elastomer actuator films which have been sprayed with electrodes in a specific pattern. Along with a protective layer to separate the electrodes from the skin, the entire polymer sheet is about 210 micrometers thick. In their study, the researchers fabricated an 11 x 14 mm sheet with Velcro on the edges, and rolled it up in the shape of a thimble to be worn on the finger. The display can convey information to the wearer when the electrodes induce a voltage across the films. A voltage causes the films to compress down and expand outward. In doing so, the films put pressure on the wearer’s skin, inducing a “mild sensation.” Like most polymers, the device is hyper-elastic, meaning that it can experience large amounts of elastic strain and recover its original shape.This simple stimulation mechanism, which doesn’t require complex electronics, is one of the greatest advantages of the soft tactile display compared with current displays. Its other benefits include efficient power usage, cost-effectiveness, and easy fabrication. As the researchers note, the new display has lower power and displacement values than is considered optimal, which may limit its applications to specific areas"

The Search Continues...

This journey led me to Nokia's "Morph", a device using nano-materials, something I think I posted about a while ago in one of my blogs. (This video reminds my of some of my more vivid technology dreams. It must be nice to be paid to transform technology dreams into prototypes and demonstration videos!)

The Video:

"Launched alongside The Museum of Modern Art “Design and The Elastic Mind” exhibition, the Morph concept device is a bridge between highly advanced technologies and their potential benefits to end-users. This device concept showcases some revolutionary leaps being explored by Nokia Research Center (NRC) in collaboration with the Cambridge Nanoscience Centre (United Kingdom) – nanoscale technologies that will potentially create a world of radically different devices that open up an entirely new spectrum of possibilities."

More haptic/tactile feedback touch interaction:

Nokia's Haptikos Touch Screen Handheld Web Browser

Immersion's Touch Screen: Mimics feel of real buttons
How it works:

Immersion's Fitness Market Brief (pdf)

UPDATE 7/4/09
Apple is working on this sort of thing, in a way. See my recent post:
Haptic/Tactile Feedback for the iPhone? MacRumors says, "YES!"

Dreams about haptic overlays: Part Two: Bridging the gap between virtual and physical controls on tabletops.

Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics