Natural User Interface Surfaces: TouchCo - IFSR technology company, founded by NYU Media Reseach Lab's Ken Perlin and Ilya Rosenburg

Earlier this year, I had a very vivid dream about haptic touch-screen overlays. That's how things go for me sometimes. At any rate, the dream inspired a blog post, "Last night I dreamt about haptic touch-screen overlays...". Since this is one of my interests, I try to keep up with developments in this field when I can.

It is amusing to see how seemingly unrelated things have similar roots. Today I came across information about a cool mobile phone concept for the visually impaired, designed by Samsung. The touch technology involved in the concept phone incorporated something called Electric Activating Plastic, or EAP. 


This sparked my curiosity, and while searching for more information, I came across an article in MIT's Technology Review that caught my eye:


A Touch of Ingenuity: Inexpensive pressure-sensitive pad could make surfaces smarter
(Kate Green, MIT Technology Review September/October 2009)

UPDATE:  As of 2/2010, TouchCo no longer exists. It was bought by Amazon:

TouchCo Bought By Amazon 2/3/2010

"..Now that more and more smart phones and MP3 players have touch-screen interfaces, people have grown accustomed to interacting with gadgets using only taps and swipes of their fingers. But on the 11th floor of a downtown Manhattan building, New York University researchers Ilya Rosenberg and Ken Perlin are developing an interface that goes even further. It’s a thin pad that responds precisely to pressure from not only a finger but a range of objects, such as a foot, a stylus, or a drumstick. And it can sense multiple inputs at once."

Ken Perlin and Ilya Rosenberg developed this system as part of their human-computer interaction work at NYU's Media Research Lab, and with the success of their work, went on to start a new company, Touchco, to commercialize and market this concept, known as IFSR (Interpolating Force-Sensitive Resistance) technology.

"The IFSR sensor revolutionizes the field of multi-touch human-computer interaction by out-performing existing solutions in both price and capabilities. Unlike traditional capacitive sensors, our patent pending system can detect any object - not just a finger - and can determine how much pressure is being applied to every point on a sensor simultaneously. IFSR sensors are natively multi-touch, use less power than capacitive sensors, and are much less expensive to produce, making them a highly disruptive technology with widespread market applications."


The Touchco website has a wealth of information regarding this technology. (Update: The TouchCo website was taken down in February 2010)

Other members of Touchco include Nadim Awad, Fang Cheng, Julien Beguin, and Christopher Nam.

SOMEWHAT RELATED
I know of Ken Perlin through his Games for Learning work, and also through his on-line application, Responsive Face, which I use in my work with students who have autism.




Ken Perlin's Blog
Games for Learning Institute

TechPsych Post:
Samsung Braille TouchPhone Prototype is Cool: Uses Electric Active Plastic for functional tactile feedback.

Dreams about haptic overlays: Part Two- SLAP: Bridging the Gap Between Virtual and Physical Controls on Tabletops

In response to my previous post about my dream about haptic touch-screen overlays,
Martin Kaltenbrunner pointed me to an interesting link SLAP: Silicone Illuminated Active Peripherals "Bridging the Gap Between Virtual and Physical Controls on Tabletops". SLAP is a project of the Media Computing Group at RWTH Aachen University, headed by Dr. Jan Borchers.



FYI - Information from the Media Computing Group website:

• Grounded in Computer Science, we develop and study new interaction theories, techniques, and systems in the areas of interaction with multimedia, ubiquitous computing environments, tangible user interfaces, and HCI design patterns. Our goal is to make the Brave New World of ubiquitous multimedia technologies useful by making it usable. A recent overview of our research approach and research directions can be found here.

SLAP: The Movies




Here are some pictures and related information from the SLAP website:

• CHI 2009 Video on YouTube or as QuickTime video (21 MB).
• Presentation of our SLAP demo at TEI 2009 (YouTube)

Publications

2009

	Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In CHI '09: Proceeding of the twenty-seventh annual SIGCHI conference on Human factors in computing systems, New York, NY, USA, 2009. ACM. .
	Malte Weiss, Julie Wagner, Roger Jennings, Yvonne Jansen, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In CHI '09: Extended Abstracts on Human Factors in Computing Systems, New York, NY, USA, 2009. ACM Press. .
	Malte Weiss, Julie Wagner, Roger Jennings, Yvonne Jansen, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. SLAPbook: Tangible Widgets on Multi-touch Tables in Groupware Environments. In TEI '09: Proceedings of the 3rd international conference on Tangible and embedded interaction, New York, NY, USA, 2009. ACM. .

2008

Malte Weiss, Roger Jennings, Julie Wagner, James D. Hollan, Ramsin Khoshabeh, and Jan Borchers. SLAP: Silicone Illuminated Active Peripherals. In Extended Abstracts of Tabletop '08, 2008. .

MORE FYI:

(Dr. Jan Borchers, the head of the Media Computing Group, recommends the following list of books, and I agree with his list, 100%.)

Dr. Jan Borchers' (Annotated) Top Ten List of Books on Human-Computer Interaction:

Note: I have read many of these books and I agree that they should be required reading anyone who plans to design, develop, program, or implement anything related to technology and people.

1. Alan Dix, Janet Finlay, Gregory D. Abowd, and Russell Beale: Human-Computer Interaction, 3rd ed., Prentice Hall, 2004. Currently the best, most well-rounded book I know to teach introductory HCI if you need to limit yourself to a single title. Technical enough, good breadth, not too fuzzy for a CS curriculum, very current, with a web site that includes resources such as sample programs, slides, etc.
   
   
2. Ben Shneiderman and Catherine Plaisant: Designing The User Interface, 4th ed., Pearson Addison-Wesley, 2004. Best overall reference book for all areas of HCI, providing an introduction and great up-to-date pointers to most sub-fields of HCI research and practice, especially different interaction techniques. His Golden Rules of User Interface Design and sample questionnaires for user testing are very useful in an introductory class. Unfortunately, the companion web site costs money after an initial trial period.
   
   
3. Donald A. Norman, The Design Of Everyday Things, Basic Books, 2002. A classic text from 1988 with an updated introduction that, while some of the technologies described or envisioned seem somewhat outdated now, still provides the best introduction to the spirit of good human-centered design. A not too technical read with hilarious stories of badly designed everyday technology, it provides some very useful basic models for human cognition, such as the Seven Stages of Action. This book also introduced the fundamental concept of affordances to HCI. Changed my view of the world of technology around me, and is probably the best initial brainwash for engineering students to "get" user-centered design.
   
   
4. Jenny Preece, Yvonne Rogers, and Helen Sharp: Interaction Design, 2nd ed., Wiley, 2007. This title focuses more on the process of designing good user interfaces, and is less technical, but excellent and up-to-date in the area it addresses. The companion web site has slides, case studies, and other materials.
   
   
5. Bill Moggridge, Designing Interactions, MIT Press, 2008. A truly beautiful "coffee-table style" book on interaction design, also covering product and industrial design of digital technology (Moggridge is a founder of IDEO). It has wonderful short essays about seminal digial product designs, from Engelbart's mouse, to the Mac and Palm, to Google and other internet services, as well as articles on digital product design theory. My own Sweet Sports and Baroque Technology article was based on one of the theory articles. Special treat: video interviews and chapters are available for free, on a weekly rotation, at http://www.designinginteractions.com/.
   
   
6. Bill Buxton, Sketching User Experiences, Elsevier, 2007. Similar to Moggridge's book in style, this book focuses on the early stages of product design. It also includes very interesting stories of key interactive products, such as Apple's iPod. And of course it's written by one of the long-time key players in HCI. More at http://www.billbuxton.com/.
   
   
7. Terry Winograd (ed.): Bringing Design to Software, Addison-Wesley, 1996. An excellent and very well edited collection of contributions from key players in HCI, from Kapor's Software Design Manifesto to Rheinfrank's Design Languages. Its particular value also comes from the profiles that link chapters and give an insider's view of how some of the most seminal UI designs came to be, from the Xerox Star to VisiCalc and HyperCard. Terry has some information about his book at http://hci.stanford.edu/bds/, and I used it with great success when I had the fortunate opportunity to teach an introductory HCI class in his program at Stanford in 2002.
   
   
8. Brenda Laurel (ed.): The Art of Human-Computer Interaction, Addison-Wesley, 1990. While ancient by today's standards, this book is another carefully compiled and very coherent collection of highly relevant articles on HCI by some of the most influential people in the field. I particularly like the article by Scott Kim on interdisciplinary design, and Tom Erickson's chapter.
   
   
9. Apple Computer: The Apple Software Design Guidelines, latest edition 2005. OK, I'm a Mac head, but then many HCI people are because Apple has such an excellent sense of doing the right thing when it comes to user interface design. These guidelines have been around since the 90's, with several new editions since then, and especially Part I ("Application Design Fundamentals") contains excellent, system-independent, hands-on advice for anybody developing interactive software, especially desktop applications. And it's free! Apple's developer website has the latest version both online and as downloadable PDF. I often recommend this as a quick read for engineering types that just want the bare essentials to help avoid major UI design catastrophes.
   
   
10. Jef Raskin, The Humane Interface, Addison-Wesley, 2000. Similar to Norman's book above, but more recent and more technical, this is another good first read to start thinking about user interface design, written by the father of the original Apple Macintosh. Some of the ideas presented here are quite unusual, and that's intended. Some related materials, such as demos of his Zoomable User Interface and The Humane Environment are at http://www.jefraskin.com/.
    
    

"So that's my top 10 list. I may add some more in the future. But I figure it's more important to restrict myself to those books I think are really outstanding than bother you with additional titles that don't really have that special something....For a good current PhD-level HCI reading list that is based more on papers and individual chapters than single books, see Terry Winograd's HCI reading list at Stanford University." -Dr. Jan Borchers

Last night I dreamt about haptic touch-screen overlays...

Technology Dreams
I just haven't had enough time during the day to focus on my passion for interactive technology. I wish I could grab a picture from my dream about haptic overlays and share it here on my blog!   To make sure this technology wasn't only in my dreams, I decided to google "haptic overlays", and here are some of the results:

Future Directions for Tactile Feedback (Peter Odum, Idlemode, 1/23/09)
"Current tactile solutions fall short either in reconfigurability or in pre-interaction feedback...And it’s easy to provide a physical sensation after the user interacts, but not to provide buttons that can be physically felt *before* the interaction is committed."

Some of the work regarding haptic or tactile feedback has focused on smaller screens and virtual reality:
Vinyl overlay for tactile feel on iPhone NES emulator
More Details on Nokia Haptikos Tactile Touchscreens
Haptic Overlay Device for Flat Panel Touch Displays (pdf)
Touchscreen Feedback Overview
Team Daimler-Chrysler Touch (pdf)
The Twelfth Haptic Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems 2004
Patent: Method and apparatus for multi-touch tactile touch panel actuator mechanisms
Benali-Khoudja, M., Hafez, M., Alexandre, J., Kheddar, A. Tactile Interfaces: A State-of-the-Art Survey (pdf) ISR 2004, 35th International Symposium on Robotics

Here is the concept in the form of a "wearable" design:

The surface of this design looks very much like the overlay material that I was working with in my dream, except that the grid-sensors were the same translucent color as the overlay. (I was trying to program this overlay in my dream, but that is another story!)


Here is some technical information about these little haptic finger overlays- with a little tweaking, they could be transformed into something really cute...

"The key material to the display is an electroactive polymer that can stimulate the skin without using any additional electromechanical transmission. The polymer consists of eight layers of dielectric elastomer actuator films which have been sprayed with electrodes in a specific pattern. Along with a protective layer to separate the electrodes from the skin, the entire polymer sheet is about 210 micrometers thick. In their study, the researchers fabricated an 11 x 14 mm sheet with Velcro on the edges, and rolled it up in the shape of a thimble to be worn on the finger. The display can convey information to the wearer when the electrodes induce a voltage across the films. A voltage causes the films to compress down and expand outward. In doing so, the films put pressure on the wearer’s skin, inducing a “mild sensation.” Like most polymers, the device is hyper-elastic, meaning that it can experience large amounts of elastic strain and recover its original shape.This simple stimulation mechanism, which doesn’t require complex electronics, is one of the greatest advantages of the soft tactile display compared with current displays. Its other benefits include efficient power usage, cost-effectiveness, and easy fabrication. As the researchers note, the new display has lower power and displacement values than is considered optimal, which may limit its applications to specific areas"

The Search Continues...

This journey led me to Nokia's "Morph", a device using nano-materials, something I think I posted about a while ago in one of my blogs. (This video reminds my of some of my more vivid technology dreams. It must be nice to be paid to transform technology dreams into prototypes and demonstration videos!)

The Video:

"Launched alongside The Museum of Modern Art “Design and The Elastic Mind” exhibition, the Morph concept device is a bridge between highly advanced technologies and their potential benefits to end-users. This device concept showcases some revolutionary leaps being explored by Nokia Research Center (NRC) in collaboration with the Cambridge Nanoscience Centre (United Kingdom) – nanoscale technologies that will potentially create a world of radically different devices that open up an entirely new spectrum of possibilities."

More haptic/tactile feedback touch interaction:

Nokia's Haptikos Touch Screen Handheld Web Browser

Immersion's Touch Screen: Mimics feel of real buttons
How it works:

Immersion's Fitness Market Brief (pdf)

UPDATE 7/4/09
Apple is working on this sort of thing, in a way. See my recent post:
Haptic/Tactile Feedback for the iPhone? MacRumors says, "YES!"

Dreams about haptic overlays: Part Two: Bridging the gap between virtual and physical controls on tabletops.

Interactive User Interfaces: Nat Torkington's list of links to new and cool technologies.

Quick post:

Nat Torkington recently wrote a post on O'Reilly Radar with links and descriptions about new interactive interfaces, such as a multi-touch Rubik's Cube, a weather map with haptic feedback, NextWindow's multi-touch screens, Cyber Goggles, NUI, and more.

Novint Falcon Haptic Controller video -

My Novint Falcon haptic controller arrived the other day! What is a haptic controller? Watch the video...

( I purchased the controller to work on games for young people who are visually impaired.)

Rome Reborn, Digital Rome: 3D animated history and archaeology

A recent article posted by Tracy Staedter on the Discovery News website highlights the Rome Reborn project. Rome Reborn is a 3-D re-creation of Rome as it stood in 320 A.D. The project has involved an interdisciplinary, international team of people, including computer scientists, artists, archaeologists, and historians from UCLA, the Politecnico di Milano (Italy), and the Institute for Advanced Technology in the Humanities at the University of Virginia in Charlottesville.

According to the article, Rome Reborn (Digital Rome) will be teaching tool as well as part of an on-line 3-D peer-reviewed journal, a place for researchers to publish and share their work on the project over time.

Here is a link to another detailed article on Ian Foster's blog about Rome Reborn/Digital Rome: http://ianfoster.typepad.com/blog/2007/06/digital_rome.html


On a related note, I am waiting in great anticipation for my Novint Falcon 3-D haptic game controller. Wouldn't it be great if we could provide people with visual impairments the chance to experience Rome Reborn in 3-D using the Novint Falcon? For more information, visit the Novint Falcon.