Last night I dreamt about haptic touch-screen overlays...

Technology Dreams
I just haven't had enough time during the day to focus on my passion for interactive technology. I wish I could grab a picture from my dream about haptic overlays and share it here on my blog!   To make sure this technology wasn't only in my dreams, I decided to google "haptic overlays", and here are some of the results:

Future Directions for Tactile Feedback (Peter Odum, Idlemode, 1/23/09)
"Current tactile solutions fall short either in reconfigurability or in pre-interaction feedback...And it’s easy to provide a physical sensation after the user interacts, but not to provide buttons that can be physically felt *before* the interaction is committed."

Some of the work regarding haptic or tactile feedback has focused on smaller screens and virtual reality:
Vinyl overlay for tactile feel on iPhone NES emulator
More Details on Nokia Haptikos Tactile Touchscreens
Haptic Overlay Device for Flat Panel Touch Displays (pdf)
Touchscreen Feedback Overview
Team Daimler-Chrysler Touch (pdf)
The Twelfth Haptic Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems 2004
Patent: Method and apparatus for multi-touch tactile touch panel actuator mechanisms
Benali-Khoudja, M., Hafez, M., Alexandre, J., Kheddar, A. Tactile Interfaces: A State-of-the-Art Survey (pdf) ISR 2004, 35th International Symposium on Robotics

Here is the concept in the form of a "wearable" design:

The surface of this design looks very much like the overlay material that I was working with in my dream, except that the grid-sensors were the same translucent color as the overlay. (I was trying to program this overlay in my dream, but that is another story!)


Here is some technical information about these little haptic finger overlays- with a little tweaking, they could be transformed into something really cute...

"The key material to the display is an electroactive polymer that can stimulate the skin without using any additional electromechanical transmission. The polymer consists of eight layers of dielectric elastomer actuator films which have been sprayed with electrodes in a specific pattern. Along with a protective layer to separate the electrodes from the skin, the entire polymer sheet is about 210 micrometers thick. In their study, the researchers fabricated an 11 x 14 mm sheet with Velcro on the edges, and rolled it up in the shape of a thimble to be worn on the finger. The display can convey information to the wearer when the electrodes induce a voltage across the films. A voltage causes the films to compress down and expand outward. In doing so, the films put pressure on the wearer’s skin, inducing a “mild sensation.” Like most polymers, the device is hyper-elastic, meaning that it can experience large amounts of elastic strain and recover its original shape.This simple stimulation mechanism, which doesn’t require complex electronics, is one of the greatest advantages of the soft tactile display compared with current displays. Its other benefits include efficient power usage, cost-effectiveness, and easy fabrication. As the researchers note, the new display has lower power and displacement values than is considered optimal, which may limit its applications to specific areas"

The Search Continues...

This journey led me to Nokia's "Morph", a device using nano-materials, something I think I posted about a while ago in one of my blogs. (This video reminds my of some of my more vivid technology dreams. It must be nice to be paid to transform technology dreams into prototypes and demonstration videos!)

The Video:

"Launched alongside The Museum of Modern Art “Design and The Elastic Mind” exhibition, the Morph concept device is a bridge between highly advanced technologies and their potential benefits to end-users. This device concept showcases some revolutionary leaps being explored by Nokia Research Center (NRC) in collaboration with the Cambridge Nanoscience Centre (United Kingdom) – nanoscale technologies that will potentially create a world of radically different devices that open up an entirely new spectrum of possibilities."

More haptic/tactile feedback touch interaction:

Nokia's Haptikos Touch Screen Handheld Web Browser

Immersion's Touch Screen: Mimics feel of real buttons
How it works:

Immersion's Fitness Market Brief (pdf)

UPDATE 7/4/09
Apple is working on this sort of thing, in a way. See my recent post:
Haptic/Tactile Feedback for the iPhone? MacRumors says, "YES!"

Dreams about haptic overlays: Part Two: Bridging the gap between virtual and physical controls on tabletops.

Joel Eden's Informative Post: Designing for Multi-Touch, Multi-User and Gesture-Based Systems

Joel Eden is a User Experience Consultant at Infragistics- he recently wrote a detailed article/post in the Architecture & Design section of Dr. Dobbs Portal, "Designing for Multi-Touch, Multi-User and Gesture Based Systems". I thought I'd share the link, since I've been writing on the same topic.

In his article, Joel explains the differences between traditional WIMP (Window, Icon, Menue, Pointer) interaction and gesture, multi-touch, and multi-user systems. These systems are also known as Natural User Interfaces, or NUI. He recommends that "rather than trying to come up with new complicated ways to interact with digital objects, your first goal should be to try to leverage how people already interact with objects and each other when designing gesture based systems."

Joel goes on to outline UX (User Experience, IxD (Interaction Design), and HCI (Human-Computer Interaction) concepts that designers should consider when developing new systems, - Affordances, Engagement, Feedback, and "Don't Make Us Think", which he summarizes in the conclusion of his article.

I especially liked Joel's references:

Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension

Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data

Gibson, John J. The Ecological Approach to Visual Perception

Krug, Steve. Don't Make Me Think: A Common Sense Approach to Web Usability, Second Edition

Norman, Don. The Design of Everyday Things

Norman, Don. Things That Make Us Smart: Defending Human Attributes In The Age Of The Machine

I would also add the following references:
Bill Buxton
Multi-touch Systems I have Known and Loved
(Regularly updated!)
Sketching User Experiences: Getting the Design Right and the Right Design

"Our lack of attention to place, time, function, and human considerations means these fancy new technologies fail to deliver their real potential to real people." - Bill Buxton

Dan Saffer
Designing for Interaction: Creating Smart Applications and Clever Devices
Designing Gestural Interfaces
SAP
Touchscreen Usability in Short
(Summary by Gerd Waloszek of the SAP Design Guild)
SAP Design Guild Resources (User-Centered Design, User Experience, Usability, UI Guidelines, Visual Design, Accessibility)
Kevin Arthur (Synaptics)
Touch Usability
Bruce "Tog" Tognazzini
Ask Tog: Interaction Design Solutions for the Real World
Inclusive Design, Part I
First Principles of Interaction Design
John M. Carroll
Human Computer Interaction (HCI) (History of HCI)
Bill Moggridge
Designing Interactions
Ben Shneiderman
Leonardo's Laptop: Human Needs and the New Computing Technologies
Edward Tufte
Visual Explanations
Beautiful Evidence
The Visual Display of Quantitative Information
Envisioning Information
Rudolf Arnheim (Gestalt)
Art and Visual Perception: A Psychology of the Creative Eye

Update: A great reading list on general HCI. Some of the authors were involved in the early days of touch, bi-manual, and multi-touch interaction.

Jan's Top Ten List of Books on Human-Computer Interaction


FYI: If you know much about Windows Presentation Foundation, you probably know that Josh Smith, WPF guru, also works at Infragistics

Touching Windows 7 - Informative post, video, and comments from the Engineering Windows 7 blog

Touching Windows 7 post on the Engineering Windows 7 blog provides a good overview of Microsoft's work with touch and gesture interfaces over the past several years, building upon earlier work with the TabletPC.

There are quite a few opinions shared in the post's comment section. It is worth the read.

Below are two videos from the Windows 7 post:

Windows 7 Gestures
<a href="http://video.msn.com/?mkt=en-US&playlist=videoByUuids:uuids:891c68b3-a534-4159-b6b2-8e4ac56b6890&showPlaylist=true" target="_new" title="Windows 7 Touch Gestures">Video: Windows 7 Touch Gestures</a>

My comments will be forthcoming.

Put-That-There: Voice and Gesture at the Graphics Interface and more Blasts from the 1980's HCI Past

bigkif's information about "Put-That-There" about Put-That-There gives a good description of this video:

Put-That-There at CHI '84

"In 1980, Richard A. Bolt from MIT wrote Put-that-there : voice and gesture at the graphics interface. It was a pioneering multimodal application that combined speech and gesture recognition.

This demo shows users commanding simple shapes about a large-screen graphics display surface. Because voice can be augmented with simultaneous pointing, the free usage of pronouns becomes possible, with a corresponding gain in naturalness and economy of expression. Conversely, gesture aided by voice gains precision in its power to reference."

Richard A. Bolt "Put-That-There": Voice and Gesture at the Graphics Interface (pdf) SIGGRAPH '80

Here is another blast from the '80's:

Kankaanpaa A. FIDS- AFlat-Panel Interactive Display System IEEE March 1988 IEEE Computer Graphics Applications(Nokia Information Systems)

"Although the needs and expectations of these various users are very diverse, they all have a common requirement: more natural and easier methods for communicating with the computer than are available today. Furthermore, they do not want to interact with the computer; they want to communicate with the application they are using. They do not want to use computer jargon; they want to use the same natural methods that they use when they perform the same tasks without a computer."

“We believe that only three of the flat-panel technologies described above, namely LCD, EL, and plasma, will be sufficiently advanced for mass production within this decade.”

Bill Buxton was working on multi-touch and gesture interaction in the 1980's, but his dreams did not become a reality until this century, for a variety of reasons. He shared his thoughts about the paradox of the speed of technology in a presentation at the 2008 IEEE International Solid-State Circuits Conference:Surface and Tangible Computing, and the “Small” Matter of People and Design”(pdf)

‘Carrying on from an earlier thesis in our department (Mehta , 1982) , we built a tablet that was sensitive to simultaneous touches at multiple locations, and with the ability to sense the degree of each touch independently (Lee, Buxton & Smith, 1984). We stopped the work in late 1984 when I saw a much better implementation at Bell Labs – one that was transparent and mounted over a CRT. The problem was that they never released the technology, so, the whole multi-touch venture went dormant for 20 years. But, I never stopped dreaming about it. (Lesson: don’t stop your research just because someone else is way ahead of you. It might be transitory, and anyhow, remember the story of the tortoise and the hare.)

“I spoke earlier about the paradox in the speed of technology development it goes at rocket speed, but that of a glacier as well; Simultaneously! In the perfect world, this would be ideal: we could go through several iterations of ideas so that by the time the new paradigms of interaction, such as Surface and Tangible computing are ready for prime time, everything will be in place. But, the rapid iteration is more directed at supporting the old paradigms faster and cheaper, rather then helping shape the new ones. The reasons are not hard to understand. From the perspective of circuit design, the problems are really hard. So, one has to have one’s head down working flat out to get anything done. But, there is a side of me that motivated this paper that asks, If it is so hard, then isn’t it worth making sure that the things one is working on are things that are worthy of one’s hard-earned skills?”

SOMEWHAT RELATED

Bill Buxton's Haptic Input References (pdf)

The Internet of Things can be Cute: MIR:ROR by Violet

The MIR:ROR application supports memory and interaction with your computers as well as your Web 2.0 applications via a RFID Stamp. You can even figure out the last time you fed your fish!

Here is some of the promotional information from the Violet website:
"Mir:ror makes your everyday objects interactive, intelligent, communicant. Stick RFID Ztamps on them and show them to the Mir:ror: your keys will send e-mail to tell someone you’ve got home, your pills know when you’ve swallowed them, your toys play videos… Thousands of uses you can easily program through a Website."

"Nano:ztags are lovable micro-rabbits with a RFID Ztamp in their tummy. Program them to play any content or application you choose each time you show them to a Nabaztag:tag or Mir:ror."

(Also posted on the Technology Supported Human-World Interaction blog)

Albrecht Schmidt's User Interface Engineering Blog: Great Links, References, and Resources

I really like Albrecht Schmidt's User Interface Engineering blog.

Albrecht Schmidt is a professor at the University of Duisburg-Essen who focuses his research on "novel user interfaces and innovative applications enabled by ubiquitous computing." Dr. Schmidt previously headed something called the "Embedded Interaction Research Group" at the Ludwig-Maximilians University in Munich.

Albrecht Schmidt will be working in collaboration with Chris Kray's group at the Culture Lab at Newcastle University in the UK. The work will focus on creating and building interactive appliances. He also mentioned the work of Jayne Wallace, one of the researchers at the Culture Lab, who creates digital jewelry.

The best thing about Albrecht's recent post was his short list of references:

[1] Wallace, J. and Press, M. (2004) All this useless beauty The Design Journal Volume 7 Issue 2 (PDF)

[2] Jayne Wallace. Journeys. Intergeneration Project.

[3] Kern, D., Harding, M., Storz, O., Davis, N., and Schmidt, A. 2008. Shaping how advertisers see me: user views on implicit and explicit profile capture. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 3363-3368. DOI= http://doi.acm.org/10.1145/1358628.1358858

Another recent post that I like was "Teaching, Technical Training Day at the EPO". The following topics were covered during a training event at the European Patent Office in Munich, Germany:

• Merging the physical and digital (e.g. sentient computing and dual reality [1])
  
• Interlinking the real world and the virtual world (e.g. Internet of things)
  
• Interacting with your body (e.g. implants for interaction, brain computer interaction, eye gaze interaction)
  
• Interaction beyond the desktop, in particular sensor based UIs, touch interaction, haptics, and Interactive surfaces
  
• Device authentication with focus on spontaneity and ubicomp environments
  
• User authentication focus on authentication in the public
  
• Location-Awareness and Location Privacy

I liked the references that Dr. Schmidt posted, given my growing interest in topics related to interactive wireless sensor networks:

[1] Lifton, J., Feldmeier, M., Ono, Y., Lewis, C., and Paradiso, J. A. 2007. A platform for ubiquitous sensor deployment in occupational and domestic environments In Proceedings of the 6th Conference on international information Processing in Sensor Networks (Cambridge, Massachusetts, USA, April 25 - 27, 2007). IPSN '07. ACM, New York, NY, 119-127. DOI= http://doi.acm.org/10.1145/1236360.1236377

[2] Naohiko Kohtake, et al. u-Texture: Self-organizable Universal Panels for Creating Smart Surroundings. The 7th Int. Conference on Ubiquitous Computing (UbiComp2005), pp.19-38, Tokyo, September, 2005. http://www.ht.sfc.keio.ac.jp/u-texture/paper.html

[3] Schwesig, C., Poupyrev, I., and Mori, E. 2004. Gummi: a bendable computer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria, April 24 - 29, 2004). CHI '04. ACM, New York, NY, 263-270. DOI= http://doi.acm.org/10.1145/985692.985726

[4] Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., and Shen, C. 2007. Lucid touch: a seethrough mobile device. InProceedings of the 20th Annual ACM Symposium on User interface Software and Technology (Newport, Rhode Island, USA, October 07 - 10, 2007). UIST '07. ACM, New York, NY, 269-278. DOI= http://doi.acm.org/10.1145/1294211.1294259

[5] Campbell, A. T., Eisenman, S. B., Lane, N. D., Miluzzo, E., Peterson, R. A., Lu, H., Zheng, X., Musolesi, M., Fodor, K., and Ahn, G. 2008. The Rise of People-Centric Sensing. IEEE Internet Computing 12, 4 (Jul. 2008), 12-21. DOI= http://dx.doi.org/10.1109/MIC.2008.90

If you take a peek at the Culture Lab website, be sure to look at the Ambient Kitchen, designed to support older people with memory difficulties:

The Ambient Kitchen probably be a great way to help busy families get meals on the table, too!

Also look at Jayne Wallace's digital jewelry website The following pictures are of a neckpiece that triggers digital visits of silent film sequences on digital displays in its vicinity:

Some of Jayne's digital jewelry was created in response to "physical, clinical, social, and emotional dynamics of memory loss".

I hope you enjoy exploring Albrecht Schmidt's User Interface Engineering blog and the Culture Lab website!