"Design is the Solution-From Visual Clarity to Clarity in the Mind" (gem of an article by Gerd Waloszek, SAP User Experience)

Design is the Solution - From Visual Clarity to Clarity in the Mind
Gerd Waloszek, SAP User Experience, 12/7/10


In this article, Gerd Waloszek provides an overview of traditional usability principles and shares his thoughts about broadening the concept of clarity to include mental states and models. His article includes charts/concept maps as well as links to great resources.


If this topic interests you, plan to block out some time to read this article and explore the links.

Child-Computer Interaction: A Featured Community at the Upcoming CHI 2011 Conference!

Last year, I attended CHI 2010 and participated in a workshop about the next generation of HCI and education.  It was a wonderful opportunity to share ideas with people from all over the world who are interested in emerging technologies, kids, and education.  I plan to attend CHI 2011 in Vancouver, Canada next May 7-11, and even though the conference is months away, I can barely wait.  The good news it that the Child-Computer Interaction community will have an important presence at the 2011 ACM CHI conference.  I wanted to share a little bit about this development on this blog. 

During CHI 2010, I signed up for the  "Designing for the iChild" course.  In one afternoon, I learned more than I had expected, especially the technique called "Layered Elaboration", a collaborative design strategy that involves inter-generational teams of children and adults.  

One of the leaders of this course was Allison Druin, Associate Professor and director of the Human-Computer Interaction Lab at the University of Maryland.  Dr. Druin's focus is in the area of child-computer interaction and how children can be meaningfully involved as partners in the design process.  

The quote below, found on the HCIL Children as Design Partners website, explains why this is so important:

"We have a chance to change technology, but more importantly we have a chance to change the life of a child. Every time a new technology enables a child to do something they never dreamed of, there are new possibilities for the future."  -Allison Druin

In my work as a school psychologist, I use technology with students quite often, especially when I'm at Wolfe, a program for students who have more complex disabilities, including severe autism.  I have been fortunate to have a new SMARTBoard at my fingertips, and access to the school's SMARTtable.  I learn from my students every day.

I believe that we are only at the "tip of the iceberg" with this sort of technology- and related applications such as the iPad and similar devices.   In my experience, well designed technologies and applications can open up a meaningful window to the world for children, teens, and others with disabilities.

Most of the information below was taking from the CHI 2011 conference website:

About the Child-Computer Interaction (CCI) community:

"At CHI, the CCI community will want to attract papers and contributions that represent real advances in the understanding of, or development and refinement of methods for, child computer interaction. It will also seek to unearth groundbreaking innovations addressing the needs, capabilities and preferences of children that have the potential to become reference works for developments in this field."

"By its very nature, The CCI community will have to be divergent in its thinking at CHI; it must also be about two of the mainstream CHI communities – engineering and design, but will potentially also be concerned with many of the communities of technologies (Smart devices, surfaces, mobile), of experiences (Play, Learning, Communication) and of methods (participatory design, evaluation)." ....

"Child Computer Interaction is a new community for CHI. It is a place for contributions where a method or a design is proposed that is especially suited to children and that could not sensibly be easily adapted for adults.  

We are keen to have contributions to all the usual CHI tracks but are also offering four special tracks for our own extra special community. These are:

• Child Partnership Projects (CPP): A design competition for teams that include children.

• Participatory Papers: Scholarly publications that are disseminated for children readers. (i.e. written in a different way)

• Lessons from the Trenches: Targeting industrial cases and experiences. A lively venue where experiences can be exchanged, and researchers can be exposed to the realities of industrial practice in this domain.

• Theatre pieces: High quality video contributions, available in a library after the conference, of methods that can be re used and learned from."

Child-Computer Interaction Chairs:
Janet C. Read
University of Central Lancashire
Panos Markopoulos
Eindhoven University of Techology
Allison Druin
University of Maryland
childcomputerinteraction@chi2010

RELATED
Walsh, G., Druin, A., Guha, ML, Foss, B., Golub, E., Hatley, L (2009)  [PDF] Layered Elaboration: A New Technique for Co-Design with Children.  ACM CHI 2009 

BlackBerry's "iPad" preview video! -- PlayBook

Thanks to Seth Sandler for the link!

More about this topic later...

Essential Interaction Design Essays and Articles: Dan Saffer's Lists, Don Norman, and Interactions Magazine

I came across a link about Dan Saffer's recent post, Essential Interaction Design Essays and Articles.  Equally important is Dan Saffer's List:  Top Ten Essential Interaction Design Books. 

Dan Saffer is one of my "important influences".  When I was taking HCI and Ubiquitous Computing courses, I bought the first edition of his book,  Designing for Interaction:  Creating Innovative Applications and Devices.  In today's world of technical convergence, it is an important read, as Saffer's content crosses a number of disciplines.

Thoughts:
It doesn't surprise me to learn that the #1 book on Saffer's Essential Interaction Design Books list is  list is Don Norman's The Design of Everyday Things.  According to Saffer,  "there’s no getting around it: this is the book. Affordances, mental models, and other bits that have all become part of the general lexicon all started with The Don’s book. A must read." 

Don Norman's book was required reading in the Human-Computer Interaction class I took a few years ago.  As I read through the book, I sensed a familiar tone.  I later learned that Don Norman was the co-author of a required textbook for one of the psychology courses I took when I was a university student the first time around.    



Don Norman's thinking has influenced me for decades - he continues to be an influence, because he writes articles for one of my favorite publications, Interactions Magazine:



It brightens up my day when I open up my mailbox- the one at the end of my real-life driveway- and find my Interactions magazine, in all of its well-designed, well-written,  semi-glossy-paged glory, waiting for me to open up and read.   The September/October, 2010 issue includes articles on topics related to authenticity in new media, the complexity of "advancement", design and usability, and the politics of development. 


A must-read is Gestural Interfaces: A Step Backwards in Usability, co-authored by Don Norman and his collaborator, Jakob Neilson, 

Here is an excerpt from the article, which highlights some of the problems of rushing to get products with natural-user interfaces out to market:

"Why are we having trouble? Several reasons:

• The lack of established guidelines for gestural control
• The misguided insistence by companies (e.g., Apple and Google) to ignore established conventions and establish ill-conceived new ones.
• The developer community’s apparent ignorance of the long history and many findings of HCI research, which results in their feeling empowered to unleash untested and unproven creative efforts upon the unwitting public"

(Interactions Magazine is a publication of ACM CHI -Association of Computing Machinery, Computer-Human Interaction interest group).


Other articles by Don Norman, published in Interactions Magazine:
The Research-Practice Gap: The Need for Translational Developers 
Natural User Interfaces are not Natural 
The Transmedia Design Challenge: Technology that is Pleasurable and Satisfying
Technology First, Needs Last: The Research-Product Gulf
To be published, available on the jnd website:
Systems Thinking:  A Product is More Than The Product  


SOMEWHAT RELATED
My resource pages:
RESOURCES: Natural User Interaction, InfoViz, Multi-touch, Blog roll, and More - a huge mega-list of links! 
Conferences, Research, Resources page


Living with Complexity
Donald Norman, to be release in October 2010



Interactions Archives


Here are a list of books/articles, suggested by Dan Saffer's readers:


Designing for Interaction – Saffer, D. (2nd Edition; 2009)
Thoughts on Interaction Design – Kolko, J. (2009)
The Humane Interface – Raskin, J.
Digital Ground – McCullough, M.
Inmates are running the Asylum – Cooper, A
Designing Interactions – Moggridge, B (ed.)
Everyware – Greenfeild, A.
Designing Social Interfaces – Malone & Crumlisch
Emotional Design – Norman, D.
Invisible Computer – Norman, D.
Persuasion Technology – Fogg, BJ
Thoughtful Interaction Design: A Design Perspective on Information Technology by Jonas Lowgren and Erik Stolterman (Paperback – Mar 30, 2007)
Designing Visual Interfaces by Mullet/San
Steve Krug – Don’t Make Me Think: A Common Sense Approach to Web Usability
Design Research: Methods and Perspectives edited by Brenda Laurel 
Information Architecture (“The Polar Bear Book”) by Peter Morville.


Thanks to Putting People First for the link to Dan Saffer's list!

Update & Links about the Interface Culture Exhibition at the 2010 Ars Electronica Festival, via Martin Kaltenbrunner

Here is an update about the the work of students in the Interface Cultures program, which will be presented at the Playful Interface Cultures at the Ars Electronica Festival in Linz, Austria:

"This year‘s presentation by students in the Interface Cultures program showcases newly emerging artistic skill profiles at the nexus of interactive media technology and interface technology. These artists adroitly combine complex disciplines such as communications technology, biosciences, physical computing, interaction design, fashionable technology and information visualization in their test environments and experimental concepts."

via Martin Kaltenbrunner (Interface Culture at the University of art and Industrial Design):
statement: http://www.interface.ufg.ac.at/arsPDF
poster: http://www.interface.ufg.ac.at/files/playful_interface_cultures.pdf
festival page: http://new.aec.at/repair/en/2010/08/16/playful-interface-cultures/
vimeo channel: http://vimeo.com/channels/interfaceculture


SOMEWHAT RELATED
Interface Culture Lab Blog
Did you know that you can get a Masters degree in Ludic Interfaces?
Ars Electronica 2010 Flickr Site
WIRED: Here comes Ars Electronica 2010 (Bruce Sterling)
Ars Electronica/Futurelab

Gesture Vocabulary from N-Trig: "N-act Hands-on"

N-Trig is a company founded in 1999 that provides pen and multi-touch solutions that integrate into LCDs and other devices, and provides opportunities for independent software vendors (ISVs) and original equipment manufacturers (OEMs) to create new interactive and hands-on computing experiences, according to the company's profile. The latest news about N-Trig's interactive capabilities was outlined in a recent article by DanaWollman, in Laptop:

N-Trig's Four-Fingered, Two-Handed Multi-touch Promises a World of New Gestures

I found the following video from N-Trig on YouTube, released on 5/11/10, that shows the new gesture set that is supported by N-Trig:

The N-act Gesture Set (depicted in the video below)
N-act3SideSweep for browsing, use fingers together for browsing
N-act2+1 - select from a displayed menu
N-act3Tap- displays open windows in a 3D carousel
N-act3Hold-rotates the 3D carousel
N-act2Scroll- scroll through a document
N-act2Tap-minimizes the open window, displays the desktop
N-act1Touch- select an item on the screen
N-act4Tap-displays customized, relevant list of web page icons; selected text/item is pasted into the chosen app.
N-act4Zoom-magnifies a movable selected area of the screen
N-act4Select-selects an area and opens a context sensitive menu

avitalntrig
Here is the promotional information from the YouTube video:
"This video demonstrates the N-trig N-act Gesture Vocabulary, a set of true multi-touch gestures for two plus one, three- and four-fingers, enabling users to perform an action directly on the screen, and providing a rich set of hand movements that enhance the overall user experience, enabling a whole new approach to how we interact with our computing devices, for a true Hands-on computing experience."


RELATED

N-Trig's Four-Fingered, Two-Handed Multi-touch Promises a World of New Gestures

Dana Wollman, 5/1/10, Laptop

www.n-trig.com
N-trig DuoSense Technology
The Future is Now:  Creating and Developing a Touch-Enabled World (pdf)
N-trig N-act Hands-On Gesture Vocabulary (N-Trig website)
Better Multi-Touch Displays Coming 
Mike Miller, Forward Thinking Blog, PC Mag (3/3/10)
DuoSense: Creating a Multi-touch Enabled World (November 2009)

Thinking Like a Storyteller: Words of Wisdom from Cindy Chastain on the use of storytelling in for interactiondesign (IXDA 10 Video)

Cindy Chastain is a Creative Director, Experience Architecture at Rapp, a global, full-service agency in NYC. Her background is in screen-writing. She also is a film-maker and technology consultant. The video below is of her presentation at IXDA 2010. It is well worth the 47 minute watch!


Here is a quote from Nasir Barday's post on the IXDA blog about Cindy Chastain's presentation:


"Designing with a narrative in mind can make a difference between a product that merely functions well and a product that engages the minds, emotions and imaginations of users." Nasir Barday, IXDA 2/26/10


Here are a few quotes from the presentation:
"How can we, as designers, provide cues that will deepen that narrative connection?"


"What can we learn from the discipline of storytelling that will help us design for more meaningful and engaging product experiences?"

"If we can move away from thinking of products in terms of interfaces and start thinking of them as representations or environments, in which agents perform actions,  we will get us to a place where we can design more fluid and engaging dialogues/experiences."



Cindy Chastain-Thinking Like a Storyteller from Interaction Design Association on Vimeo.

Near the beginning of this video, Cindy discusses the concept of the user's narrative, described as a stream of self-talk occurs when someone interacts with a design product. Two types of narratives are likely to occur.  According to Cindy, this phenomenon was noted by cognitive scientists.  One is narrative of use,  and involves the person's self-talk about  the products features and affordances. The other is a personal narrative, which focuses on what the product means, how it might fit into one's life, and how it might be used.

Johnny Holland on-line Magazine- some of my favorite posts.

If I had the time, I could spend several hours catching up with the on-line Johnny Holland Magazine.  Why?  It is full of stuff I like to think about.

"Johnny is an open collective talking, sharing, and finding answers about the interaction between people, products, systems or processes."

Here are a few of my favorite posts:

Learning from Games: A Language for Designing Emotion
Joe Lamantia, 8/3/2009

Lamantia's article discusses the thoughts of Nicole Lazzaro, a games researcher and design consultant, who suggests that there are four kinds of fun, as outlined in the chart below:
  --XEO Design Inc.

(The references for Lamantia's article are worth taking a look.)

Engaging the User: What We Can Learn from Games
Marc Sasinski, 8/31/09

When Data Gets Up Close and Personal
Stephen Anderson, 1/27/10

The Social Life of Visualization (4-part series by Jeremy Yuille and Hugh Macdonald)
The Social Life of Visualization, Part 2:  Creation Phase
The Social Life of Visualization, Part 3: Interpretation
The Social Life of Visualization, Part 4: The Capture Process

"Jeremy Yuille and Hugh Macdonald are interaction design researchers with ACID at RMIT University. They research the effects of social media on different industries, ranging from sport to finance."

The Future is Touchable
"Jeroen van Geel is founder of Johnny Holland and a senior interaction designer. He works part-time for Fabrique Communication & Design and the other part for his own clients."


Visit Johnny Holland Magazine to find out more about the "Johnnies" below:


:





(The "Johnnies" without pictures can be found on the Johnny Holland website.)

Design and Design Failures: Nicolas Nova's Interact 10 Presentation Slides (great pics!)

Today I thought I'd share slides from Nicolas Nova's presentation at Interact 10, a conference held recently in Savannah, G.A. I really wanted to attend this conference.

Design and Designed Failures: From Observing Failures To Provoking Them

Nicolas Nova is the author of the Pasta & Vinegar blog, and works at Lift Lab.

Here are a few of my favorite Pasta & Vinegar posts- I'll add a few more links later.

Tablet PC's strike back
Digital keypads in Paris
ATM interface complexity
(I've taken similar pictures while traveling!)

A review of Interact 10 can be found on Experientia's Putting People First blog.

TEI '10 Info and Links: Fourth Annual International Conference on Tangible, Embedded, and Embodied Interaction

In my dreams, I am a full-time tech student. Fortunately, I can follow my inner geek and share what I find on this blog. The information below was inspired by links from a Facebook status update by Laurence Muller, author of the Multi-Gesture blog. 

The video below is a montage of TEI'10 hands-on studio:

TEI Studios from jay silver on Vimeo.

"From TEI 2010. These are the hands-on studios (like workshops) where 200 people participated in building and making all day long elbow to elbow, getting into the details and taking perspectives."

About TEI:

TEI '10:  Fourth International Conference on Tangible, Embedded, and Embodied Interaction, January 25-27, Cambridge, MA.
"TEI, the conference on tangible, embedded, and embodied interaction, is about HCI, design, interactive art, user experience, tools and technologies, with a strong focus on how computing can bridge atoms and bits into cohesive interactive systems."



Here is a link to the keynote:
http://www.vikmuniz.net/

Here is a link to one of the papers presented at TEI:
Electronic Popables:  Exploring Paper-Based Computing through an Interactive Pop-Up Book (pdf)- Jie Qi and Leah Buechley, MIT Media Lab, High-Low Tech Group

More about Laurence Muller:
Laurence Muller (M.Sc.) , is a Fellow at the Harvard University (USA) at the School of Engineering and Applied Science (SEAS) / The Initiative in Innovative Computing (IIC) in the Scientists' Discovery Room Lab (SDR Lab). Currently he is working on innovative scientific software for multi-touch devices and display wall systems. (I took Laurence's information from his blog.)

More to come!

There is a need for multi-touch/gesture designers/developers!

If you are a talented interactive web designer/developer, game designer/developer, traditional programmer with a creative bent, or someone who who is thinking about working with technology in the future as a programmer or designer,  I urge you to consider thinking about designing/developing multi-touch applications in the near future.

In my opinion, there will be a need for multi-touch web applications as well as for multi-touch education and collaboration applications for the SMART Table, Microsoft's Surface,  multi-touch tablets like the rumored iTablet from Apple, and the multi-touch laptops and all-in-ones (Dell, HP, etc.).

Below are direct links to some of my blog posts related to multi-touch applications and screens. If you are fairly new to multi-touch, I'm sure that looking through some of my blog posts will be helpful.  All of the posts have links to resources, and most have photos and video clips of multi-touch in action.

If you are new to this blog, I have a great deal of information, links, photos, and video clips of various multi-touch screens and applications. The best way to find the stuff is to enter in a keyword in the search box for this blog:  multitouch, touch screen, gesture, multi-touch, etc. on  this blog.

Also do a search on my other blog: The World Is My Interface http://tshwi.blogspot.com

Here are some links:
Do you have an HP TouchSmart, Dell Studio One or NextWindow touch-screen? NUITech's Snowflake Suite upgrade provides a multi-touch plug-in
http://bit.ly/5tdlhc

The following blog post has a video clip that shows someone from Adobe painting with a multi-touch application in development:
More Multi-Touch!: Rumor of the mobile apple iTablet; AdobeXD & Multitouch; 10-finger Mobile Multitouch: http://bit.ly/4S9Upm

Ideum's GestureWorks: http://bit.ly/4C1p7M

Interactive Walls, Interactive Projection Systems, GestureTek's Motion-Based Games: http://bit.ly/6GRGtW

Intuilab's Interfaces: Multi-touch applications/solutions for presentations, collaboration, GIS, and commerce: http://bit.ly/7RK7qN

For software developers:
How to do Multitouch with WPF 4 in Visual Studio 2010: http://bit.ly/7c4YqC

Thoughts about technology on a cruise ship, and other reflections...

It is January 4 2010, and I am enjoying my Caribbean cruise trip. 

I’m a little disappointed that technology on cruise ships has not moved forward as I’d hoped over the past few years.   On my ship, which is less than three years old,  Wi Fi is available in each stateroom, in addition to the common areas.  This is a good thing, but it is very expensive.  The pay-as-you-go rate is 75 cents a minute!   If you have a 3G iPhone or Smartphone, you’ll have to pay outrageously high fees to connect to the internet from the ship’s connection, I’m told.

I was pleasantly surprised by some of the digital displays on the ship, especially the “show-reel” of the beautiful destination points and exciting activities that everyone looks forward to when going on a cruise.  I was also impressed with the digital touch-screen poker table in the casino, even if I don't play poker.

I even liked some of the digital signage that were basically slide show posters of nice vacation pictures.

My biggest disappointments?  

• The  large touch-screen flat-panel display that served as an interactive shore excursion kiosk.  It was tucked away in a poor location, and it didn’t make any sense!

• The interactive TV experience, specifically the the shore excursion selection process.  This experience made me hate TV remote controls more than ever!

• The cruise ship wayfinding system.   Arrrggghh.
  

I will devote at least one blog post to each of the above disappointments when I’m done editing the video.

I guess I shouldn’t have had such high technological expectations for my trip.  I’m on a Carnival cruise ship, and I know that the line is owned by the same company that owns the Holland America ships. From previous cruises on Holland America ships, I know that they are more upscale than Carnival.    I guess I got too excited when I recently learned that a few Holland America ships provide cruise-goers with the magic experience of Microsoft Surface in their lounges, and also adopted the Windows 7 operating system.  On the Carnival Freedom, things aren’t quite so advanced. 

Why is this important to me?

• I’m interested in studying how technology can facilitate collaboration, communication, information-gathering, and decision making in public spaces, and since I have plenty of cruise ship travel experience, cruse ship spaces.

• I’d like to follow up on the work I did on a student project.  Three years ago, I did  a lot of people-watching during a cruise-ship vacation, which inspired the topic of my Human Computer Interaction team project during the semester after my trip. I took another cruise ship during that semester, which further informed my thinking about this topic.  Since then, I’ve been on 4 cruises.

• I think that much of the information I obtain from my observations related to travel experiences, including cruise ships, can inform work in other related domains, such as shopping malls, museums, historical points of interest, libraries, airports, bus, railroad, and subway terminals, parks and squares, and so forth.  I also think this work can inform educational applications and simulations, such as 3D “Virtual Field Trip” games, following Universal Design for Learning principles.

I spent some of my time yesterday, our "day at sea", reading two books and jotting down some of my reflections from 1) and educational/universal design for learning perspective, and 2)  a ubiquitous computing/interactive displays in public spaces + collaborative perspective:

• Acting With Technology: Activity Theory and Interaction Design (Victor Kaptelinin and Bonnie A. Nardi)
• Thoughts on Interaction Design (Editor:  Jon Kolko)
  

UPDATE TO COME!

The Post-WIMP Explorers' Club: Update of the Updates, Morning of 12/31/09

What is the Post WIMP Explorers Club?  
I came up with the name of this semi-fictional club as a way to organize my thoughts (and blog posts) regarding the development of a new metaphor for post-WIMP applications and technologies, related specifically to natural user interfaces, natural user interaction design, and off-the-desktop user experience.  


Update, morning of 12/31/09:
Josh Blake, author of the blog "Deconstructing the NUI", posted Metaphors and OCGM  this morning.  It fleshes out post-WIMP concepts, addressing metaphors & interfaces.  The premise is that NUI metaphors will be less complex than GUI (WIMP) metaphors.    My feeling is that on the surface, this will hold true, especially for consumers/users and people creating light-weight applications and software widgets.  


Underneath the surface,  where designers and developers brains spend more time than users & consumers, things might be more complex.  Why? The technology to support the required wizardry is more complex.  With convergence, the creation of new technologies, applications, communication systems, and even electronic entertainment, is  now dependent upon the work and thinking of people from a wider range of disciplines.  Each discipline brings to the table a set of terms rooted in theory, and even research practices.

Update,  late afternoon, 12/30/09:

Richard Monson-Haefels response to Ron George's "Part 2".  The concept of OCGM might be growing on him now... OCGM: George's Razor : "If Ron George can explain how OCGM encompasses Affordances and Feedback than I'll be convinced that OCGM works for NUI. Otherwise, I think OCGM is a great start that would benefit from an added "A" and "F"." -Richard   

• OCGM relates to Occam's Razor.  It is helpful to read a bit about it if you are are interested in the post WIMP conversations. (The link is to an an article from "How Stuff Works", via Richard Monson-Haefel.)

UPDATE 12/30/09  -- This post is part of a discussion between several different bloggers, and was written before Ron George wrote his latest post, Welcome to the OCGM Generation!  Part 2, which I recommend that you read now, or within the same time frame, as this post.   Since I'm not ready to write "Part 2" of this post, I tweaked what I had and added some links to a handful of my previous posts that touch on this and related topics.  The links can be found at the bottom of this post.




START HERE FOR THE "ORIGINAL" POST FROM  12/29 & 12/20/09:


Background
About a year ago I responded to a conversation between Johnathan Brill, Josh Blake, and Richard Monson-Haefel discussing "post-WIMP" conceptualization regarding natural user interfaces and interaction, otherwise known as NUI.  The focus of the discussion was on Johnathan's post, "New Multi-touch Interface Conventions". At the time, we were reading Dan Saffer's book, Designing Gestural Interfaces, and contemplating new ways that technology can support human interaction and activities in a more natural, enjoyable, and intuitive manner.  

A few days later, I shared some of the concepts from the discussion on a post on this blog, "Why "new" ways of interaction?".  The post includes video of Johnathan Brill discussing PATA, a post-WIMP analogue to assist with multi-touch/gesture based application development, which he describes as follows:
Places
"Lighting, focus, and depth, simplified searching and effecting hyperlinked content."
Animation "Using animation to subtly demonstrate what applications do and how to use them is a better solution than using icons. Animations makes apps easier to learn."
Things "Back in the days of floppy disks, objects helped us organize our content. This limitation was forced by arcane technology, but it did have one huge advantage. We used our spatial memory to help us navigate content. Things will help us organize content and manipulate controllers across a growing variety of devices."
Auras "Auras will help us track what we are tracking and when an interaction has been successful."
(For reference, I've copied some of my responses to the first discussion, which can be found near the end of this post)


A year later....
What has changed?   Everything post-WIMP has been covered like a blanket by the NUI-word.  "NUI" now functions as a generic term for anything that is not exactly WIMP.  There is a sense of urgency now to figure out how best to conceptualize post-WIMP interfaces and interactions.  Newer, affordable technologies enable us to interact with friends and family while we are on-the go. Netbooks, e-Readers, SmartPhones, large touch screen displays, interactive HDTV, and new devices with multi-modal I/O's abound.  Our grandparents are on Facebook and twitter from their iPhones.  Our world no longer requires us to be slaves to the WIMP mentality.


So what is the problem?
The technology has moved along so fast that application designers and developers have not had a chance to catch up. (The iPhone is an exception.)  The downturn in the economy has made it difficult for many to take the leap from traditional software or web development and gain new skill sets.  On top of it all, most of us over the age of 15 have been brainwashed from years of working within the constraints of WIMP. It doesn't matter if we are users, consumers, students, designers, or developers.


Even the folks least likely to have difficulty expanding into the post-WIMP world have had some difficulty.  If you've had training in HCI (Human-Computer Interaction), you were inadvertently brainwashed with the best. The bulk of the theory and research you contemplated was launched at a time when WIMP was king, even as the Web expanded. Many of the of the principles held dear to traditional HCI folks have been shattered, and no-one has come up with a "theory of everything" that will cover all of the human actions and interactions that are supported or guided by new technologies.


The problem, in part, is that letting go of WIMP is hard to do, as illustrated by the following post from the Ars Technica website:  Light Touch:  A Design Firm Grapples with Microsoft Surface  (Matthew Braga, 6/29/09) "Ditching the mouse and keyboard means a whole lot more than just doing without two common peripherals.  As those who have worked with Microsoft Surface have found out, you have to jettison decades of GUI baggage and start with a whole new mindset...In actuality, few multi-touch gestures are really anything like what we experience in the physical world. There is no situation in which we pull on the corners of an image to increase its size, or swipe in a direction to reveal more content. So, in the context of real-world interaction, these types of gestures are far from natural...gestures should not only feel natural, but logical; the purpose that gestures like these serve, after all, is to replace GUI elements to the end of making interaction a more organic process."   (Be sure to read the comments.)

Now that the Surface is taking root in more places, and touch-screen all-in-one PC's and tablets are starting to multiply, more people are giving "NUI" some thought. Ron George, an interaction and product designer with experience working with Microsoft's Surface team has contributed to the post-WIMP discussion and spent some time sharing ideas with Josh Blake, a .NET, SharePoint, and Microsoft Surface Consultant for InfoStrat and author of Deconstructing the NUI blog. The outcome of this discussion was Ron George's December 28th blog post, "OCGM (pronounced Occam['s Razor] is the replacement for WIMP", and Josh Blake's post, "WIMP is to GUI as OCGM (Occam) is to NUI".   (Be sure to read the comments for both of these posts!)


OCGM (as conceptualized by Ron George)


Objects "are the core of the experience. They can have a direct correlation with something physical, or they can just be objects in the interface."


Containers "will be the “grouping” of the objects. This can manifest itself in whatever the system sees fit to better organize or instruct the user on interactions. They do not have to be, nor should they be, windows. They can be any sort of method of presentation or relationship gathering as seen fit."


Gestures "I went into detail about the differences in Gestures and Manipulations in a previous post [check it out for a refresher]. Gestures are actions performed by the user that initiate a function after its completion and recognition by the system. This is an indirect action on the system because it needs to be completed before the system will react to it."


Manipulations "are the direct influences on an object or a container by the user. These are immediate and responsive. They are generally intuitive and mimic the physical world in some manner. The results are expected and should be non-destructive. These are easily performed and accidental activations should be expected and frequent."

To illustrate a point regarding the validity of the OCGM analogy proposed by Ron George, Josh Blake shares the following video of a presentation from REMIX 2009, in which August de los Reyes, the Principle Director of User Experience for Surface Computing at Microsoft, briefly discusses the TOCA (Touch, Objects, Containers, and Actions) concept, suggested to replace the WIMP concept:

The video wouldn't embed, so go to the following link:


Predicting the Past: A Vision for Microsoft Surface
"Natural User Interface (NUI) is here. New systems of interaction require new approaches to design. Microsoft Surface stands at the forefront of this product space. This presentation looks at one of the richest sources for inventing the future: the past. By analyzing preceding inflection points in user interface, we can derive some patterns that point to the brave NUI world." 

The concepts outlined in the presentation are similar to Microsoft's Vision for 2019


Richard Monson-Haefel added his thoughts about the discussion about OCGM in his recent blog post, "What is NUI's WIMP?"  Richard disagrees with the OCGM concept, as he feels it doesn't encompass some important interactions, such as speech/direct voice input.   He'd probably agree that NUI is NOT WIMP 2.0.


Post-NUI, Activity Theory, and Off-the-Desktop Interaction Design:
As I was reading the recent posts and discussions regarding NUI/OCGM, I also contemplated some of what I've been reading over my holiday break, "Acting With Technology:  Activity Theory and Interaction Design", written by Victor Kaptelinin and Bonnie A. Nardi.   Victor Kaptelinin is the co-editor of "Beyond the Desktop Metaphor: Designing Integrated Digital Work Environments" (MIT Press, 2007), and has an interest in computer-supported cooperative work.  Bonnie Nardi brings to the IT world her background in anthropology, and is the co-author of "Information Ecologies:  Using Technology with the Heart" (MIT Press, 1999). The authors know what they are talking about. 


It is important to note that activity theory-based interaction design is viewed as a "post-cognitivistic", and informed by some of what I studied in psychology, education, and social science years ago. Within the field of activity theory are some important differences, which I'll save for a future post. 


Below are some concepts taken from the book. I am still mulling them over through the prism of NUI, post-WIMP, PATA, TOCA, OCGM, etc.  That's why there will be at "Part II", with specific examples.


"Means and ends, the extent to which the technology facilitates and constrains attaining user's goals and the impact of the technology on provoking or resolving conflicts between different goals


Social and physical aspects of the environment - integration of target technology with requirements, tools, resources, and social rules of the environment
Learning, cognition, and articulation,  internal vs external components of activity and support of their mutual transformations with target technology


Development -Developmental transformation of the above components as a whole" 
"Taken together, these sections cover various aspects of the way the target technology supports, or is intended to support, human actions".  (page 270)


I especially like the activity checklist included in the appendix of the book, as well as the concept of tool mediation. "The Activity Checklist is intended to be used at early phases of system design or for evaluating existing systems.  Accordingly, there are two slightly different versions of the Checklist, the "evaluation version" and the "design version".  Both versions are implemented as organized sets of items covering the contextual factors that can potentially influence the use of computer technology in real-life settings.  It is assumed that the Checklist can help to identify the most important issues, for instance, potential trouble spots that designers can address". (page 269)


"The Checklist covers a large space.  It is intended to be used first by examining the whole space for areas of interest, then focusing on the identified areas of interest in as much depth as possible...there is a heavy emphasis on the principle of tool mediation"  (page 270).


Other Thoughts
What is missing from this picture is a Universal Design component, something that I think holds up across time and technologies.  Following the principles of Universal design doesn't mean dumbing down or relying on simplicity. It is a multi-faceted approach, and relies on conctructing flexibility in use, one of the key concepts of Universal Design. I'd like to see this concept embedded in the post-WIMP conceptualization somehow. 


Because of my background in education/psychology/ special education, I try to follow the principles of  Universal Design for Learning (UDL) when I work on technology project.  I've spent some time thinking about how the principles of UDL could be realized through new interaction/interface systems.   Although this approach focuses on the educational technology domain, it is important to consider, given that a good percentage of our population - potential users, clients, consumers - has a temporary or permanent disability of one kind or another.


Components of Universal Design for Learning:
Multiple Means of Representation
Provide options for perception
Provide options for language and symbols
Provide options for comprehension
Multiple Means of Action and Expression
Provide options for physical action
Provide options for expressive skills and fluency
Provide options for executive functions
Multiple Means of Engagement
Provide options for recruiting interest
Provide options for sustaining effort and persistence
Provide options for self-regulation
-Adapted from the UDL Guidelines/Educator Checklist, which breaks down the components into more specific details.


Note:  The concept of Universal Design for Learning shares historical roots with some of the work behind Activity Theory and Interaction Design. Obviously, there is still much to contemplate regarding OCGM and other permutations of post-WIMP concepts!   


Here are my comments to the discussion on Johnathan Brill's blog from January 2009:
Thoughts: I'm seeing a trend to "optimize" WIMP-type applications so they can be manipulated by gesture and touch. People who program kiosks, ATM's and POS touch screens are examples of what I'm talking about. Touch and hold, two-fingered touch, and double-tap are just a slight transformation of the WIMP world-view, and in my opinion, are still WIMP (wimpy!). The mouse interaction "pretenders" are fine for using legacy productivity applications, OK in the short run.

For example, I have an HP TouchSmart, but I don't use the touch screen as often as I'd hoped. Try using using Visual Studio to code something on a touch screen. There is so much more that can be done! I know from the touch-screen prototype/demos I've worked on in various classes that applications that support collaboration and decision-making are important, and not just for work.

What do people DO, really? First of all, we are social beings, most of us. Think of what we share and discuss with others, and think about what sort of interactions on a display might best correspond with this interaction. Here are some of the things I've been DOING recently that involved some sort of technology and communication/collaboration with others:



---Travel planning - I recently went on a cruise and with various family members, selected activities I wanted to do on the ship as well plan my shore excursions (a complicated process)


---Picture sharing- I came back from the cruise with lots of pictures that I uploaded on Flickr. Related to this process: Picture annotating, tagging, choosing/comparing & editing it would be SO cool if I could use two sliders to enhance my pictures just so!


---Talking on the phone and responding to e-mail with friends and family members about the pictures, and what they wanted to see on Flickr- "you know, the ones of the dogs in Jamaica"... of which I had about 68!


---Financial planning with my husband. (I took info-viz last semester, so I know the possibilities are there.)


---Using the touch-screen to check-in at my eye-doctor's office: This was a user-unfriendly experience. Such a nice little screen. I was provided with a WIMPY PowerPoint-like interface which was confusing to use- and time consuming!


---Shopping at the new Super Wal-Mart: I asked the greeter if there was a map, and he said, "Food is over there, and the rest of the stuff is that way." Flat panel displays were all over the store, but of course,they weren't interactive. I had no idea where anything was, and the few items I was looking at had no bar-code tags. There wasn't a clerk in sight.   Wal-Mart TV rolled on-and-on via the display above my head. If I could only harness the display and have the talking head answer my questions! I gave up on my shopping trip when I was in the facial lotion/potion section. Too many choices, and too much fine print to read.

---Shopping at the mall with my young-adult daughters... I'll have to hold my thoughts on that one for now!

Some suggestions:
I think the artist/designers, (even dancers,) who are interested in multi-touch and gesture interaction have some interesting things to consider. (I linked to some of my previous posts.)


Again:
I am still mulling things over through the prism of NUI, post-WIMP, PATA, TOCA, OCGM, etc.  So that is why there will be at "Part II".  With specific examples!


RELATED
Multimedia, Multi-touch, Gesture, and Interaction Resources


My thoughts:
2007 Letter to the Editor, Pervasive Computing
Useful Usability Studies (pdf)
2007 Blog Post
Usability/Interaction Hall of Shame (In a Hospital)
2008 Blog Posts
Emerging Interactive Technologies, Emerging Interactions, and Emerging Integrated Form Factors
Interactive Touch-Screen Technology, Participatory Design, and "Getting It"
An Example of Convergence: Interactive TV: uxTV 2008
2009 Blog Posts
Why "new" ways of interaction?
Microsoft: Are You Listening?  Cool Cat Teacher (Vicki Davis) Tries out Microsoft's Multi-touch Surface Table
Haptic/Tactile Interface:  Dynamically Changeable Physical Buttons
The Convergence of TV, the Internet, and Interactivity:  Update
UX of ITV:  The User Experience and Interactive TV (or Let's Stamp Out Bad Remote Controls)
Digital Convergence and Interactive Television;  Boxee and Digital Convergence 
ElderGadget Blog: Useful Tech and Tools


Other People's Thoughts
Ron George's blog, OCGM (pronounced Occam['s Razor] is the replacement for WIMP  12/28/09
Ron George: Welcome to the OCGM Generation! Part 2 
Stephen, Microsoft Kitchen: OCGM, A New Windows User Experience

OCGM: A New Windows User Experience

Richard Monson-Haefel's blog, Multi-touch and NUI:  What is NUI's WIMP?
Richard Monson-Haefel:  OCGM: George's Razor
Josh Blake's blog,  Deconstructing the NUI: WIMP is to GUI as OCGM (Occam) is to NUI
Bill Buxton: Gesture Based Interaction (pdf) (Updated 5/2009)
Bill Buxton: "Surface and Tangible Computing, and the "Small" Matter of People and Design" (pdf) - ISSCC 2008
Dan Saffer, Designing for Gestural Interfaces: Touchscreens and Interactive Devices
Dan Saffer, Designing for Interaction 
Mark Weiser,  Computer for the 21st Century  Scientific American, 09, 1991
Touch User Interface:  Readings in Touch Screen, Multi-Touch, and Touch User Interface
Jacob O Wobbrock, Meredith Ringel Morris, Andrew D. Wilson User-Defined Gestures for Surface Computing CHI 2009, April 4–9, 2009, Boston, Massachusetts, USA.