I've been following the evolution of Microsoft's Kinect, and recently discovered a few interesting videos that show how far the system has come. According to Josh Blake, the founder of the OpenKinect community and author of the Deconstructing the NUI blog, the Kinect for Windows SDK v1.7 will be released on Monday, March 18th, from http://www.kinectforwindows.com. More details about this version can be found on Josh's blog as well as the official Kinect for Windows blog. It is possible to create applications for desktop systems that work with the Kinect in interesting ways, as you'll see in the following videos. I think there is potential here for use in education/edutainment!
Below is a video of Toby Sharp, of Microsoft Research, Cambridge, demonstrating Kinect Fusion. The software allows you to use a regular Kinect camera to reconstruct the world in 3D.
KinEtre: A Novel Way to Bring Computer Animation to Life According to information from the YouTube description, "KinÊtre is a research project from Microsoft Research Cambridge that allows novice users to scan physical objects and bring them to life in seconds by using their own bodies to animate them. This system has a multitude of potential uses for interactive storytelling, physical gaming, or more immersive communications."
The following videos are quite long, so feel free to re-visit this post when you have time to relax and take it all in!
Kinect Design Considerations This video covers Microsoft's Human Interface Guidelines, scenarios for interaction and use, and best practices for user interactions. It also includes a preview of the next major version of the Kinect SDK.
Kinect for Windows Programming Deep Dive This video discusses how to build Windows Desktop apps and experiences with the Kinect, and also previews some future work.
Note: Although I recently received my developer kit for Leap Motion, another gesture-based interface, I haven't lost interest in following news for Kinect.
Over the past couple of months, I've come across many interesting links related to multi-touch and gesture interaction, but I haven't had time to devote a thoughtful post to each one. "Part I", is a nice collection of experimental, commercial, and non-commercial efforts by a variety of creative technologists, with a smattering of industry news that might be of interest to IMT readers.
Smithsonian American Art Museum to Open Education Center Sara Beladi, NBC Washington News, 4/4/11 (Rumor has it that the Smithsonian American Art might include touch and multi-touch displays in it's plans for a new education center. The center was funded by an anonymous $8 million dollar gift.)
Bill Buxton, Microsoft Research, 4/7/11 - Includes lots of pictures, links to videos, and more information of what might be the first touch-screen. Also see Bill Buxton's companion website, Multi-Touch Systems that I have Known and Loved, updated on 3/21/11. Bill Buxton knows all (almost!)
"The MTbiggie uses the "Front Diffused Illumination" multitouch technique, with ambient infrared light and a DIY infrared webcam. The MTbiggie is similar to the MTmini, but includes a projected image and infrared webcam (rather than a normal webcam)...The MTbiggie isn’t the most stable and robust setup, but it is the easiest to build. To see other methods of building more stable multitouch displays, view the full multitouch display list." -Seth Sandler
(Also check out NodeBeat, a multi-touch music/audio sequencer/generator app by Seth Sandler and Justin Windle)
"IntuiLab, a global leader in surface computing software applications, today announced support for the revolutionary Microsoft Kinect device across its full line of IntuiFace products and solutions including IntuiFace Presentation and IntuiFace Commerce...Microsoft Kinect brings distant gesture control to interactive solutions. These gesture controls allow users to interact with displayed digital assets from a distance at their own pace and path – for example, browsing through a large quantity of products in a store catalog or manipulating 3D models (such as a mobile phone) – all without having to actually touch the screen..." -IntuiLab (Take a look at the IntuiLab team- an interactive page!)
9 This bit of news excited me, but don't get your hopes up. If anyone knows what will happen with the Kinect SDK, please leave a comment.)
"Update 4/18 7:34pm: Mary Jo Foley picks up this story, but the Microsoft spokesperson she talked to denied that the Kinect SDK will be open source. As she notes, Microsoft has pulled 180’s before regarding Kinect. After spokespeople initially were hostile to the idea of Kinect hacking, Xbox executives later embraced the idea that people are using Kinect for non-gaming purposes on the PC. Let’s hope Microsoft stays open to this idea." -Josh Blake
JazzMutant Lemure Version 2 : "The only multi-touch and modular controller for sequencers, synthesizers, virtual instruments, vjing and lights, now even better."
"The multitouch microscope brings new dimensions into teaching and research. Researchers at the Institute for Molecular Medicine Finland (FIMM) and Multitouch Ltd have created a hand and finger gesture controlled microscope. The method is a combination between two technologies: web-based virtual microscopy and a giant size multitouch display." "The result is an entirely new way of performing microscopy: by touching a table- or wall-sized screen the user can navigate and zoom within a microscope sample in the same way as in a conventional microscope. Using the touch control it is possible to move from the natural size of the sample to a 1000-fold magnification, at which cells and even subcellular details can be seen." -Multitouchfi Also see the Multitouch website.
In Chapter 1 of Natural User Interfaces in .NET, Josh Blake asks and answers a question posed by many people who have been under the spell of keyboard input and GUI/ WIMP interaction:
Why bother switching from GUI to NUI? The answer? Read Chapter 1 (pdf) of the book - the chapter is free.
Here are a few of my personal reasons: 1. I want to buy the next version of the iPad or something like it. 2. I want to buy a new large-screen Internet HD TV.
3. I want to buy a Kinect.
4. I do NOT want to interact with my new TV with a Sony remote. Too many tiny buttons!
5. I do NOT want to interact with my new TV with a keyboard, because it reminds me of...work.
6. Most importantly:
I want to design apps for the people I care about, and others with similar needs:
My mom.
My grandson.
Moms and dads with kids in tow. People with special needs and/or health concerns, and the people who care and guide them. Knowledge sharers and (life-long) learners....
Microsoft Surface 2.0 was unveiled at CES 2011 a few days ago, the result of a collaboration between Microsoft and Samsung. Surface 2.0 is a step up! The 40 inch 1080p high-definition LCD display no longer requires a projection/camera system, which clears the area below the screen of bulky hardware. The best part about Surface 2.0, in my opinion, is that it doesn't have to be used as a table. It can be configured in a variety of ways, even mounted on walls. For this reason, it will be useful in a variety of settings and situations.
Below is a quote about Surface 2.0 from Steve Ballmer's recent keynote address at CES 2011 that outlines the new technology that is embedded in the Surface 2.0 display:
"But what's really amazing about this technology, what really makes it magical, is the sensor itself. So, those first-generation Surface PCs needed cameras underneath that would look up to try to see what was going on. But what we have here is called PixelSense. PixelSense is new technology we've invented where there's infrared sensors all across this screen. Every single pixel is actually acting as a camera. The PC, the Surface here, can actually see." -Steve Ballmer: My Keynote Address at the 2011 International Consumer Electronics Show"(Huffington Post, 1/6/2011)
The good news is that developers have been busy at work to create applications for Surface 2.0. Below is a video demonstration of what the folks at InfoStrat have recently created to support collaborative information visualization activities:
Here's the information about the controls from the Infostratcville YouTube channel:
"This is a sneak preview of a suite of data visualization controls developed by InfoStrat for Microsoft Surface 2.0. The controls will be made available as open source software at no charge on CodePlex.com in the first half of 2011."
"This data visualization control suite provides multi-touch versions of the following controls: - DeepZoom multi-resolution image control that allows high performance display of very high-resolution imagery - PowerPoint Viewer which enables slide decks to be arranged and presented using multi-touch - PivotViewer chart control that allows dynamic sorting and categorization of data - PhysicsCanvas which provides an infinite, dynamic canvas for viewing and organizing content"
InfoStrat Releases Next-Generation Data Visualization Controls for Microsoft Surface 2.0
Washington DC – January 6, 2010 – InfoStrat today announced plans to support Microsoft Surface 2.0 by releasing a control suite that accelerates the development of next-generation multi-touch data visualizations. The controls will be made available as open source software at no charge on CodePlex.com in the first half of 2011.
This data visualization control suite provides multi-touch versions of the following controls:
§ Deep Zoom multi-resolution image control that allows high performance display of very high-resolution imagery
§ PowerPoint Viewer which enables slide decks to be arranged and presented using multi-touch
§ Pivot Viewer chart control that allows dynamic sorting and categorization of data
§ Physics Canvas which provides an infinite, dynamic canvas for viewing and organizing content
Other features of the controls:
§ Works on both Microsoft Surface and Microsoft Windows 7 with touch
§ A single application built with the data visualization framework can support multiple hardware form factors including: horizontal multi-touch tables, tablets, and large format vertical touch screens
§ Innovative object recognition to enable rapid data manipulations (only on Microsoft Surface)
InfoStrat is a member of Microsoft’s Technology Adoption Program (TAP) for Microsoft Surface. As a Microsoft Surface 2.0 TAP member, InfoStrat receives early access to hardware and software, allowing InfoStrat to gain expertise and influence the development of the product before it was released to the public.
In 2008, InfoStrat solved the problem of using Bing Maps 3D on Microsoft Surface in a way that performed well and was WPF-friendly. InfoStrat open-sourced the solution as a reusable control for the WPF and Surface community. Since then, the control has received over 120,000 page views and has over 8200 downloads, and has also been featured in many of our own applications. This control, known as InfoStrat.VE, has become one of the most popular controls for building mapping applications on Microsoft Surface: http://bingmapswpf.codeplex.com
“We are proud to be part of the Microsoft Surface development community,” according to Jim Townsend, president of InfoStrat, “and excited about the possibilities of Microsoft’s new version of Surface.”
Microsoft Surface provides a new way to experience and use information and digital content, engaging the senses, improving collaboration and empowering people to interact. Microsoft Surface is at the forefront of developing software and hardware that uses vision-based technology to fundamentally change the way people use computing devices. More information can be found at http://www.surface.com.
Information Strategies ("InfoStrat") is an award-winning Microsoft Gold Certified Partner and a Microsoft Surface Strategic Partner and member of the Technology Adopter Program.
Josh Blake recently interviewed Tamir Berliner, one of the founders of PrimeSense. If you haven't heard, Microsoft's Kinectwas based on work by PrimeSense, and licensed their technology. PrimeSense provides consumer electronics with natural user interaction capabilities. The good news is that the company recently released open-sourced middleware for natural interaction and depth-camera drivers. It will be interesting to see how this will play in the near future!
In the interview, Tamir discussed a number of topics related to postWIMP technologies. He also announced the newly created OpenNI, "an industry-led, not-for-profit organization formed to certify compatibility and interoperability of Natural Interaction (NI) devices, applications, and middleware." It is good to see this level of support for the cause!
Here is a quote from the interview that I especially liked:
"I believe that till today the devices we’ve been using, made us learn greatly lot about them before we could use them and gain their value. I’m pretty sure everyone who is reading this has got at least 3 remotes sitting on his living room table, and at least once a week needs to help someone use their computer/media center/phone/etc. It’s time for that to change and it’s up to us, the technologists to make this revolution happen, it’s time for the devices to take the step of understanding what we want and making sure we get that, even without asking if it’s a trivial task as opening a door when we approach, closing the lights when we leave the room, even making sure we have hot water to shower with when we return from work or wake up in the morning, depends on what we normally do." -Tamir
RELATED Here are a couple of videos from the OpenNI website that demonstrate OpenNI-compliant applications:
OpenNI-compliant real time skelton tracking by PrimeSense
OpenNI-compliant real time SceneAnalyzer by PrimeSense
FYI:
Josh Blake is the author of the Deconstructing the NUI blog. Over the past couple of years, he's explored natural user interfaces and interactions through his work on applications designed for Microsoft Surface and Win7 with Windows Presentation Foundation.
About a month ago, Josh organized OpenKinect, an on-line community to support collaboration among people interested in exploring ways to use Kinect with PCs and other devices. An example of this effort is the open source code, libfreenect, which includes drivers and libraries for Windows, Linux, and OS X.
The Surface is located in the Smithsonian's Castle, and is part of "The Wonder of Light: Touch and Learn!" exhibit, which opened on Tuesday, November 9th (2010). Microsoft donated the Surface unit to the Smithsonian.
Below is slideshow of the interactive exhibit:
The video below provides a closer look at the applications created by Infostrat for the Smithsonian exhibit:
Josh Blake is the Tech Lead of the InfoStrat Advance Technology Group in DC. He has been creating multi-touch applications Microsoft's Surface multi-user table-tops for a while. Recently, his team built a suite of applications designed for use by young children at a museum. Below is a video demonstration of some of this work. It really looks exciting!
In this article, Charles Petzold continues his discussion of the multi-touch support in version 4 of Windows Presentation Foundation (WPF). I like the introduction to his article: "Programming is an engineering discipline rather than a science or a branch of mathematics, so rarely does there exist a single correct solution to a problem. Varieties and variations are the norm, and often it’s illuminating to explore these alternatives rather than focus on one particular approach." -Charles Petzold I also appreciate Petzold's discussion about smooth Z transitions, his thoughtful code samples and related links.
RELATED Multi-touch Manipulation in WPF Charles Petzold (August, 2010, MSDN Magazine, UI Frontiers) "Just within the past few years, multi-touch has progressed from a futuristic sci-fi film prop to a mainstream UI. Multi-touch displays are now standard on new models of smartphones and tablet computers. Multi-touch is also likely to become ubiquitous on computers in public spaces, such as kiosks or the table computer pioneered by Microsoft Surface."
"The only real uncertainly is the popularity of multi-touch on the conventional desktop computer. Perhaps the greatest impediment is the fatigue known as “gorilla arm” associated with moving fingers on vertical screens for long periods of time. My personal hope is that the power of multi-touch will actually provoke a redesign of the desktop display. I can envision a desktop computer with a display resembling the configuration of a drafting table, and perhaps almost as large." -Charles Petzold
Information from Josh's YouTube channel: "This is a video of some of the cool multi-touch and Natural User Interface (NUI) applications I designed and developed for Surface and Windows 7." The InfoStrat.VE map control for WPF and Surface is available for free at http://virtualearthwpf.codeplex.com.
I especially like the moving ring-menu concept, as it facilitates smoother collaboration between people on an interactive table or surface, where flexible orientation control is important.
At 3:15, the demonstration of Josh's ink-shape recognition begins. This is a feature that would be great to incorporate in my applications for children with disabilities who have some fine-motor limitations.
What is the Post WIMP Explorers Club? I came up with the name of this semi-fictional club as a way to organize my thoughts (and blog posts) regarding the development of a new metaphor for post-WIMP applications and technologies, related specifically to natural user interfaces, natural user interaction design, and off-the-desktop user experience.
Update, morning of 12/31/09: Josh Blake, author of the blog "Deconstructing the NUI", posted Metaphors and OCGM this morning. It fleshes out post-WIMP concepts, addressing metaphors & interfaces. The premise is that NUI metaphors will be less complex than GUI (WIMP) metaphors. My feeling is that on the surface, this will hold true, especially for consumers/users and people creating light-weight applications and software widgets.
Underneath the surface, where designers and developers brains spend more time than users & consumers, things might be more complex. Why? The technology to support the required wizardry is more complex. With convergence, the creation of new technologies, applications, communication systems, and even electronic entertainment, is now dependent upon the work and thinking of people from a wider range of disciplines. Each discipline brings to the table a set of terms rooted in theory, and even research practices.
Update, late afternoon, 12/30/09:
Richard Monson-Haefels response to Ron George's "Part 2". The concept of OCGM might be growing on him now... OCGM: George's Razor : "If Ron George can explain how OCGM encompasses Affordances and Feedback than I'll be convinced that OCGM works for NUI. Otherwise, I think OCGM is a great start that would benefit from an added "A" and "F"." -Richard
OCGM relates to Occam's Razor. It is helpful to read a bit about it if you are are interested in the post WIMP conversations. (The link is to an an article from "How Stuff Works", via Richard Monson-Haefel.)
UPDATE 12/30/09 -- This post is part of a discussion between several different bloggers, and was written before Ron George wrote his latest post, Welcome to the OCGM Generation! Part 2, which I recommend that you read now, or within the same time frame, as this post. Since I'm not ready to write "Part 2" of this post, I tweaked what I had and added some links to a handful of my previous posts that touch on this and related topics. The links can be found at the bottom of this post.
START HERE FOR THE "ORIGINAL" POST FROM 12/29 & 12/20/09:
Background About a year ago I responded to a conversation between Johnathan Brill, Josh Blake, and Richard Monson-Haefel discussing "post-WIMP" conceptualization regarding natural user interfaces and interaction, otherwise known as NUI. The focus of the discussion was on Johnathan's post, "New Multi-touch Interface Conventions". At the time, we were reading Dan Saffer's book, Designing Gestural Interfaces, and contemplating new ways that technology can support human interaction and activities in a more natural, enjoyable, and intuitive manner.
A few days later, I shared some of the concepts from the discussion on a post on this blog, "Why "new" ways of interaction?". The post includes video of Johnathan Brill discussing PATA, a post-WIMP analogue to assist with multi-touch/gesture based application development, which he describes as follows: Places
"Lighting, focus, and depth, simplified searching and effecting hyperlinked content." Animation "Using animation to subtly demonstrate what applications do and how to use them is a better solution than using icons. Animations makes apps easier to learn." Things "Back in the days of floppy disks, objects helped us organize our content. This limitation was forced by arcane technology, but it did have one huge advantage. We used our spatial memory to help us navigate content. Things will help us organize content and manipulate controllers across a growing variety of devices." Auras "Auras will help us track what we are tracking and when an interaction has been successful." (For reference, I've copied some of my responses to the first discussion, which can be found near the end of this post)
A year later.... What has changed? Everything post-WIMP has been covered like a blanket by the NUI-word. "NUI" now functions as a generic term for anything that is not exactly WIMP. There is a sense of urgency now to figure out how best to conceptualize post-WIMP interfaces and interactions. Newer, affordable technologies enable us to interact with friends and family while we are on-the go. Netbooks, e-Readers, SmartPhones, large touch screen displays, interactive HDTV, and new devices with multi-modal I/O's abound. Our grandparents are on Facebook and twitter from their iPhones. Our world no longer requires us to be slaves to the WIMP mentality.
So what is the problem? The technology has moved along so fast that application designers and developers have not had a chance to catch up. (The iPhone is an exception.) The downturn in the economy has made it difficult for many to take the leap from traditional software or web development and gain new skill sets. On top of it all, most of us over the age of 15 have been brainwashed from years of working within the constraints of WIMP. It doesn't matter if we are users, consumers, students, designers, or developers.
Even the folks least likely to have difficulty expanding into the post-WIMP world have had some difficulty. If you've had training in HCI (Human-Computer Interaction), you were inadvertently brainwashed with the best. The bulk of the theory and research you contemplated was launched at a time when WIMP was king, even as the Web expanded. Many of the of the principles held dear to traditional HCI folks have been shattered, and no-one has come up with a "theory of everything" that will cover all of the human actions and interactions that are supported or guided by new technologies.
The problem, in part, is that letting go of WIMP is hard to do, as illustrated by the following post from the Ars Technica website: Light Touch: A Design Firm Grapples with Microsoft Surface (Matthew Braga, 6/29/09) "Ditching the mouse and keyboard means a whole lot more than just doing without two common peripherals. As those who have worked with Microsoft Surface have found out, you have to jettison decades of GUI baggage and start with a whole new mindset...In actuality, few multi-touch gestures are really anything like what we experience in the physical world. There is no situation in which we pull on the corners of an image to increase its size, or swipe in a direction to reveal more content. So, in the context of real-world interaction, these types of gestures are far from natural...gestures should not only feel natural, but logical; the purpose that gestures like these serve, after all, is to replace GUI elements to the end of making interaction a more organic process." (Be sure to read the comments.) Now that the Surface is taking root in more places, and touch-screen all-in-one PC's and tablets are starting to multiply, more people are giving "NUI" some thought. Ron George, an interaction and product designer with experience working with Microsoft's Surface team has contributed to the post-WIMP discussion and spent some time sharing ideas with Josh Blake, a .NET, SharePoint, and Microsoft Surface Consultant for InfoStrat and author of Deconstructing the NUI blog. The outcome of this discussion was Ron George's December 28th blog post, "OCGM (pronounced Occam['s Razor] is the replacement for WIMP", and Josh Blake's post, "WIMP is to GUI as OCGM (Occam) is to NUI". (Be sure to read the comments for both of these posts!)
OCGM (as conceptualized by Ron George)
Objects "are the core of the experience. They can have a direct correlation with something physical, or they can just be objects in the interface."
Containers "will be the “grouping” of the objects. This can manifest itself in whatever the system sees fit to better organize or instruct the user on interactions. They do not have to be, nor should they be, windows. They can be any sort of method of presentation or relationship gathering as seen fit."
Gestures "I went into detail about the differences in Gestures and Manipulations in a previous post [check it out for a refresher]. Gestures are actions performed by the user that initiate a function after its completion and recognition by the system. This is an indirect action on the system because it needs to be completed before the system will react to it."
Manipulations "are the direct influences on an object or a container by the user. These are immediate and responsive. They are generally intuitive and mimic the physical world in some manner. The results are expected and should be non-destructive. These are easily performed and accidental activations should be expected and frequent."
To illustrate a point regarding the validity of the OCGM analogy proposed by Ron George, Josh Blake shares the following video of a presentation from REMIX 2009, in which August de los Reyes, the Principle Director of User Experience for Surface Computing at Microsoft, briefly discusses the TOCA (Touch, Objects, Containers, and Actions) concept, suggested to replace the WIMP concept:
The video wouldn't embed, so go to the following link:
Predicting the Past: A Vision for Microsoft Surface "Natural User Interface (NUI) is here. New systems of interaction require new approaches to design. Microsoft Surface stands at the forefront of this product space. This presentation looks at one of the richest sources for inventing the future: the past. By analyzing preceding inflection points in user interface, we can derive some patterns that point to the brave NUI world."
The concepts outlined in the presentation are similar to Microsoft's Vision for 2019
Richard Monson-Haefel added his thoughts about the discussion about OCGM in his recent blog post, "What is NUI's WIMP?" Richard disagrees with the OCGM concept, as he feels it doesn't encompass some important interactions, such as speech/direct voice input. He'd probably agree that NUI is NOT WIMP 2.0.
It is important to note that activity theory-based interaction design is viewed as a "post-cognitivistic", and informed by some of what I studied in psychology, education, and social science years ago. Within the field of activity theory are some important differences, which I'll save for a future post.
Below are some concepts taken from the book.I am still mulling them over through the prism of NUI, post-WIMP, PATA, TOCA, OCGM, etc. That's why there will be at "Part II", with specific examples.
"Means and ends, the extent to which the technology facilitates and constrains attaining user's goals and the impact of the technology on provoking or resolving conflicts between different goals
Social and physical aspects of the environment - integration of target technology with requirements, tools, resources, and social rules of the environment Learning, cognition, and articulation, internal vs external components of activity and support of their mutual transformations with target technology
Development -Developmental transformation of the above components as a whole" "Taken together, these sections cover various aspects of the way the target technology supports, or is intended to support, human actions". (page 270)
I especially like the activity checklist included in the appendix of the book, as well as the concept of tool mediation. "The Activity Checklist is intended to be used at early phases of system design or for evaluating existing systems. Accordingly, there are two slightly different versions of the Checklist, the "evaluation version" and the "design version". Both versions are implemented as organized sets of items covering the contextual factors that can potentially influence the use of computer technology in real-life settings. It is assumed that the Checklist can help to identify the most important issues, for instance, potential trouble spots that designers can address". (page 269)
"The Checklist covers a large space. It is intended to be used first by examining the whole space for areas of interest, then focusing on the identified areas of interest in as much depth as possible...there is a heavy emphasis on the principle of tool mediation" (page 270).
Other Thoughts What is missing from this picture is a Universal Design component, something that I think holds up across time and technologies. Following the principles of Universal design doesn't mean dumbing down or relying on simplicity. It is a multi-faceted approach, and relies on conctructing flexibility in use, one of the key concepts of Universal Design. I'd like to see this concept embedded in the post-WIMP conceptualization somehow.
Because of my background in education/psychology/ special education, I try to follow the principles of Universal Design for Learning (UDL) when I work on technology project. I've spent some time thinking about how the principles of UDL could be realized through new interaction/interface systems. Although this approach focuses on the educational technology domain, it is important to consider, given that a good percentage of our population - potential users, clients, consumers - has a temporary or permanent disability of one kind or another.
Components of Universal Design for Learning: Multiple Means of Representation Provide options for perception Provide options for language and symbols Provide options for comprehension Multiple Means of Action and Expression Provide options for physical action Provide options for expressive skills and fluency Provide options for executive functions Multiple Means of Engagement Provide options for recruiting interest Provide options for sustaining effort and persistence Provide options for self-regulation -Adapted from the UDL Guidelines/Educator Checklist, which breaks down the components into more specific details.
Note: The concept of Universal Design for Learning shares historical roots with some of the work behind Activity Theory and Interaction Design. Obviously, there is still much to contemplate regarding OCGM and other permutations of post-WIMP concepts!
Here are my comments to the discussion on Johnathan Brill's blog from January 2009:
Thoughts: I'm seeing a trend to "optimize" WIMP-type applications so they can be manipulated by gesture and touch. People who program kiosks, ATM's and POS touch screens are examples of what I'm talking about. Touch and hold, two-fingered touch, and double-tap are just a slight transformation of the WIMP world-view, and in my opinion, are still WIMP (wimpy!). The mouse interaction "pretenders" are fine for using legacy productivity applications, OK in the short run.
For example, I have an HP TouchSmart, but I don't use the touch screen as often as I'd hoped. Try using using Visual Studio to code something on a touch screen. There is so much more that can be done! I know from the touch-screen prototype/demos I've worked on in various classes that applications that support collaboration and decision-making are important, and not just for work.
What do people DO, really? First of all, we are social beings, most of us. Think of what we share and discuss with others, and think about what sort of interactions on a display might best correspond with this interaction. Here are some of the things I've been DOING recently that involved some sort of technology and communication/collaboration with others:
---Travel planning - I recently went on a cruise and with various family members, selected activities I wanted to do on the ship as well plan my shore excursions (a complicated process)
---Picture sharing- I came back from the cruise with lots of pictures that I uploaded on Flickr. Related to this process: Picture annotating, tagging, choosing/comparing & editing it would be SO cool if I could use two sliders to enhance my pictures just so!
---Talking on the phone and responding to e-mail with friends and family members about the pictures, and what they wanted to see on Flickr- "you know, the ones of the dogs in Jamaica"... of which I had about 68!
---Financial planning with my husband. (I took info-viz last semester, so I know the possibilities are there.)
---Using the touch-screen to check-in at my eye-doctor's office: This was a user-unfriendly experience. Such a nice little screen. I was provided with a WIMPY PowerPoint-like interface which was confusing to use- and time consuming!
---Shopping at the new Super Wal-Mart: I asked the greeter if there was a map, and he said, "Food is over there, and the rest of the stuff is that way." Flat panel displays were all over the store, but of course,they weren't interactive. I had no idea where anything was, and the few items I was looking at had no bar-code tags. There wasn't a clerk in sight. Wal-Mart TV rolled on-and-on via the display above my head. If I could only harness the display and have the talking head answer my questions! I gave up on my shopping trip when I was in the facial lotion/potion section. Too many choices, and too much fine print to read.
---Shopping at the mall with my young-adult daughters... I'll have to hold my thoughts on that one for now!
Some suggestions:
I think the artist/designers, (even dancers,) who are interested in multi-touch and gesture interaction have some interesting things to consider. (I linked to some of my previous posts.)
Again: I am still mulling things over through the prism of NUI, post-WIMP, PATA, TOCA, OCGM, etc. So that is why there will be at "Part II". With specific examples!
