For the Tech Curious: "Get in Touch with Touchless": Multi-touch with just a webcam and the free demo application!

Via the Seattle Tech Report's Microsoft Blog:



You can find the demo code on the Codeplex website. Here is a quote:

"The Touchless SDK enables developers to create multi-touch based applications using a webcam for input. Touch without touching."

"Touchless started as Mike Wasserman’s college project at Columbia University. The main idea: to offer users a new and cheap way of experiencing multi-touch capabilities, without the need of expensive hardware or software. All the user needs is a camera, which will track colored markers defined by the user."

(I posted about the Touchless SDK previously, but I didn't have the video.)

Every Surface a Computer: "Scratch" Capturing Finger Input on Surfaces using Sound. Video by Chris Harrison and Scott Hudson's Video - UIST '08

Chris Harrison and Scott Hudson, from the Human-Computer Interaction Group at Carnegie-Mellon University, presented their latest research at the UIST '08 conference. Take a look at the video below to see how gestures that result in sounds can can transformed on unpowered finger input surfaces, using a stethoscope sensors and filters:



Yes, every surface is a computer!
(Even your pants...)

For detailed information, read the paper presented at UIST '08 by Chris Harrison and Scott E. Hudson:
Scratch Input: Creating Large, Inexpensive, Unpowered, and Mobile Finger Input Surfaces

RELATED:

The Best Paper Award at UIST '08 was "Bringing Physics to the Surface", by Andrew Wilson, of Microsoft Research, and Ahahram Izadi, Otmar Hilliges, Armando Garcia-Mendoza, and David Kirk, of Microsoft Research, Cambridge.

Here is the abstract:

"This paper explores the intersection of emerging surface technologies, capable of sensing multiple contacts and of-ten shape information, and advanced games physics engines. We define a technique for modeling the data sensed from such surfaces as input within a physics simulation. This affords the user the ability to interact with digital objects in ways analogous to manipulation of real objects. Our technique is capable of modeling both multiple contact points and more sophisticated shape information, such as the entire hand or other physical objects, and of mapping this user input to contact forces due to friction and collisions within the physics simulation. This enables a variety of fine-grained and casual interactions, supporting finger-based, whole-hand, and tangible input. We demonstrate how our technique can be used to add real-world dynamics to interactive surfaces such as a vision-based tabletop, creating a fluid and natural experience. Our approach hides from application developers many of the complexities inherent in using physics engines, allowing the creation of applications without preprogrammed interaction behavior or gesture recognition."
Preparation for the Internet of Surfaces & Things?



(Cross-posted on the Technology-Supported Human World Interaction blog)

Multi-Touch News from WinHEC and PDC

I received the following videos and links from Anthony Uhrick, who happens to be at WinHEC this week and was at PDC 2008 last week. Touch screen, multi-touch, and gesture technology is starting to catch on. (Anthony is the VP of sales for NextWindow, the company that created the touch screen for the HP TouchSmart and other multi-touch enabled displays.)

Below is a video clip of a multi-touch photo presentation system running Windows 7: Gesture + Touch - has gesture and physics engines.

Apparently the application can run on Vista, Win 7, and Win 7 Touch.

Here is an HP TouchSmart PC, running a Touch Map application on Windows 7:


The following clip is of a newscaster using a multi-touch transparent screen.The display is from U-Touch Ltd. a partner of NextWindow. In my opinion, the application enhances the viewers understanding of the various news topics, and is visually appealing as well.


The graphics engine used in this application was developed by Vizrt, the same folks who were behind CNN's video hologram. Here are a few pictures from the Vizrt website:

The workflow behind the CNN hologram (click to enlarge)

The holograph "transporter" room.

For more videos using Windows 7 apps, see creamhackered's YouTube channel. (Videos appear to be from NeoWin Net.)

Windows 7 Design Concepts and Usability Tests

Searching for Multi-Touch Info? Drivers for Windows 7 Available from NextWindow & HP TouchSmart...More about N-Trig..Multi-Touch Resources

Multi-Touch News and Links

Next Window Releases Touch Screen Drivers for Microsoft Windows 7

"The technology to build multi-touch applications for next year's operating system is available today."

Pleasanton, CA (PRWEB) November 4, 2008 --

"NextWindow, the leader in optical touch technology, released a touch screen driver for Windows 7 developers this week. The introduction of multi-touch gestures to this operating system will generate applications giving users more flexibility when engaging with content."

"NextWindow introduced this technology at Microsoft's Professional Developer's Conference (PDC) in October and will be presenting it at the Windows Hardware Engineering Conference (WinHEC) November 4 - 6, 2008. At this event, NextWindow will be showcasing their latest in optical touch solutions including the integrated touch display found on the HP TouchSmart All-in-One computer."

"Providing a touch screen solution for Windows 7 will give end-users an intuitive method to interact with their computers," said Anthony Uhrick, VP of Sales, NextWindow. "We've created a driver that enables developers to take full advantage of multi-touch functionality."

Software developers interested in touch solutions for Windows 7 are invited to contact NextWindow or download the driver directly from http://www.nextwindow.com/windriver. " More information about NextWindow is available at nextwindow.com."

Contact:
David Villarina, NextWindow
dvillarina @ nextwindow.com

N-trig Delivers Innovative Hands-On™ Computing to PC Industry

"Multi-Touch and Pen Technology to Humanize the Computer Interface and Create New Markets"

KFAR SABA, Israel & AUSTIN, Texas--(BUSINESS WIRE)--"N-trig, providers of DuoSense^TMTM evolution. With the industry’s only combined pen and multi-touch capabilities, N-trig is transforming the way people interact with computers. technology, combining pen and capacitive touch in a single device, brings the power of technology and the human touch together to begin a new era in interface technologies and lead the Hands-on computing evolution. With the industry’s only combined pen and multi-touch capabilities, N-trig is transforming the way people interact with computers"
"...Realizing the power of the human interface, N-trig’s DuoSense digitizers are designed to integrate easily, support any type of screen, keep devices slim, light and bright, and can support numerous applications from small notebooks to large LCDs. Combined, pen and touch enables users to open files, manipulate pictures and browse the desktop as they would the files on their desk...Currently available on the Dell Latitude XT and additional OEM designs planned to come to market in early 2009, N-trig is opening a window onto a world where multi-touch is the accepted standard for computer interfaces."
Related

N-trig

Microsoft WinHEC 2008 Conference

NextWindow's Optical Touch Screen Technology

N-trig's Capacitive Touch Technology

All you ever wanted to know about Multi-Touch:
Bill Buxton's Multi-Touch Systems that I Have Known and Loved

All you ever wanted to know about interactive tables that support collaboration:
Pasta & Vinegar's List of Interactive Tables
(From 2005, but has been updated.)

All you ever wanted to know about tangible user interfaces:
5 Lessons About Tangible Interfaces (pdf) - Nicolas Nova

All you ever wanted to know about interactive gestures:
Interactive Gestures (wiki)

All you ever wanted to know about open-source multi-touch & related technology:
NUI Group (Natural User Interface)

Resources and Links about Touch Screens, Tables, and Multi-touch

Note: I highlight news, thoughts, and reflections about interactive multimedia, multi-touch, and related emergent technologies on this blog.

If you don't see what you are looking for on this post, feel free to do a search on this blog, or my other blog, Technology-Supported Human-World Interaction.

For Multi-Touch Interaction Humor:
Multi-touch Parody of CNN's Magic Map Wall: Saturday Night Live's Weekend Update MegaPixel Giant Touch Map

Microsoft Surface Parody

Emerging Interactive Technologies: SecondLight from Microsoft, at PDC2008

Picture from PCMAG.Com
The videoclip below is a demonstration of the prototype of SecondLight, a Surface-like application developed by a team of researchers at Microsoft Research UK. If you place a tracing-paper like object on the surface, secondary information regarding the content on the surface can be revealed. The system relies on a projector system and a special liquid crystal on the display.



-via Thoughtware.TV

Here is another video from Slashgear that describes SecondLight. According to the video, SecondLight is a multi-point display, displaying images on a surface, and images through a surface. The system has two hi-res cameras, two projectors with optical shutters, infrared illumination, and an electrically switchable diffuser.

SecondLight can track images from a distance, track IR reflective objects, and also track IR emissive objects with mobile, multi-point touch detection.

Steven Sinofsky Discusses Multi-touch and the HP TouchSmart, Windows 7, and more at PDC2008

Steven Sinofsky at the PDC2008 Open Space


In this video, Steven discusses Windows 7 and confirms that Windows 7 works with the HP TouchSmart: "The hardware is multitouch." Sinofsky also discusses how there might be a need for multi-touch drivers for the HP TouchSmart.

Take the 25 minutes or so to watch the video. It is worth it, even the more technical aspects, including how events will be managed to minimize boot time.

Dude!
Sinofsky says you gotta say "dude!". Watch the video to get it!

bgC3: A new think tank for Microsoft's Bill Gates? via Todd Bishop and TechFlash

I'm on a Microsoft roll this week.

When I checked out a post by Marshall Kirkpatrick on the ReadWrite web today, noticed a link to an article about "Bill Gates' Mysterious new company", bgC3, writtten by Todd Bishop, co-founder and managing editor of TechFlash, and author of the Todd Bishop's Microsoft Blog.

I know that Gates predicts that every surface will one day be a computer. Maybe his new company will be addressing this concept behind the scenes?

"Federal trademark filings provide more clues – describing bgC3 as a think tank, under a generic trademark classification that corresponds broadly to areas including "scientific and technological services," "industrial analysis and research," and "design and development of computer hardware and software."...The “bg” is Bill Gates, the insider says, but the “C” stands for “catalyst.” The idea is that Gates will play that role as he brings together new people and ideas."

For more information about surface computing, read my previous post, "The Internet of Surfaces? Microsoft's Pete Thompson discusses screens and surfaces of all sizes"

If you've heard more about the focus bgC3 LLC, leave a comment!

Time for More Touch! Part Two: Microsoft's "Oahu", a hypothetical (?), affordable version of the Surface multi-touch table..

Long Zheng, from the I Started Something blog, was privy to a survey from Microsoft about "Oahu" (via someone named Kerien).

The following description of Oahu is a quote from Long Zheng's website, and reportedly was the introductory section of Microsoft's survey:

"The following questions refer to a computing device called “Oahu” that has an innovative multi-touch screen. Oahu is a flat screen that sits horizontally like a table top. You can interact with Oahu by touching the screen, instead of using a mouse, and more than one person can interact with Oahu at the same time. You and others can move objects on the screen with your hands and touch icons to open up programs, games, or music. People using the device can also use their fingertips to expand and shrink objects on the screen. The screen recognizes people’s hand movements and touches and reacts accordingly. You can bring up an on-screen keyboard to input information. Oahu also works with other devices (such as digital cameras, cell phones, and MP3 players) by getting information from or sending information to them. Oahu is on with no waiting time to start up. Oahu can come as a freestanding table, placed into a piece of furniture, or built into a countertop. The type of Oahu devices we are asking about today are not portable but if they are furniture or tables, they can be placed anywhere in your home."

mmmm.... sounds just like a Surface....


Photo via I Started Something

The price of Oahu quoted in the survey? $1,499.00. A substantial savings, considering that the price of Microsoft's Surface is $10,000.00.

This price approaches the affordable range for schools. I wonder if any questions in the survey addressed the learning aspects of the Oahu, other than helping children with homework. With the upcoming Windows 7 OS and its multi-touch capabilities, I'm sure we'll be seeing the spread of this technology.

FYI:
Long Zheng is working on a Business Commerce and Multimedia Systems double degree at Monash University in Australia. His purpose in blogging is to be on the cutting edge of first-breaking news related to technology.

For more information about Windows 7, see the Engineering Windows 7 blog.

Microsoft Surface and the Nintendo Wii?! Interesting mashup from Stimulant

Stimulant: Microsoft Surface + Nintendo Wii Mashup from Stimulant on Vimeo.


Stimulant, a company that has partnered with Microsoft to develop applications for the Surface multi-touch table:

"What do you get when you mash-up Microsoft Surface with a Wii Balance Board? Pressure-sensitive surface computing! Yes, the Surface is sitting directly on the Balance Board (it supports 600 pounds, we checked). Here, Josh demonstrates a simple application that lets users create bubbles of various sizes and roll them around the table using pressure on the edges of the Surface. You also get a sneak peek at the WPF/Silverlight physics engine we've been working on as well. Pressure sensitivity adds an extra dimension to the Surface experience and opens new doors on an already highly advanced platform."

I'd like to learn more about Stimulant's other Surface experiments!

Nintendo Wii

Microsoft Surface's Hotel Concierge Application: Let's see an affordable Surface, deployed in classrooms and libraries!

Microsoft Surface at the Sheraton Hotel:



Think how this could play out in classrooms, libraries, information centers, and other public spaces.

Need I say more?

(Cross posted on TechPsych and Technology-Supported Human-World Interaction blogs)

Microsoft Research project: MouseMischief - Multi-user, Multi-Mice Interaction on Large Displays

This is an interesting demonstration of the use of multiple mice, controlled by children on an interactive whiteboard. The collaborative application uses Microsoft's Multi-Point technology. For more information and free downloads, go to MouseMischief.org.

The Internet of Surfaces? Microsoft's Pete Thompson discusses screens and surfaces of all sizes.

I came across this video of Microsoft Surface's general manager, Pete Thompson, on the GottaBeMobile website. According to Thompson, the people who worked with TouchWall discussed in the video, were also involved with Microsoft Surface.

Apparently, the Surface and Wall folks at Microsoft aren't sure of what they are doing with screens of all sizes.

(If you are interested in surface form factors, see my previous blog post, Emerging Interactive Technologies, Emerging Interactions, and Emerging Integrated Form Factors.)

There are many unanswered questions from my perspective.

Bill Gates portends that every surface will be a computer, a concept that is echoed in the video. If so, what are people doing to ensure that surface-supporting environments are universally designed?

Before Microsoft and other companies unleash "surface" technology to the masses, they must get a few things right.

Will they?

Will the researchers at Microsoft find out how various screens play out in classrooms, in the community settings, on-the job, and in-between?

For example, large interactive displays in urban and retail settings have the potential to provde people with a rich amount of information about what is around them. These displays serve little purpose if they are user-unfriendly, and no purpose at all if they are not accessible.

If developers and designers are not following basic user-centered design guidelines and usability standards now, how can we expect the nextgen systems of display surfaces to support universal usability? As our population ages, this will be more of a problem.

From what I can tell, there will be more opportunities for people to use their mobile devices to interact with larger screens and surfaces when they are out and about. For example, when I was at the airport recently, I noticed that there was a large display that offered cell-phone ringtone downloads. Microsoft was behind this display.

Interconnectivity and interoperability between devices and screens of all sizes is important to think about. If universal usability guidelines are not followed, our mobile devices will be difficult to use in the world of surfaces.

It isn't much of a leap to see the big picture. Just think about the problems we have with our remote controls and entertainment centers in our homes! We might have to carry all sorts of devices just to get from point A to point B. I'm not kidding. It will not be a pretty sight, especially if the privacy and security issues are not resolved as we move to a world that supports the internet of things.

Or shall I say, the internet of surfaces?

Off-topic: Now I know why some people think I'm a Microsoft shill!

A while ago, I posted a response on another blog, and someone mistakenly thought that I was a Microsoft shill. Me?!

Now I know why.



When I was looking at my blog stats, I noticed that one of my visitors found a recent post via a Google search on microsoft new interactive technology. Curiously, the post was at the very top of the list:

New Interactive Technology: Tag Galaxy, Windows 7, PaperVision 3D, Microsoft Surface, Touch Screens, and Blog Posts Revisited

Results 1 - 10 of about 29,600,000 for microsoft new interactive technology. (0.12 seconds)

How did my post beat out over 29 million pages on the web?!

I've had an Apple computer, in one form or another, since 1985. When I decided to take a programming class a few years ago, I soon realized I had to adapt to the Windows world. I like C#, Silverlight, WPF, and XNA Game Studio. This does not make me a Microsoft shill.

I just happen to be overly passionate about interactive displays of all sizes, and fascinated with the possibilities of mult-touch. iPhones, touch-tables, touch-walls, devices that interact with displays (think Johnny Chung Lee), gesture + touch, urban screens, interactive exhibits, interactive way-finding... and the multimedia applications that support this sort of interaction.

I've written several posts about Apple's new technologies, like the iPod Touch and the new 3G iPhone. I'll be writing more about Apple as they research and develop flat panel displays:

"Apple is looking for a Senior Panel Process Engineer to lead the engineering activities to develop the new process and design for the multi-touch panel used in Apple products. This individual will serve as a focal point in the designing and the process development of advanced multi-touch panels from the concept to the product ramp."

I'd say I'm more of a cheerleader for all of those out there who are working so hard to create useful+cool technology for the future.

Update:
The Next Generation HP TouchSmart PC
The new TouchSmart PC's, which can display high definition content, aren't out on the market yet. HP is taking pre-orders.

NextWindow, a company that produces large interactive touch-screen displays, including those with multi-touch capabilities, was responsible for the touch-screen embedded in the TouchSmart PC.

Related:

HP Redefines Home Computing, Putting the Digital Lifestyle at People's Fingertips with New TouchSmart PC's

HP Introduces World's First Affordable Color-critical Display

Windows 7 Demo on a Dell laptop: More Multi Touch

Video: Multi-Touch in Windows 7

Via SoapBox, Greenbush Labsand CNET

If you are new to this blog and would like to learn more about multi-touch interaction and technology, enter a keyword in the search box.

Also take a look at the Technology Supported Human-World Interaction blog.

For multi-touch DYI, check out the NUI Group!

FireFly Game for Microsoft Surface: Demo by Carbonated Games; A look at a Multi-touch from Next Window

Watch the video clip to see more about Firefly game that was developed by Carbonated Games to take advantage of Microsoft's multi-touch enabled Surface table computer system. The game is in demo form only.For more information, see the article from Gaming Today.

For those of you who need multi-touch before you can get your hands on the Surface, you might be interested in NextWindow's multi-touch display, recently demonstrated at an expo:



Here are links to touch-screen project prototypes I worked on for HCI and Ubicomp classes I took in 2007:
poetry picture share photo-globe explorer

Cross Post: Microsoft Research SenseCam

From the Microsoft Research SenseCam Website:

"SenseCam is a wearable digital camera that is designed to take photographs passively, without user intervention, while it is being worn. Unlike a regular digital camera or a cameraphone, SenseCam does not have a viewfinder or a display that can be used to frame photos. Instead, it is fitted with a wide-angle (fish-eye) lens that maximizes its field-of-view. This ensures that nearly everything in the wearer’s view is captured by the camera, which is important because a regular wearable camera would likely produce many uninteresting images."

Technology Review Article

The SenseCam application has the potential to be a resource for people who have developmental delays, traumatic brain injury, severe attention deficits, and autism spectrum disorders. It would be a great tool for special educators, occupational and speech/language therapists, and rehabilitation specialists.

Images from Microsoft Research:

Microsoft UK: "ThinSight" Multi-touch Technology on a Laptop

(Photos from New Scientist)

Tom Simonite's article in a recentNewScientist article reviews a prototype for a touch-screen lap-top using "ThinSight", developed by Microsoft Research. Other companies are researching similar systems.

Multi-touch screens and surfaces support the work of people who increasingly rely on visualization or interactive multimedia to do their jobs. In addition, 3D multi-touch applications created for laptops and tablet PC's might be good for education, training simulations, and serious games.

Video: Photosynth Demonstration

Microsoft Live Labs: Photosynth

"The Photosynth Technology Preview is a taste of the newest - and, we hope, most exciting - way to view photos on a computer. Our software takes a large collection of photos of a place or an object, analyzes them for similarities, and then displays the photos in a reconstructed three-dimensional space, showing you how each one relates to the next.

In our collections, you can access gigabytes of photos in seconds, view a scene from nearly any angle, find similar photos with a single click, and zoom in to make the smallest detail as big as your monitor."

It looks like PhotoSynth has the potential to be used in creative, engaging ways in educational settings, especially for visual learnings. Imagine what students could do if given the opportunity to interact with PhotoSynth content on large screen displays!

Yet another video about an interactive touch-table desk with potential for educational settings

Here is yet another video of a touch table from Microsoft's Center for Information Work that I think has great potential for use in education settings and for "serious games".

Microsoft appears to be targeting business, retail, and entertainment for new "surface computing" applications. I'd like to see Microsoft focus some of their efforts with this technology on education, as I think this technology would support "engaged learning". It might also inspire young people to consider future careers in technology. Given that we have a high school dropout-rate of about 30% in the US, we don't have too much time to waste.


This system allows for hands-free search via voice, which would be great for people who have physical disabilities.

As a school psychologist, I can see many uses for this sort of set-up to support what I do at work. I can envision interactive assessments on this table- if you work in education or psychology, or if you are interested in this sort of technology, leave a comment!

Microsoft's Center for Information Work, the group working on the technology behind the table shown in the video clip.

For a glimpse of some of the work I did last semester using touch-screen displays, take a look at the following posts:

Touch Screen Poetry Picture Share

Google Earth with photo-overlays

Google Earth photo-overlay on a touch screen

Microsoft Surface multi touch screen table - I wish I had one for my projects last semester!

Microsoft Research recently unveiled Microsoft Surface: http://microsoft.com/surface

This multi - touch table can be used for a variety of applications, as outlined in the video from CNET and YouTube below:



I'd like to work on applications for use on a touch-table to support students with special needs, especially those who have autism spectrum disorders.

Last semester, I worked on prototype applications for use on a touch-screen surface -here is a photo. We used a NextWindow Human Touch large-screen display, which provided great screen resolution and touch-response.






This application was part of a travel-planning prototype developed for a course in Human-Computer Interaction. The application was demonstrated on a NextWindow Human Touch large screen display. Would it work on the iPhone?





Update: Examples from some of my other posts:


Here is another demo videoclip of a globe created in GoogleEarth using photo-overlays, with links to video clips uploaded to YouTube and embedded in individual posts on a blog. The above photo and the video clip show the application on a NextWindow Human Touch large-screen display.

This application would be great on a touch-table or touch-table set up on a drafting board. Although it was designed for a travel-planning application, it would work well in educational settings in subjects such as geography.


Poetry Picture Share



This was my first attempt at a "poetry picture share" application. It was designed for use on a multi-touch table and can be accessed remotely so people in different places can move things around on the screen. The video shows how the application works on a NextWindow Human Touch interactive large-screen display.

I am planning on adapting this application for use with students with special needs, such as those who have autism or other communication disorders.