Showing posts sorted by relevance for query gesture. Sort by date Show all posts
Showing posts sorted by relevance for query gesture. Sort by date Show all posts

Jan 28, 2011

"Microsoft is Imagining a NUI Future". You can, too!

Microsoft is Imagining a NUI Future
Steve Clayton, Next at Microsoft Blog, 1/26/11


"Our research shows that the vast majority of people polled in both developed and emerging markets see great potential for NUI applications beyond entertainment. This is especially true in China and India, where 9 out of 10 respondents indicate they are likely to use NUI technology across a range of lifestyle areas – from work, education and healthcare, to social connections, entertainment and the environment. We believe that taking technology to the next billion can be aided by NUI – making technology more accessible and more intuitive to a wider audience". - Steve Clayton, Microsoft


The people at Microsoft don't own the concept!  I'm a member of the NUI Group (May, 2007) and SparkOn.  Both are on-line communities where you can find people who live and breathe NUI, learn about their work, and even share designs and code. If you are intrigued by NUI - as a designer, developer, or user, please join us.


Note: 
I've been an evangelist and cheerleader for the NUI cause for many years.  If you search this blog for "post-WIMP", "NUI", "multi-touch", "gesture", "off-the-desktop""natural user interaction", "natural user interface", or even "DOOH", you'll be provided with an overwhelming number of posts that include videos, photographs, and links to NUI-related resources, including scholarly articles.  There is a small-but-growing number of people from many disciplines, quietly working on NUI-related projects.


RELATED
Microsoft Plans a Natural Interface Future Full of Gestures, Touchscreens, and Haptics
Kit Eaton, Fast Company, 1/26/112
Rethinking Computing (video)
Craig Mundie, Microsoft
Interactive Touch-Screen Technology, Participatory Design, and "Getting It" - Revised
Touch Screen Interaction in Public Spaces:  Room for Improvement, if "every surface is to be a computer".

Jan 10, 2011

ImageFlow for streaming image search; Content-Aware Dynamic Timeline for Video Browsing, and more from Microsoft Research

I thought I'd share a couple of interesting videos about the work of some researchers at Microsoft Research and their colleagues:



RELATED
Japani, V., Ramos, G., and Drucker, S.  ImageFlow: Streaming Image Search (pdf) (Microsoft Research Publication, 11/18/2010)
Link to abstract

Fisher, D., Drucker, S., Fernandez,R. and Ruble,S. Visualizations Everywhere: A Multiplatform Infrastructure for Linked Visualizations, in Transactions on Visualization and Computer Graphics, IEEE, Salt Lake City, UT, November 2010


The following video and article focuses on some research that I think will be very useful for creating interactive multimedia timelines in the future.  I'd like to learn more about ways this interaction can be implemented in gesture-based systems!





Content-Aware Dynamic Timeline for Video Browsing
From Gonzalo's YouTube channel: "When videos have more frames than pixels in the player's timeline slider, frames become inaccessible and scrolling actions cause sudden jumps in a video's continuity. We propose a content-aware dynamic timeline control that decouples video speed and playback speed and allow salient shots to be presented at an intelligible speed."


Article
Pongnumkul, S., Wang, J., Ramos, G., Cohen, M.  Content-Aware Timeline for Video Browsing (pdf). UIST '10, ACM


Note: The researchers are investigating the use of audio/sound in a similar manner, making this form of video browsing/search truly multimedia. An example of this is included near the end of the above video clip.


The following video is a demonstration of something I could use in my work as a school psychologist, since I use a lot of video for assessing students with disabilities, including those with autism.  I also use video to create digital social stories for many of the students.

Dec 22, 2010

Multi-touch SmartBoard! (SMARTBoard 800 Series)

Take a look at the video demonstration of the new SMARTBoard (800 series) that offers multi-touch and gesture interaction support so that two students can interact with the board at the same time.

  • Students can use 2 finger gestures to enlarge objects and move them around.
  • Two students can interact with the board at the same time to complete activities.
  • SMARTInk/Calligraphic Ink creates stylized print as you write. Whatever is written or drawn on the SMARTBoard becomes an object in the SMARTNotebook, allowing for things to be resized or rotated.   (2:04)
  • Multi-touch gestures enabled in Window 7 and Snow Leopard work with the SMARTBoard.
  • Software development kit (3:28):  Example of a physics application developed by a 3rd-party developer.  The application supports two students working at the SMARTBoard at the same time
This video, in my opinion, does not provide viewers with the full range of possibilities that the new features provide.   I'd like to see a "redo" of this video using a live teacher and a group of students.  For example, it would be interested in seeing how the physics application would be incorporated into a broader lesson or science unit.   I'd love to hear what real students have to say as they interact with the physics application, too.

Comment:
I think a multi-user interactive timeline would be a great application for the new SMARTBoard, because students could work together to create and recreate events.  This would be ideal for history, literature, and humanities activities, across a wide span of grade levels.

Dec 6, 2010

UPDATE: Demo 2 of the Kinect Theramin, Therenect, by Martin Kaltenbrunner

I recently posted about the Therenect, a gesture-controlled digital theremin created for Microsoft's Kinect, created by Martin Kaltenbrenner - Therenect: Theremin for the Kinect! (via Martin Kaltenbrenner)  It looks like Martin has been busy polishing up the application over the past few days, as you can see from the video below:

Therenect - Kinect Theremin - 2nd Demo from Martin Kaltenbrunner on Vimeo.

RELATED
Virtual Theremin Made with Kinect; Real Thereminists Will Make it Useful
Peter Kirn, Create Digital Music, 11/30/10

Nov 29, 2010

International Conference on Multimodal Interaction: ICMI 2011 Call for Papers

The information below was taken from the website for the 13th International Conference on Multimodal Interaction. I'm excited about the range of topics that the conference will cover.  I look forward to sharing more about the work of the members of this group on this blog in the future!  (I've highlighted the topics that interest me the most.)

INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION CALL FOR PAPERS

The International Conference on Multimodal Interaction, ICMI 2011, will take place in Alicante (Spain), November 14-18, 2011, just after the ICCV 2011 (in Barcelona, Spain). This is the thirteenth edition of the International Conference on Multimodal Interfaces, which for the last two years joined efforts with the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI 2009 and 2010). Starting in this edition the conference uses the new, shorther name.

The new ICMI is the premium international forum for multimodal signal processing and multimedia human-computer interaction. The conference will focus on theoretical and empirical foundations, varied component technologies, and combined multimodal processing techniques that define the field of multimodal interaction analysis, interface design, and system development. ICMI 2011 will feature a single-track main conference which includes: keynote speakers, technical full and short papers (including oral and poster presentations), special sessions, demonstrations, exhibits and doctoral spotlight papers. The conference will be followed by workshops. The proceedings of ICMI 2011 will be published by ACM as part of their series of International Conference Proceedings and will be also distributed to the attendees in USB memory sticks.


Topics of interest include but are not limited to:

  • Multimodal and multimedia interactive processing
    Multimodal fusion, multimodal output generation, multimodal interactive discourse and dialogue modeling, machine learning methods for multimodal interaction.
  • Multimodal input and output interfaces
    Gaze and vision-based interfaces, speech and conversational interfaces, pen-based and haptic interfaces, virtual/augmented reality interfaces, biometric interfaces, adaptive multimodal interfaces, natural user interfaces, authoring techniques, architectures.
  • Multimodal and interactive applications
    Mobile and ubiquitous interfaces, meeting analysis and meeting spaces, interfaces to media content and entertainment, human-robot interfaces and interaction, audio/speech and vision interfaces for gaming, multimodal interaction issues in telepresence, vehicular applications and navigational aids, interfaces for intelligent environments, universal access and assistive computing, multimodal indexing, structuring and summarization.
  • Human interaction analysis and modeling
    Modeling and analysis of multimodal human-human communication, audio-visual perception of human interaction, analysis and modeling of verbal and nonverbal interaction, cognitive modeling.
  • Multimodal and interactive data, evaluation, and standards
    Evaluation techniques and methodologies, annotation and browsing of multimodal and interactive data, standards for multimodal interactive interfaces.
  • Core enabling technologies
    Pattern recognition, machine learning, computer vision, speech recognition, gesture recognition.

Important dates

Workshops proposalMarch 1, 2011
Paper and demo submissionMay 13, 2011
Author notificationAugust 5, 2011
Camera ready deadlineSeptember 2, 2011
ConferenceNovember 14-16, 2011
WorkshopsNovember 17-18, 2011


General Chairs

Hervé Bourlard (Idiap)
Thomas S. Huang (Univ. of Illinois)
Enrique Vidal (Tech. Univ. of Valencia)

Program Chairs

Daniel Gatica-Perez (Idiap)
Louis-Philippe Morency (Univ. South. California)
Nicu Sebe (Univ. of Trento)

Demo Chairs

Kazuhiro Otsuka (NTT Comm. Sci. Lab.)
Jordi Vitrià (UB/CVC, Barcelona)

Workshop Chairs

Fernando de la Torre
(Carnegie Mellon Univ.)
Alejandro Jaimes (Yahoo! Research, Barcelona)

Publication Chair

Jose Oncina (Univ. of Alicante)

Student & Doctoral Spotlight Chair

Li Deng (Microsoft Research and Univ. of Washington)

Sponsorship Chair

Nuria Oliver (Telefónica I+D)

Publicity Chair

Helen Mei-Ling Meng (CUHK, Hong Kong)

Local Organization Chair

Luisa Micó (Univ. of Alicante)

Treasurer

Jorge Calera (Univ. of Alicante)

Local organizers

Xavier Anguera (Telefónica I+D)
A. Javier Gallego Sánchez (Univ. of Alicante)
Ida Hui (CUHK, Hong Kong)
Jose Manuel Iñesta (Univ. of Alicante)
Alejandro Toselli (Tech. Univ. of Valencia)



RELATED
Accepted Papers for ICMI-MLMI 2010


NOTE:  ICMI 2011 will be held after ICCV 2011, the 13th International Conference on Computer Vision in Barcelona.

Nov 11, 2010

Interactive Touch-Screen Technology, Participatory Design, and "Getting It", Revisited

I've been planning on updating one of my popular posts, "Interactive Touch-Screen Technology, Participatory Design, and Getting It" for a while. 


Here is a compromise - since much still rings true two years since I wrote it, the bulk of this post remains the same.   I've updated a few sections with additional video examples of interactive touch-screen applications, good and bad, along with a few links and resources, located at the bottom of the post. 


(The missing piece of information?  An update about apps for the iPad and similar touch-screen tablets.)

Sit back and enjoy!


http://www.ehomeupgrade.com/wordpress/wp-content/uploads/hp_touchsmart_pc.jpghttp://www.wired.com/images/article/full/2008/08/han_interview_630px.jpg
-Images: HP; Wired

There's been some discussion over the reasons why so many people don't understand touch screen, or "surface" computing, even though research in this area has been going on for years.

As the new owner of the HP TouchSmart, I know that I get it.

The research I've conducted in this area suggests that people will "get-it" only if there is a strong commitment to develop touch-screen "surface" applications through a user-centered, participatory design process. In my view, this should incorporate principles of ethnography, and ensure that usability studies are conducted outside of the lab.


This approach was taken with
Intel's Classmate PC. Intel has about 40 ethnographic researchers, and sent many of them to work with students and teachers in classrooms around the world. (A video regarding ethnographic research and the Intel Classmate project can be found near the end of this post.)

http://download.intel.com/pressroom/kits/events/idffall_2008/images/Picture007.jpghttp://www.classmatepc.com/images/advocateImage.jpg
-Images: ClassematePC


Where to start?
K-12 classrooms and media centers. Public libraries. Malls. Hospital lobbies and doctor's offices. Any waiting room. Staff lounges in medical centers, schools, and universities. Community festivities and events. Movie theater lobbies. Museums and other points of interests.


I believe we need to take a "touching is believing" approach.

Here are some thoughts:
When I try to explain my fascination with developing touch-screen interactive multimedia applications, (interactive whiteboards, multi-touch displays and tables, and the like), many of my friends and family members eyes glaze over. This is particularly true for people I know who are forty-ish or over.

Even if you are younger, if you never saw the cool technology demonstrated in the movie Minority Report, or if you have limited experience with video games, or if you haven't came within touching distance of an interactive whiteboard, the concept might be difficult to understand.


The reality?
Even people who have the opportunity to use surface computing technology on large screens do not take full advantage of it. Multi-touch screens are often used as single-touch screens, and interactive whiteboards in classrooms are often serve as expensive projector screens for teacher-controlled PowerPoint presentations.


Most importantly, there are few software developers who understand the surface computing approach, even with the popularity of the iPhone and iPod Touch. Most focus on traditional business-oriented or marketing applications, and have difficulty envisioning scenarios for surface computing.  There is a need for a breath of fresh air!

Another factor is that not all people entrusted to market surface or touch screen computing fully understand it.
http://blogs.msdn.com/blogfiles/healthblog/WindowsLiveWriter/MicrosoftHUGWishyouwerehereDay2_82D3/IMG_0550_thumb.jpg
Despite a cool website showing off the goods, Microsoft's Surface multi-touch table has been slow to take off, limiting hope of bringing down the price tag to a price most families or schools could afford. (The picture above depicts an application for the Surface designed for health care professionals, not K-12 science education.)

Although you can't buy a Surface table for your family room, it is possible to buy a TouchSmart.  
HP's TouchSmart website is engaging and highlights some examples of touch-screen interaction, but most people don't seem to know about it. (Since this post was last written, there are many more touch and multi-touch options available to the public, such as the  Dell, the iPad, etc.)

Unfortunately, you wouldn't have a clue that the HP TouchSmart exists browsing the aisles at Circuit City or Best Buy!

When I was shopping for my new TouchSmart, I noticed that from a distance, the TouchSmart looked just like the other larger flat-screen monitors filling up the aisles. The salespeople at both stores were not well-informed about the system. The only reason I knew bout the new TouchSmart was related to my obsession with interactive multimedia touch-screen applications- designing them, developing them, studying them, reading about them, blogging about them.... ; }

More thoughts:

After studying HCI (Human-Computer Interaction), and relating this knowledge to what I know as a psychologist, my hunch is that the "Window Icon Mouse Pointing-device" (WIMP) and keyboard input mind-set is embedded in our brains, to a certain extent. Like driving a car, it is something automatic and expected. This is true for users AND developers. (Update - See The Post WIMP Explorers' Club: Update of the Updates for a review of a discussion among passionate post WIMP folks)

Think about it.

Suppose one day, you were told that you no longer were allowed to control your car by turning on the ignition, steering the wheel, or using your feet to accelerate, slow down, or stop the car! Instead, you needed to learn a new navigation, integration, and control system that involved waving your hands about and perhaps speaking a few commands.

For new drivers who'd never seen a car before, this new system would be user-friendly and intuitive. Perhaps it would be quite easy for 16-year-old kids to wrap their heads around this concept. For most of us, no. Imagine the disasters we would see on our streets and highways!

When we think about how newer technologies are introduced to people, we should keep this in mind.

In my mind, spreading the word about surface computing is not a "if you build it, they will come" phenomenon, like the iPhone. We can't ignore the broader picture.

From my middle-age woman's vantage point, I believe that it is important that the those involved with studying, developing, or marketing surface computing applications realize that many of us simply have no point of reference other than our experiences with ATMs, airline kiosks, supermarket self-serve lanes, and the like. (The video clip at the very end of this post provides a good example of touch-screen technology gone wrong.) -UPDATE: additional videos were added to this post.

Be aware that there are substantial numbers of people who might benefit from surface computing who prefer to avoid the ATMs, airline kiosks, and self-serve grocery shopping.

Realize that the collective experience with technology, in many cases, has not been too pretty. Many people have had such user-unfriendly experiences with productivity applications, forced upon them by their employers, that any interest or desire to explore emerging technologies has been zapped.

My own exposure to interactive "surface" related technology was somewhat accidental.
A few years ago, a huge box was deposited into the room I worked in a couple of days a week as a school psychologist at a middle school. After a week or so, I became curious, and found out that it was a SmartBoard. Until then (2002!), I did not know that interactive whiteboards existed.

The boxed remained unopened in the room for the entire school year, but no worry. I played with the only other SmartBoard in the school, and found a couple at the high school where I also worked. I hunted for all of the applications and interactive websites that I could find, and tried them out. That is when I was hooked. I could see all kinds of possibilities for interactive, engaging subject area learning activities. I could see the SmartBoards potential for music and art classes. With my own eyes, I saw how the SmartBoard engaged students with special needs in counseling activities. (By the way, if you are working with middle school students, PBS Kid's ItsMyLife website activities work great on an interactive whiteboard.)

A few years have passed, and reflecting on all of my fun experiences with interactive whiteboards, with and without students, I now understand that many teachers still have had limited exposure to this technology.

This school year, many teachers are finding themselves teaching in classrooms recently outfitted with interactive whiteboards, scrambling along with educational technology staff development specialists, to figure out how it works best with various groups of students, and what sort of changes need to be made regarding instructional practice.


For the very first time, interactive whiteboards were installed in two classrooms at one of the schools I work at. One of the teachers I know thanked me for telling her about interactive whiteboards and sharing my resources and links.

If I hadn't let her know about this technology, she wouldn't have volunteered to have one installed in her classroom. It has transformed the way she teaches special needs students.

In the few months that she's used the whiteboard, I can see how much it has transformed the way the students learn. They are attentive, more communicative, and engaged. The students don't spend the whole day with the whiteboard - the interactive learning activities are woven into lessons at various times of the day, representing true technology integration.

Now let's see what happens when all-in-one touch-screen PC's are unleashed in our schools!

UPDATE:  Take a look at a post I wrote for Innovative Interactivity just after SMART Technologies acquired NextWindow - the post describe in detail how interactive whiteboards are transforming learning and teaching in a program for students with special needs.
SMARTTechnologies Acquires NextWindow: A "smart window" to the world


There are some interesting changes going on at the intersection of HCI and educational technology research.  I participated in a workshop at CHI 2010 last April and was impressed by what is going on in this area, around the globe:   Next Generation of HCI and Education

Value of ethnographic research:
Ethnographic Research Informed Intel's Classmate PC
"Intel looked closely at how students collaborate and move around in classroom environments. The new tablet feature was implemented so that the device would be more conducive to what Intel calls “micromobility”. Intel wants students to be able to carry around Classmate PCs in much the same way that they currently carry around paper and pencil." -via Putting People First and Ars Technica

The video below is from Intel's YouTube Channel. Information about Intel's approach to ethnographic research in classrooms during the development of the Classroom PC is highlighted. This approach uses participatory design and allows the set of applications developed for the Classmate PC to reflect the needs of local students and teachers. Schools from many different countries were included in this study.




FYI: TOUCH SCREEN DISPLAYS:  NEED FOR IMPROVEMENT!

Touch Screen Coke Machine at the Mall: 90 Seconds to get a Coke


User-Unfriendly Interactive Display in the Rain (Ballantyne Village Shopping Center)

User-Unfriendly Information Kiosk Interactive Map
I encountered this puzzling and frustrating interactive directory/map at the Cleveland Clinic.  When I went to visit a relative at the hospital a year or so later, the map was no longer there.


BETTER EXAMPLES OF INTERACTIVE SCREENS:
Here are some interesting pictures from lm3labs, which are in my interactive usability hall of fame:

http://catchyoo.typepad.com/photos/uncategorized/2008/06/30/4654.jpghttp://farm3.static.flickr.com/2172/2233673451_6a48db8bff.jpg?v=0


Samsung's new Omnia SDG i900 was re-created in a much larger size, using lm3lab's Ubiq'window touchless technology.For more about lm3labs, including several videoclips, take a look at one of my previous posts: Lm3Labs, Nicolas Leoillot, and Multimedia Interaction

Midwife Toad App on a Microsoft Surface, Discovery Place Science Center


TellTable:  Digital Storytelling on the Surface:  Microsoft Research, UK


DECEMBER 31, 2009 -Interactive Soda Machine for Fun

The interactive screen on the Coke machine attracted the attention of this young child. He loved spinning the image of the bottle. So did the dad! He said, "I'd like something like this for my home!". I told him about the HP TouchSmart - both the dad and the mom did not know that there were affordable all-in-one touch screens available, but they knew about SMARTboards, because their children's classrooms had them.  Note:  No one from this family actually purchased a soft drink.  I was hoping to time how long it would take them to do so!



Some resources:
lm3labs (catchyoo, ubiq'window)
NUITeqNUI Group (See member's links)
Sparkon (See members links and multi-touch projects)

(More information and resources can be found by doing a "multitouch" or related search on this blog or The World Is My Interactive Interface.)

If you have plenty of time, take a look at my Post WIMP Explorers' Club YouTube playlist.
"Natural user interfaces, gesture interaction, multi-touch, natural interaction, post WIMP examples and more..."

FYI: I visited the Ballantyne Village shopping center a couple of months ago to follow up on the interactive displays, including the one I tried to use while it was raining.   The shopping center changed ownership, and the displays were replaced by the old-fashioned kind, pictured below:



Oct 12, 2010

Oh! No! Sony's "Mother of Remote Controls" for Google TV. 74 Buttons and Counting.

Today we switched from DISH to Time Warner Cable, and tonight I had to battle with a new remote control, the UR5U-8780L.  The experience with this remote led me to search for something better. What a coincidence!  In this day and age of touch-screens,  I was hoping for something better than....


Sony's Mother of Remote Controls!

-From the SonyStyle website: Television, meet internet.

I first learned of this complex addition to the world of TV/Internet surfing from an article and a video in a recent article in Engadget:  Sony's Google TV controller outed on ABC's Nightline (video) Ross Miller,10/5/10.  Harry Brignull also posted about the new controller- Sony, Sony, what have you done?
(Harry is a UX Consultant at Madgex, and author of the 90percent of everything blog.)



ABC Video, via engadget

It is 2010, and with TVs connected to the internet, we'll be interacting with content in ways we could only dream of in the recent past.  Interactive TV is here.  Do I really have to push a lot of buttons in order to have the best "interactive" experience?  




OTHER OPTIONS
Xfinity Remote Prototype for the iPad


Turn Your iPhone into a TV Remote Samuel Axon, Mashable/Apple
L5 Remote: Turn your iPhone or iPod touch into a universal remote control:
L5 remote


Not Yet Available:  Vizio's XRT100 touchscreen remote
Vizio_touch_remote.jpg

My Fancy New Remote, Instructions Included:

Sep 22, 2010

PD-NET Project: "Exploring..large scale networks of pervasive public displays..."

I love the concept of interactive, networked public displays!   The PD-NET project has a fairly new website and a Facebook page.  If this interests you, take the time to learn about the PD-NET project, a collaborative effort between researchers from several different universities in Europe. I'd like to see this take hold in the US.  


PD-NET Project Objectives (From the PD-NET Website):
  • To create enabling technologies for large-scale pervasive display networks through the design, development, and evaluation of a robust, scalable, distributed and open platform for interconnecting displays and their sensors.
  • To establish Europe as the international centre for work on pervasive display networks.
  • To address key scientific challenges that may inhibit the widespread adoption of pervasive display network technology:  Tensions between privacy and personalization, situated displays, business and legislative requirements, User Interaction.
Here is a list of participating universities and researchers, taken from the PD-NET website:


Participants from Lancaster University (UK):
Participants from University of Duisburg-Essen (Germany):
Participants from University of Lugano (Italy):
Participants from University of Minho (Portugal):
Publication
J. Müller, F. Alt, D. Michelis, and A. Schmidt, "Requirements and Design Space for Interactive Public Displays," in Proceedings of the 18th annual ACM international conference on Multimedia (Multimedia 2010), to appear, 2010.

COMMENT
If you are familiar with this blog, you know that I am passionate about interactive displays, especially in public spaces.  I've devoted numerous posts to this topic on this blog, and also on  The World Is My Interactive Interface, a blog I started a few years ago when I was working on projects for large interactive displays.  


I am interested in how interactive displays, of any size, can be accessible and universally designed, especially those that can inter-operate with mobile devices, including devices that support communication for people with special needs.


SOMEWHAT RELATED
Here are links to some of my previous blogposts related to the topic of interactive displays in public spaces.  Some have links to scholarly publications.  
What is DOOH and Why Should We Care? (DOOH- Digital Out-Of-Home)
Interactive Displays and Interaction (Presentation via Daniel Michelis)
Thoughts about technology on a cruise ship, and other reflections
Multi-touch and gesture interaction out-and-about
UPDATED: More News, Videos, and Links about Multi-Touch and Screen Technologies
Technology-Supported Shopping and Entertainment User Experience at Ballantyne Village:  "A" for concept, "D" for touch screen usability.
Usability/Interaction Hall of Shame (In a Hospital)
Think Globally, Act Locally:  Exploring the Problem Space - Top-down, bottom-up, local and the global...