Frequently asked questions

From NUI Group Community Wiki

Jump to: navigation, search

See also:

Contents

About NUI Group

What is NUI Group?

Natural User Interface or ~ NUI Group is an interactive media group researching and creating open source machine sensing techniques to benefit artistic and educational applications. http://nuigroup.com/log/about/

What are your activites?

Currently most of our discussions take part in our forums and partly in this wiki. There's an ongoing plan to use a content management system in order to foster NUI Group community and enable active participation of its members.

How can I help your group?

Please check our Contribution section for information on contributing.

How can I promote your group?

Post your videos on Youtube / Vimeo and mention multitouch and NUI Group in the title, as well as tags. You may want to put the link to our community on other community forums that share similar content to NUI Group (ie. technology, multimedia, interface websites/forums). Furthermore, you can help promote our community by placing our logo and promotional link (found at http://nuigroup.com/forums/viewthread/492/ ) on your blog and/or website.

Multitouch

What is multitouch?

Even if the prefix MULTI stands for 2 or more, multitouch technology often refers to a screen that allow users to interact with a computer system with N number of fingers, therefore allowing for multiple object manipulation, gestures and multiple users interaction. - lolec

For a list of multitouch related terms, go to Multi-Touch Terminology link.

What are the possible ways of making a multitouch device?

There are many ways to create multitouch devices. While commercial technologies tend to rely on more expensive approaches like capacitive sensing, the techniques widley used at NUIGroup are considered optical, or vision techniques. To find the four main do-it-yourself techniques (FTIR, DI, LLP, DSI) used on the forum, please see Getting Started With Multitouch.

There's also an illustration on the forum which shows different kinds of multitouch systems.

What is a “compliant surface” and why do/don’t I need one?

The compliant surface or compliant layer is simply an additional layer between your projection surface and your acrylic. It enhances the finger contact and gives you more robust blobs, particularly when dragging as your finger will have less adhesion to the surface then.

The compliant surface is needed in order to perform the index of refraction [1] , that means how fast and well the light travels from one medium to another.

With the FTIR technique , the infra red light is emitted inside the acrylic sheet, the lights keeps traveling inside the medium (Frustrated into the acrylic), when you touch it , light escape from it (refraction),creating the so-called blobs (bright luminescent objects )

The compliant layer is only of any use in FTIR displays, if you are building DI, LLP, or other methods, you will not use a compliant layer. If you are building FTIR, a good compliant layer is the difference between a display that works...mostly, and a consistent force display. There is much experimentation ongoing in the quest for the ‘perfect compliant layer’.

Some materials used to success include Sorta Clear 40 and similar catalyzed silicon rubbers, Lexel, and various brands of RTV silicone. Others have used fabric based solutions like silicon impregnated interfacing and SulkySolvy.

The original successful method, still rather popular, is to ‘cast’ a smooth surface directly on top of your acrylic and then lay your projection surface on that after it cures. This requires a material that closely approximates the optical properties of the acrylic as it will then BE a part of the acrylic as far as your transmitted IR is concerned, hence the three ‘rubber’ materials mentioned earlier...they all have a refractive index that is very close to that of acrylic.

Gaining in popularity is the ‘Cast Texture’ method. Tinkerman has been leading the pack in making this a simple process for DIY rather than an involved commercial process. But essentially, by applying your compliant layer to the underside of your projection surface, and texturing it, then laying the result on your acrylic, you gain several benefits. Your compliant surface is no longer a part of the acrylic TIR effect so you are no longer limited to materials with a similar refractive index to that of acrylic, although RTV and Lexel remain the most popular choices, edging out catalyzed silicons here. Since it is largely suspended over the acrylic by the texture, except where you touch it, you get less attenuation of your refracted IR light (brighter blobs).

Fabric based solutions have a smaller following here, and less dramatic of a proven success rate, but are without question the easiest to implement if an appropriate material can be sourced. Basically they involve lining the edges of your acrylic with two sided tape, and stretching the fabric over it, then repeating the process to attach your display surface. - Trackzilla

more infos and how to

Where can I buy silicone rubber sheets or liquid silicone for pouring my own?

Professional silicone like SortaClear 40 can be purchased at molding stores or special effects stores for movies professionals.

Basic clear silicone can be found at paint stores, kitchen & bathroom stores, or at any hardware store. Rubson, DAP, and GE are commonly used brands.

What's a fiducial?

A fiducial or fiduciary marker is often a graphical symbol that is identified by a computer vision system. The marker is known beforehand to the computer vision algorithm and therefore the only task for the computer vision system is to identify and locate the fiducial symbol in the image. Practically speaking, a fiducial symbol is a sticker that needs to be attached to a physical object in order to make it visible to the computer vision algorithm. For an example of such a maker, see: http://reactivision.sourceforge.net/images/reactivision02.png

Multitouch methods and techniques

What is the best multitouch technique?

Generally there is no 'best' technique. This forum thread includes a list of methods and techniques related to building a MT table. You should also evaluate the options yourself using this thread, which describes the pros/cons of each technique, as well as the parts required.

Do I have to use a projector?

Most setups feature some kind of a display on the touch surface. To do this, you can either use a projector or use a LCD. Projectors are preferred for larger touch surfaces, but LCDs are far cheaper for setups less than ~22" (56 cm). If a display on the touch surface is not needed, you do not need a projector or LCD.

What projectors are best?

Throw distance is the main criteria for choosing a projector. This distance will define the size of the setup and the needs for mirrors to reduce it. Other factors to consider are the brightness (measured in ANSI Lumens) and contrast ratio.

How do I check if my LCD has FFC issues?

FFC issues arise when certain connector cables within the monitor are not long enough to allow the LCD matrix to be detached from the circuitry of the monitor. LumenLab, a DIY projector community, has a list of some LCD monitors and whether or not they have FFC issues. You can find it here: LumenLab Database.

What kind of LEDs are best and where can I buy them?

This answer depends mainly on the technique used. Before buying, it's strongly advised to check the data sheet of the LED. Wavelength, angle, and radiant intensity are the most important specifications for all techniques.

Wavelength: 780-940nm. LEDs in this range are easily seen by most cameras and visible light filters can be easily found for these wavelengths. The lower the wavelength, the higher sensitivity which equates to a higher ability to determine the pressure.

Radiant Intensity: Minimum of 80mw. The ideal is the highest radiant intensity you can find, which can be much higher than 80mw.

Angle for FTIR:
Minimum half angle: +/- 48 degrees
Minimum full angle: (48 + 48) = 96 degrees
Maximum half angle: +/- 60 degrees
Maximum full angle: (60 + 60) = 120 degrees

Anything less than +/- 48 will not take full advantage of TIR (total internal reflection), and anything above +/- 48 degrees will escape the acrylic. In order to ensure there is coverage, going beyond +/- 48 degrees is fine, but anything above +/- 60 is really just a waste as (60 - 48 = +/- 12 degrees) will escape the acrylic.

Angle for Diffused Illumination:
Wider angle LEDs are generally better. The wider the LED angle, the easier it is to achieve even illumination. The wavelength and radiant intensity

How many LEDs do I need?

Most people find that one LED per inch works best - this is merely a guideline. The more sides the better, but FTIR can be achieved with as one side of illumination.

What is infrared and why is it needed?

Infrared (IR in short) is a portion of the light spectrum that lies just beyond that than can be seen by humans. It is a range of wavelengths longer than visible light, but shorter than microwaves. ‘Near Infrared’ is the part of the infrared spectrum that is invisible to humans, but still acts like visible light as far as most silicon is concerned, and is typically considered wavelengths of between 700nm and 1000nm. Most digital camera sensors are also sensitive to at least NIR and are often fitted with a filter to remove that part of the light spectrum so they see more like we do and give us the image we normally expect to see. By removing that filter and replacing it with one that removes the visible light instead, we create a camera that only sees what we don’t.

Then we can use normal imaging methods (like projectors or LCD displays) to create a video image for us to see, and also illuminate our display with an infrared light source to create an image of our fingers, hands, or other objects that the camera can see without our display getting in the way of that image.

Of interest, some people, particularly those who have had cataract surgery, can see the beginnings of NIR, reportedly as far as 850nm. Many brands of acrylic sheet are intentionally designed to reduce their IR transmission above 900nm to help control heat when used as windows. Some cameras sensors have either dramatically greater, or dramatically lessened sensitivity to 940nm, which also has a slightly reduced occourance in natural sunlight.

Do you have to use IR LEDs?

IR LEDs are not required, but a IR light source is. In most multitouch optical techniques (especially LED-LP and FTIR), IR LEDs are used because they are efficient and effective at providing infrared light. On the other hand, Diffused Illumination (DI) does not require IR LEDs, per se, but rather, some kind of infrared light source like a infrared illuminator (which may have LEDs inside). Laser light plane (LLP) uses IR lasers as the IR light source.

How do I wire my IR LEDs?

It is recommended to wire your LEDs in parallel, so that if one LED is damaged or malfunctions the other LEDs in the line will continue to function. To learn how to wire LEDs refer to the How to Wire LEDs NUI Group wiki page.

Is drilling the acrylic for LEDs better than putting them next to polished edge? (FTIR)

It’s not worth the effort. By putting the LEDs inside the acrylic you’re essentially changing the LED lens (angle). Might as well just use the right LEDs and not deal with drilling the holes. Polishing the edge and putting the LEDs flush against it will be enough to provide good results.

What kind of a camera do I need?

See here for information on the type of camera to use. Usually the goal is to find one that can be easily modified (i.e. removing the IR blocking filter and adding a visible light blocking filter).

Is camera resolution an important factor?

Camera resolution is independent of projector resolution, therefore one can use any camera resolution. Calibration is what transforms the camera space to screen (projector) space.

Is direct sunlight an important factor?

The techniques used on the forum don’t have to be used in the dark. They just can’t be used in direct sunlight (unless you’re doing front DI). You can have lights on as long as there isn’t more IR light in the ambient room than in the technique you’re using then it works just fine with the lights on. There are window films that can block some IR, but not all. They also can degrade the image quality.

What does diffusion mean and what is a diffused surface or diffuser?

Diffused reflection is light that reflects in multiple directions off an uneven surface. A diffused surface is the surface that the light is reflecting off of.

In multi-touch tables it can be used in a number of ways:

  • In a projector based setup, the projection surface is sometimes called the diffuser, because it stops a lot of the projected image's light.
  • In an LCD based setup, a diffuser is needed to go under the LCD screen so that the back light can evenly cover the screen. Common materials used are vellum, tracing paper, or one of the diffusion layers from the back light.
  • In a Rear Diffused Illumination (Rear DI) setup, a diffuser is placed on the touch surface (often the projection layer), IR light is projected out through the touch surface and when it hits your finger, is reflected back to the camera. The camera sees the diffused light off your finger as being brighter and thus makes a blob out of it.
  • In a Front Diffused Illumination (Front DI) setup, a diffuser is placed on the touch surface (often the projection layer), IR light is projected from above the touch surface and creates shadows when you touch the surface. The camera sees the shadows off your finger as being darker than the diffusion layer, and after reversing the image, the touch is brighter, and thus makes a blob out of it.

What kind of light illuminator is best and where can I buy one?

What type of projection surface is best?

For a comparison of projection surfaces for rear DI illumination, see comparison of rear projection films entry on Wiki.

Mirrors / My projector doesn’t produce a big enough image, what do I do?

There is a process called mirror folding, where you use mirrors to allow the project to have a greater throw distance in a smaller space. Use this site to help with placement of mirrors in your setup.

My projector emits infrared light and the camera is seeing it on the touch surface, what do I do?

To fix this issue, you’ll need a "hot mirror." A hot mirror, typically a piece of transparent glass, reflects infrared light and passes visible light. Placing a hot mirror directly in front of the projector lens will remove any infrared from being projected on the touch surface. You can find hot mirrors at: www.bhphotovideo.com and www.thorlabs.com/

What do I need to create a FTIR table?

Required:

  • Acrylic/Polycarbonate/Plexiglass
  • Infrared LEDs (or similar infrared light source)
  • Compliant Surface (silicone rubber ie. Sorta Clear 40, Elastosil, Lexel, etc. )
  • Infrared Camera/Modified Webcam

For display:

  • Projector & Projection Surface (Rosco Grey, Vellum, Mylar, other, etc.)
  • LCD monitor & Diffuser (can be same type of material as listed for projection screen) under screen to distribute back light evenly.

What do I need to create a front/rear DI table?

Required:

  • Clear sturdy surface (glass, acrylic, plexiglass)
  • Infrared illuminator (Rear DI only)
  • Diffuser/projection surface (Vellum, Mylar, Lee Filter, other, etc.) *Note: The Diffuser/Projection Surface must let some light through.
  • Infrared Camera/Modified Webcam

For display:

  • Projector & Projection Surface (Rosco Grey, Vellum, Mylar, other, etc.)
  • LCD monitor (Usually cant use an LCD screen for a Rear DI, as most LCD screens don't allow IR light to pass out through the screen)

What do I need to create a LLP table?

Required:

  • Clear, flat surface (Glass, Acrylic, Polycarbonate, Plexiglass, etc.)
  • Infrared Laser(s)
  • Line Generating Lens
  • Infrared Camera/Modified Webcam

For display:

  • Projector & Projection surface
  • LCD monitor

What do I need to create a DSI table?

Required:

For display:

  • Projector & Projection Surface (tracing paper, lee filter, geriets optitrans , etc)
  • LCD monitor

How do I build an interactive floor?

This wiki page explains all details about how to build an interactive floor.

What is the cheapest and quickest way to make a multitouch input device?

You can make a multi touch device for less than $10 USD, if you have a webcam. The only materials you need are a cardboard box, a piece of sturdy transparent material (glass, acrylic, plexiglass, etc.), any kind of paper, a computer, in addition to the webcam. A video tutorial of how to make a MTmini (coined by cerupcat) is here.

When your fingers hit the surface of the MTmini, shadows are created, and Touchlib, an application that can translate these shadows into touch events, recognizes/tracks the shadow(s) and relays it to an application. You can download a version of Touchlib optimized for the MTmini here.

More detailed links:



What can I do about non-uniform IR light distribution in DI setups?

Depending on size and configuration of the table, it can be quite challenging to achieve a uniform distribution of IR light across the surface. While certain areas are lit well and hence touches are deteced easily, other areas are darker, thus requiring the user to press harder in order for a touch to be detected. The first approach for solving this problem should be to optimize the hardware setup, i.e. positioning of illuminators, changing wall materials, etc. However, if there is no improvement possible anymore on this level, a software based approach might help.

Currently, touchlib applies any filter with the same intensity to the whole input image. It makes sense, though, to change the filter's intensity for different areas of the surface to compensate for changing light conditions. A gradient based approach [2] can be applied where a grayscale map is used to determine on a per-pixel-base how strong the respective filter is supposed to be applied to a certain pixel. This grayscale map can be created in an additional calibration step.

Software

What are the possible different tracking software currently available?

Up until recently, tracking options were limited, depending on your operating system. Touchlib was used by most Windows (and some Linux) users, and there were several options for OS X - Touche, by Georg Kaindl and BBTouch, by Ben Britten. Reactivision is a cross-platform tracker (Win, Mac, Linux) which supports fidiciuals.

In 2008, NUIGroup released tbeta, a cross-platform (Win, Mac, Linux) tracker with an enhanced user interface, better tracking, and more user options. tbeta is available for download here.

Which tracking software is best?

There is no 'best' tracking software. Your operating system quickly narrows down your choices - Windows users can select between Touchlib, Reactivision, and tbeta. OS X users can select between Touche, bbTouch, Reactivision, or tBeta. Linux users have Touchlib, Reactivision, and tBeta as options. If you need good fiducial support, Reactivision is probably your best bet.

All trackers listed with the exception of tBeta have freely available source code, and while tbeta is open source, its authors are currently cleaning up the codebase before they release it.

I would argue that tbeta is best for beginners - simple, well-documented installation and simple steps to get it running. They're all good trackers - it essentially depends on personal preference. This wiki page outlines each framework, with related links. --rbedi100

What programming language is the best to develop in?

Again, no best answer. The level of documentation depends on each language. Actionscript 3 is a well-documented, easy to learn language (quick results), but it is slower to run. Most examples are written in flash, giving you a larger sample to learn from. C++ is harder to code for, but will give you faster programs. Python, Java and C# (WPF) are also options. This thread discusses the pros and cons of many of the languages, and this wiki page gives an overview of each language and its pros/cons

Where can I find multitouch applications?

Most available applications are mostly demos to show the ability of multiTouch. Some applications are collected in this thread. Most of them can be found here and here. NUI Group also has plans to release a formal webpage where applications can be shared, collaborated and built upon.

Are there test videos I can use since I don’t have a multitouch table yet?

Yes. You can download test videos on the tbeta website http://tbeta.nuigroup.com/ under downloads -> tools -> sample test videos. Run the video as your camera source in your blob tracker.

How can I start building applications without a multitouch device?

As noted above, you can use test videos to do so. You can also use a TUIO simulator, to simulate multiple touches using your mouse. Two such simulators are listed here.