GSoC 2011 Ideas Page

From NUI Group Community Wiki

Jump to: navigation, search

Google Summer of Code 2011

GSoC 2011 is starting and this page is dedicated to general information and ideas that can be used for student proposals. The ideas listed on this page are just that - ideas. They are deliberately somewhat vague, because you are meant to fill in the details. We don't want to see project proposals which are basically cut-n-pasted from our ideas page. Don’t do that! If you do, you will be tossed out very early in the application sorting process. Be creative and extend our project ideas with additional description and your own feedback. Also listed below is contact information for the community and specific mentors.

Preparing to Apply for Google Summer of Code

The most common question we get from students is "How do I prepare for Google Summer of Code?" The simple answer is not to wait for the program to get involved in open source. Start checking out projects now: take a look at documentation, check out their source code and idle in some IRC channels to see what interests you. The projects who have been past participants in Google Summer of Code are a great place to start;You can also find the list of student projects for 2010 linked from each accepted organization's page on the ( 2011 site. Google will announce the list of accepted mentoring organizations for 2011 on March 18th.

We don't have exact template for your project proposal to use but we provide a list of information that we would like to see in it which includes: name, email, timezone, age, education, blog url, location, your project proposal, why you'd like to complete this particular project, are you an active NUI Group member, have you been working on multitouch, natural user interfaces, HCI and related topic before, the reason you're the best individual to do so, what's your academic and industry background, and of course open source development experience, explanation of your development methodology is a good idea, as well. Other details as you see fit.

Note for students: Be prepared to supplement your proposal text with links to an external site, for example full version of your proposal in pdf file. You should also plan to provide an abstract of your proposal, including a brief list of deliverables, via the GSoC web app to ensure that your work receives sufficient review.

Official Google Pages:

Getting Started with GSoC

Here are the steps and some tips on getting started:

1) Think of a good idea – For help and references see the Ideas section below...
2) Do some research yourself based on the idea, write up a proposal draft
3) Post it up on the forum @ (GSoC dedicated section) to get feedback
4) Write up full proposal (4-10 pages with graphics and resume)
5) Submit it to google by the following process (see steps below)

i. Login to
ii. Click on the “Register as a Student link” on the left menu(Will appear from 28th March 19:00 UTC to 8th April 19:00 UTC)
iii. Fill up all the information (You will be able to change this in future)
iv. After filling up, you are registered as a student
v. Now click on the “Submit your Student Proposal” Link on the left menu
vi. A Page will appear on which all the accepted organizations will be listed. Now click on The organization you want to apply in (e.g. NUI Group in this case)
vii. A direct method will be using the following link**your_link_id**?organization=nuigroup (from 28th March 19:00 UTC)
viii. You can put the full proposal in the rich text editor there. Also the full proposal link/ any external link can be provided in the “Additional Info” textbox.
ix. Now your proposal is submitted. You can now see the proposal and subscribe to updates (Link will be on the top of the proposal view page).
x. As per the rule you can submit up to 20 proposals.

Proposing Your Ideas

You really don't have to use project ideas we covered below. These are projects that we have in mind to start working on or think that would be really useful for NUI Group community. Think of them as starters, even thought some developers think it’d be great to see them done. If you have your own idea, bring it on. We are always open for new innovative project ideas, so feel free to amaze us with yours. We’ll judge applications on their strengths, not on who wants the project done or who thought of it. However you might consider and find some of the project ideas listed below interesting. If you (as a student) have a new idea that is not listed here feel free to edit this project ideas wiki page and add your idea with your contact info.

Here you can find some additional documents and webpages that could be useful when writing your project proposal. Sometimes they link to other open source projects, but contain many useful guidelines which can be used when writing your proposal.

Student/Mentor Contacts

The fastest and quickest way to get feedback is by posting your idea or questions on the forums!

Here: You can find some ideas and articles below to help get started.

Otherwise, please any questions about becoming a GSoC Student or Mentor to this email: Image:1177495748.Png

Projects Overviews:

Mentor Alumni:

  • Christian Moore (Nick: naturalui)
  • Pawel Solyga (Nick: solydzajs)
  • Thomas Hansen (Nick: thansen)
  • Mathieu Virbel (Nick: tito)
  • Christopher Ruff (Nick: mt4j)
  • Martin Kaltenbrunner (Nick: tuio)
  • Seth Sandler (Nick: cerupcat)
  • Stjepan Rajko (Nick: rajko)
  • Laurence Muller (Nick: falcon4ever)

Projects Ideas

Some ideas might not be suitable for GSoC proposal because they are too easy and it's development won't last 3 months. Remember that you can still select this idea and combine it with other ideas in order to write GSoC proposal which meets GSoC requirements. In addition, some of the projects ideas might be to hard to develop during the summer by one student developer, therefore you can limit it and just work on particular parts. If you become interested you can always work on the rest features after GSoC - that's our little open source world :-)


  • User Interface and Experience Design (Human Computer Interaction)
  • Computer vision (tracking, optical flow, image processing)
  • Touch, gesture and stylus input
  • Machine learning (AI, Neural Nets, Bayesian clustering, genetic algorithms)
  • Industrial design for sensing and projection/paneled interactive displays
  • Mobile interaction concepts
  • Programming Languages: Actionscript 3.0, C-Sharp, C++, Java, Python and VVVV.
  • Custom Hardware: Microcontrollers, Sensors and open source hardware projects such as Arduino


Community Core Services (C/C++):

  • Community Core Audio (CCA) ( (GSoC 2010 - Alpha) - A cross platform server which manages voice inputs, convert voice to text, and output resulting messages to network. Alpha shipped during GSoC 2010 including a new addon (ofxASR) abstracting signal processors and sources. 2011 Goals - Ship 1.0 release stable on all platforms and broaden natural language processor support (SAPI, Sphinx, OSX-SRS) / CCW Management
  • Community Core Vision (CCV) ( (GSoC 2008/2009/2010 - Beta 1.4) - A cross platform computer vision server for high fidelity tracking of objects such as fiducials, hands or fingers. First release during GSoC 2008 and with great community response. In 2009/2010 we continued to develop new tracking techniques and enhanced stability. 2011 Goals - Ship 1.5 with Depth Sensor (Kinect Devices) / Augmented Reality / CCW Management
  • Community Core Web (CCW) ( (GSoC 2010 - Alpha) - A cross platform light weight web server which supports plugins for web development standards such as: Moongoose/PHP/MySQL/SQLite etc. Manages C++/Javascript communications allowing CCA/CCV to be controlled via minimal web client. Alpha started during GSoC 2010 but needs further development until official community beta. 2011 Goals - Ship 1.0, optimize, stable on all platforms, continue development of HTML/JS based management utility and binding to core services (CCA/CCV)

Community Client Frameworks:

  • Actionscript/Flash - (cm[at] - A client UX framework and set of example applications written in Actionscript (Started in 2006 with Touchlib)
  • C/C++ - A client UX framework and set of example applications written in C++
  • Java/MT4J - A client UX framework and set of example applications written in Java (GSoC 2009/2010 projects)
  • Python/PyMT - A client UX framework and set of example applications written in Python (GSoC 2008/2010 projects) 2011
  • Framework Goals - Development and standardization of target frameworks including structure, documentation and release packages.

Community Tools:

  • Signal Simulators - Utilities to generate signals when hardware is not available. (QMTSim GSoC 2008)
  • Signal Translators - Client/Server Translation Utilities (flosc, wm_touch2tuio, mtvista)

With ongoing developments for supporting projects:

  • OpenFrameworks - Continue contributing to Addons/Framework through C/C++ developments
  • Protocols – Socket support on clients and servers to broaden TUIO/OSC support
  • UX Design - A general focus on emerging experience design topics related to NUI
  • Web Applications - Community Search, Publishing and Communication (LAMP Stack)

CCA (Community Core Audio)

Community Core Audio is a cross platform server which manages voice inputs, convert voice to text, and output resulting messages to network. In GSoC 2010 we shipped our first Alpha of this project and hope to get feedback from the community on this preview and look forward to the future results.

  • GUI library - Research and migrate the logic to another GUI library to get best from code practice, performance and community.
  • Integration of more voice providers and processors. - Currently CCA uses ofxASR to process signal but should open up to others such as SAPI.
  • Continued UI Development and Optimization - Continue working on UI developments which include determining application features and goals.
  • New Easy-to-use Dialogs - Currently CCA uses config file to set reference texts and preferences. Add some GUI dialogs to do these jobs.
  • Platform Testing and Device Support - Test audio input and output devices on different operating systems.

Projects Page:

CCV (Community Core Vision)

Community Core Vision is a cross platform computer vision tracking solution for high fidelity situations such as head, hand or finger tracking. Started in GSoC 2008 and with great community support we are looking to further develop new tracking techniques and platform stability.

  • Code refactoring and modularization of vision techniques/filters and controls.
  • Region of Interest Selection
  • Kinect and Depth Sensing Integration
  • Code Optimization
  • Implement Threading and Separation of Tracking and GUI
  • GPU Acceleration (existing but not fully implemented)
  • Dynamic Filter Addition and Adjustment (in the chain)
  • Object/Fidicual/Tag and Pattern Tracking
  • Multiple Camera Control and Support (Multiple Cameras/Stitching and Alignment)
  • Creation of Open Framework Addons supporting community projects
  • Enhance Setup GUI - Create more detailed interface for setup and configuration of the vision engine and offer a more dynamic calibration system.
  • Wii Remote support for object/finger recognition/tracking - Wii Remote support so we could use the wii-remote as an input device, we are looking for Windows, Linux and Mac versions
  • Add multi-camera support - Implement stereo vision techniques and algorithms. Have tools for camera alignment and calibration.
  • Physical Object Tracking - Work on tag (fidicual) recognition, create a separate library which can be shared other project or merge existing library (reactivision) into CCV
  • Testing framework - write a testing framework for easy blob detection and tracking algorithms testing, write unit tests for touchlib, write automated tests
  • Windows Service support for(a daemon) - add Windows Service support to touchlib so it can run in the background as system service instead of user desktop application, define and write communication layer between service and configuration application. Autostart setup possibilty for exhibits. like start osc, start flosc and start flashdemo.swf (possible with batch files, more user friendly approch)
  • Support of Tuio 2 - create an use a Tuio 2 library. Tuio 2 is more extensible, and we can have more information from the tracker, as the shape of the touches, hands etc.

Project Page:

Getting Started:

CCW (Community Core Web)

Community Core Web is a light weight cross platform web server which supports plugin modules for PHP/MySQL/SQLite and related web development services. Also acts as a C++/Javascript bridge allowing CCA and CCV to be controlled via the web browser. Alpha started in GSoC 2010 but needs development until official community integration.

  • Continue Plugin Architecture Design and Implementation
  • Expand Plugin Library (Ruby, Perl, Linq)
  • Develop XML based UI rendering system inline with hosted services for settings

Project Page: (Pending)

TUIO Protocol

  • Develop higher level application frameworks based on the existing TUIO client implementations (C++, C# ...)
  • Implement TUIO client libraries for programming languages or environments that are not yet supported
  • Implement TUIO server modules for existing tracker applications that are not yet TUIO enabled
  • Write a TUIO application for Android phones and tablets (similar to the TuioPad for the iOS)
  • Develop a TUIO 2 library

Project Page:

Kivy Framework (PyMT)

  • More interactions/Widgets - Research and implement widgets based on more than 2 touches and hands. The question to answer is: What can we do in terms of interaction with additional information such as orientation? Implementing cool interactions as widgets or other Kivy features so that they are reusable or extensible is what we are looking for.
    • Intuitive 3D interactions for moving objects like navigating a 3D space, rotating, sizing etc.
    • Intuitive interactions for selecting, arranging objects on a tabletop.
    • Implement intuitive multitouch menus that take into consideration challenges like display orientation, multiuser, touch input etc.)
  • Gesture Framework - Create new or enhance existing gesture framework. Specifically, specifying, recording, and recognizing multi finger gestures. Design and implement the gesture API to work nicely with other multi-touch programming paradigms or maybe even come up with new ones (research paper opportunity?!). Some challenges to think about: How are regular touch events reported while a gesture is being performed, how to check if a gesture is being performed/in progress (maybe state machine based).
  • More Input providers - Implementing native input providers for specific input devices. We have already: TUIO, mouse, Windows 7 multitouch (WM_TOUCH), windows pen. Multi pointer X would be nice for Linux, maybe wiimote support nicely integrated to Kivy. (One input provider probably isn't worth a whole summer project, but one could do a couple or expand on the idea otherwise)
  • Improve Video & Camera support - Kivy's core is very modular, so we can use different backends and you can implement new providers for things liek video support, images, etc. We could really use e.g. a Quicktime Video provider or better webcam support cross platform (especially osx is kind of a pain right now with gstreamer)
  • Use current platform's native APIs - This adds to the previous point in that we'd love to have platform specific providers for pretty much everything. In the ideal world, we'd just ship the Kivy code without additional dependencies. For OS X, this would be the implementation of providers using Core Video/QTKit, Core Audio, Core Image, Cocoa. Or equivalently for Windows, use their native APIs. Prior experience or willingness to learn new APIs required. Not that easy, but certainly a very rewarding task.
  • Extend and enhance support for mobile platforms - We already have an Android port in progress. An iOS port for example would be nice. This is challenging, but potentially awesome results.
  • Network serialization - Be able to exchange widgets and their state between different Kivy processes (across the network). So you can e.g. throw a multi-touch ball at someone on a remote table. Some work on this has been done, and we could help at the very least with conceptually how to do this.
  • Cross platform, no hassel PDF (or other widely used filetype) support - We'd really like to be able to integrate e.g. PDF documents easily, the state of doing this is not very good in terms of cross platform support without having to install or compile all kinds of stuff.
  • Kivy IDE Build a Kivy Integrated Development Environment or GUI builder using Kivy itself - We've played with this idea, and might be able to help significantly if we get a good start or exciting proposals. We also have had some fun ideas like live coding, being able to change the UI at any time by switching to IDE mode etc. Python is very flexible in terms of loading code at runtime etc. so many things would be possible
  • A game or really cool application idea - Come up with a really fun game or useful application that uses multitouch input in new or interesting ways. Let your imagination run wild, and show the world what Kivy is capable off.
  • Debugging Tools - Multi touch can be really hard to debug (mouse simulating multi touch helps, but only gets you so far). A good project would come up with useful debugging concept for debugging multitouch development and implement them so they can be used with Kivy (e.g. Kivy modules, or debugging support built into the core).
  • Calibration suite - Implement semi-automatic calibration of multitouch displays. This is especially useful for DIY hardware. Think of the following calibration steps: DoubleTap speed calibration, Improved touch flow (e.g. see retaintouch module), find & ignore hotspots, Dejitter, implement coordinate transformations if the tracker isn't properly calibrated we can do it inside Kivy.
  • Community Documentation or Example Apps - Writing good documentation and example apps or tutorials is hard, but really helpful to new people checking out a library. A summer project would involve multiple smaller parts of writing example applications, documentation, and tutorials. Great way to learn about Kivy if no prior experience is present.
  • Better integration of Optical tracker - CCV, Reactivision, Movid... are only GUI. You might want sometime to trigger again a calibration, background capture, changes other parameters... directly inside your current application. Idea would be to create a standard protocol and a library for controlling optical tracker in a generic way, and spread that library into all the trackers.
  • Enhanced text Input - We have keyboard and virtual keyboards. Other text entry methods could be fun to explore see e.g.
  • Enhanced input interactions - Interactions that take into consideration more than just touch input. New technologies allow e.g. for hand tracking, to know hand orientation, etc. Develop widgets that take advantage of this new input data. (Example:Keyboard could be splitted in half: One half for each hand, and the respective half follows the hand it belongs to and rotates with it automatically, done last year in Gsoc 2010)

Project Page:

Project Page:

MT4J Framework

  • Shader Effect integration -> Integrate a little GLSL shader framework into MT4j to allow effects like blurring or glowing for example for any visible MT4j component.
  • Create the groundwork for zoomable user interfaces (ZUIs) using MT4j -> While zoom-in functionality is already present in MT4j there is no way to "semantically" zoom to navigate to another part of an application or to another application entirely and smoothly zoom in while streaming and loading the new content to be displayed. This would allow for "endless zooming" and to navigate huge data conveniently.
  • Complete multi-touch UI toolkit -> Create a comprehensive MT4j widget toolkit (windows, lists, tables, audioplayer etc) with features like skinnable appearance, automatic layouts using the model-view-controller paradigm
  • Create and explore new, specialized UI-Components tailored to usability and interaction on multi-touch displays -> For example: new forms of menus (e.g. circular menus - see:, lists (e.g. like in Cooliris or other components that allow to browse content in a user friendly, innovative way
  • Network serialization -> Create the possibility to send and share MT4j components across networks with other MT4j applications and implement a showcase application (e.g. document sharing, image sharing etc).
  • Test framework -> Create the possibility to have automatic testing by pre-recording user input and then playing it back while asserting the current state. This could also include stress-tools that send random input to applications or widgets to check if they break.
  • 3D Model animation -> Implement 3D model animations with bone animation and skinning to allow to animate imported 3D models.
  • 3D modeling tool -> Create a multi-touch 3D modeling tool using MT4j
  • Multi-Touch games -> Create innovative multi-touch games like a warcraft-like real time strategy game for example, using MT4j

Project Page:

NUI Clients Applications

Typically developed in AS3, C/C++/C#, Java, Python or a Visual Programming Language (Max/PD)

  • Rich Search Engine UI
  • New 3D Depth Tracking Scenarios
  • Module Based Application Loading
  • Advance Menu System for Application Loading and Control
  • Media Loader (Document, PDF, Video, Image)(jpg,jpeg,tga,png,dds,gif,tif, tiff, bmp,mov,avi,wav,ogg)
  • Particle Sandbox (C/C++)
  • Physics Sandbox (C/C++)
  • Paint and Drawing Sandbox (Brushes, Filters and Tools)
  • Educational Games (Match/Math Quiz/Beat the Chimp)
  • 3D Object Viewing and Manipulation (Standard 3D files)
  • Gesture Sandbox - Gesture recognition and programming that saves to flat files.
  • On Screen Keyboard and Input Devices
  • Dynamic Loading of Files via USB/Firewire or Bluetooth
  • Image Manipulation - Use MT to bend images etc
  • Audio Creation and Games - (Bloom... Loop Sequencer)
  • GSI Mapping
  • Web Browser
  • Tablet User Interface Concepts
  • 3D Navigation Concepts

Mobile - Android/iPad/iPhone/iPod Concepts

  • Touch App - propose innovative multitouch iPhone application or educational game
  • Gesture Implementation- Write multitouch gesture recognition library for iPhone (defining your own gestures)
  • Audio Analyzer- Basic audio analyst tool for iphone (spectrum/sonogram/FFT)
  • Accelerometer- Collects and sends Acceleration Data out via OSC

Community Web Apps

Custom developed web applications typically written in Coldfusion/PHP/Ruby/Python/Java

  • Spam Filtering
  • Ruby and PHP ( - developments)
  • Community Research and Resource Centralization
  • Forum Data Analyst and Restructuring
  • Custom NUI Search Engine and Google SEO
  • Automatic Multitouch Creator (input goal, output parts list and specifics, etc)
  • Any Web Application that could benefit Community...

Other Project Ideas

  • Linux-based stand alone blob tracker - create a lightweight Linux-distro for ultra light pc that includes firewire-camera drivers, blobtracking software and sends the touch information trough ethernet or even better trough USB to the main computer. Also a controlling software is needed for every operating system to configure this linux-box. Also is important to determine how fast processor is really needed for tracking
  • MultiTouch mouse drivers for every OS - create mouse drivers for OsX, Windows and Linux so that they can be used without external mouse trough MT-display
  • On-Screen Keyboards - create on-screen keyboard software for OsX, Windows and Linux so no external keyboard is needed to operate regular programs trough MT-display
  • Collaborative multisite collaboration using multitouch screens - write an application/framework which will allow for multisite collaboration on multitouch screens, what it means is that you can have many multitouch screen in different locations connected via internet or internal network and mt screen works as shared desktop with gestures support and synchronization between screens. You can have a look at croqodile project (, TeaTime protocol used in Croquet project for syncing, solution for any platform will be most welcome, whether it's Flash/Flex/AIR or OpenGL or any other platform specific software
  • WiiMote multitouch framework - create a framework for writing multitouch applications such as interactive whiteboard using WiiMote under Mac OS X (DarwiinRemote project might be helpful), write some example applications
  • Work on WiiMote Whiteboard project - create new example applications, port WiiMote Whiteboard to Linux and Mac, write unit tests and create installer with binary distribution
  • Improve blob detection and tracking algorithms - write optimized library for blob detection and tracking so that it can be used later with different projects ex. tbeta, touchlib, bbtouch
  • Your innovative multitouch application - propose new innovative multitouch application you would like to work on (interaction in 3d environment, creating scenes, documents), amaze us with your ideas
  • Work on WiiMoteTUIO project - example applications, bug fixing, create a cross-platform solution, write test and documentation
  • Automatic projector calibration - propose a project for automatic projector calibration used with multitouch screens in DI and FTIR setups
  • Multitouch gesture library - write multitouch gesture recognition library with support for gestures that could consist of several strokes like /\ + - = A, simple one finger gestures like mouse gestures and multitouch gestures (zooming, pinching, rotating and many more), use fingerworks gesture dictionary for example gestures (
  • Multitouch applications on Mac OS X - propose and write multitouch application for Mac OS X, using TUIO protocol for events and new Leopard graphic frameworks like Core Animation, Core Image, Core Video, QuickTime etc.
  • Firefox TUIO extension - write a Firefox TUIO client extension which maps TUIO events to JavaScript events and adds gestures support in Firefox using same JavaScript extensions used in WebKit/Safari introduced in iPhone SDK
  • Research multicomputer use for image processing - work on distributed blob detection and tracking solution, dividing computation between many machines, this would be really useful for big installations like multitouch walls and with many cameras, test your solution and integrate with OpenTouch or touchlib
  • MB Pro and MB Air multitouch trackpad application for Mac OS X - propose and write multitouch application which supports MacBook Air and MacBook Pro multitouch trackpad gestures or extend existing open source project to use the power of intuitive trackpad gestures
  • GPU accelerated blob detection and tracking algorithms - write GPU accelerated blob detection and tracking algorithms for detecting and tracking blobs. You can use CG for NVidia or compare usage of GPUCV vs OPENCV

Related Resources

Community Resources

Comments or Questions

We are open for any comments and questions here...


Feel free to contact us whenever you have any questions or if you want to use an idea, but you need more details.

Join us at NUI Group forum at or our GSoC NUI Group discussion group at to discuss GSoC & NUI Group project ideas and related topics.

You can also drop an email to our organization administrator at Image:1177495748.Png

Additional Resources