Add 3D Interaction to Your App Using OSC

If your app uses Open Sound Control (OSC), it’s easy to get it to work with hand gestures. And free too.

Antoine Rennuit, Technical Product Owner at SoftKinetic, made this bridge in his spare time to make it easy for people who create, make or adapt things to play around with 3D cameras. This neat piece of software is free to download, easy to set up and enables you to create meaningful user experiences using natural hand gestures.

Here’s what you’ll need:
1. Hardware
   - A Windows PC
   - A short range, 3D gesture recognition camera (SoftKinetic’s DepthSense 325 (DS325) or the Creative Senz3D), plugged and installed onto your PC.  You could also use the software with a long range camera such as the SoftKinetic DepthSense 311 (DS311).

2. Software
   - SoftKinetic’s iisu middleware: to analyze the 3D images given by the camera and provide hand gesture information
   - Antoine’s ‘iisuOSCbridge’ to link data sent from SoftKinetic’s DS325 or the Creative Senz3D camera to OSC messages.
   - An OSC-compatible application such as ‘Processing’: to receive the OSC stream.

Follow the instructions below to download, install and set up these applications.

Step 1: Download, Install and Setup “iisu”

Go to www.softkinetic.com, click on the ‘Download iisu’ button on the homepage, download iisu Free (you will need to register first) and install it.
Once downloaded, set it up as follows:
- Start iisu Launchpad
- Go to Select Camera and select one
- Close the Select Camera window
- Go to Advanced Configuration and make sure that the Layers starting with ‘CI_’ are checked (CI stands for Close Interaction)
- Now start iisu Toolbox to check that everything is working and have a look at what the camera sees.

iisu is now setup (and your computer can see in 3D! ).
You can now close it and take the next step…

Step 2: Download and Install Antoine’s IisuOSCbridge

- Go to his blog and find the ‘download’ post.
- Download the Windows Installer and run it.

Step 3: Create Your OSC Stream

If you want to skip this step simply download the zip file at the bottom of this page.

- Start iisuOSCbridge.exe
- Fill in the ‘IP’ field by identifying the computer to which the message is to be sent. The default IP address is set to 127.0.0.1 – that’s your own computer and probably the one you want. If, though, you want to send it to another machine on your network, enter its IP address. To find out your IP address visit: http://www.wikihow.com/Find-the-IP-Address-of-Your-PC.
- Now set the port. Just leave it at 8000, unless you have a good reason to change it. You will have to enter that port number to your listening application so that it knows what to listen to.
- Now create your OSC messages. OSC messages are organized like a tree, similar to your file system. So first you need to create the ‘root’ by clicking Add. Rename it ‘Root’ by double clicking its name (by default this is ‘New Osc Path Bit’). It’s best NOT to add value to nodes that have children, so don’t change the ‘New iisu Path’ for this one.
- Insert a child by clicking Add Child. Rename it ‘Hand1’.
- Insert a grandchild by selecting Hand1 and clicking Add child again. Now select an actual value for this node by double clicking on its ‘new iisu path’ and scroll down to explore what data is available. Ideally, choose CI.HAND1.Status and rename that node ‘Status’. This ‘Status’ is a number that will indicate whether or not the camera sees a hand. Please refer to the User Guide from SoftKinetic for more information about the available data.
- Now add another item by clicking on Add. Choose another value for this one. We recommend that you choose the 3D position of the palm (in relation to the camera): CI.HAND1.PalmPosition3D. And rename it PalmPosition.
- Finally, create one last node by clicking Add again. Choose CI.Hand1.TipPosition3d as the path and rename it ‘TipPosition’.

That’s it. All the paths are created and your data is ready to be heard!
For more detailed guidance, please refer to the in-depth documentation in the ‘Doc’ folder of your installation (‘C:\iisuOscBridge\Doc).

Step 4: Plugging to Your Application

Your OSC-enabled application (or device) can receive the data by listening to the port 8000 (or whatever other value you chose) for the following path:
- Root/Hand1/IsOpen
- Root/Hand1/PalmPosition/X
- Root/Hand1/PalmPosition/Y
- Root/Hand1/PalmPosition/Z
- Root/Hand1/TipPosition/X
- Root/Hand1/TipPosition/Y
- Root/Hand1/TipPosition/Z

Note: If you wish to create a two-hand application, just add a ‘Hand2’ node and add children as you did previously ‘Hand1’. Your path will be:
- Root/Hand2/IsOpen
- Root/Hand2/PalmPosition/X
- Root/Hand2/PalmPosition/Y
- Root/Hand2/PalmPosition/Z
- Root/Hand2/TipPosition/X
- Root/Hand2/TipPosition/Y
- Root/Hand2/TipPosition/Z

You can connect this stream to many different applications. Please refer to your application's documentation to find out how to connect it to an OSC stream. If you’re using ‘Processing’ then you can find the sketch at the bottom of this page.

And, if you want to go further, take a look at iisu Interaction Designer that was installed along with iisu. There you will be able to create little scripts to combine and create new gesture data that you can export to OSC.

Now just use your imagination and find new ways of using 3D cameras to bring apps to life. Put a comment up and let us know what you’ve built.