Augmented Reality Sandbox

At System76 we manufacture Linux computers for Scientists, Engineers, and Makers. We also offer various Weekend Projects to make with your family. The Augmented Reality Sandbox is just one example of inspiration gone wild. When Oliver Kreylos started the project, his goal was

..."to develop a real-time integrated augmented reality system to physically create topography models which are then scanned into a computer in real time, and used as background for a variety of graphics effects and simulations. The final product is supposed to be self-contained to the point where it can be used as a hands-on exhibit in science museums with little supervision."

And it has really been a hit for science museums across the country and the world! Kids everywhere are learning more about topography by getting their hands on these sandboxes. Plus it is really fun!

Let's find out how to make one for ourselves, shall we?

You can also find these instructions and other projects here: https://system76.com/showcase

Step 1: Collect the Required Hardware

This tutorial focuses on installing and calibrating the software, but won't cover the details of the hardware setup.

For detailed information on hardware setup, see the AR Sandbox hardware tutorial, but in brief you'll need:

1. A first generation Kinect
2. A short-throw digital projector like the BenQ MW632ST
3. A sandbox for your sand (our sandbox is 40" by 30" by 8")
4. Roughly 200 lbs of white sand like Sandtastik White Sandbox Sand
5. A Linux-friendly computer with fast NVIDA GPU, running Ubuntu 14.01 or 16.04 (any flavor).

In terms of System76 computers, we would recommend either:

• Oryx Pro laptop with a 980M GPU, or
• Wild Dog Pro desktop with a GTX 980 GPU

Step 2: Install Software

Start by getting your computer up and running (remember that it needs Ubuntu 14.01 or 16.04)

Add the needed PPA and install the software by opening a terminal and running these three commands:

sudo add-apt-repository -ys ppa:system76-dev/weekend-project

sudo apt-get update

sudo apt-get install arsandbox

Find out what your user-name is by running this command in the terminal:

whoami

Add yourself to the vrui-grp group with this command, replacing USERNAME with the user-name returned by the whoami command above:

sudo adduser USERNAME vrui-grp

Reboot your computer so all the above changes take effect.

Step 3: Calibrate Kinect

Plug in your first generation Kinect device, then open a terminal and run:

KinectUtil getCalib 0

This will download the intrinsic calibration parameters directly from your Kinect's firmware and then write the result to a file in /etc/Vrui-3.1/Kinect-2.8/.

Step 4: Align Kinect Above Sandbox

Open the terminal and run:

RawKinectViewer -compress 0

Maximize the window so it's easier to see your sandbox.

The Augmented Reality Sandbox only uses the depth view, but the camera view is still helpful for aligning your Kinect (depth view is on the left in the above image while camera view is on the right).

The depth view needs to cover the entire interior of your sandbox. It's okay if it overlaps slightly.

Hit Esc to close the RawKinectViewer.

Step 5: Calculate Base Plane - Part 1

There are two ways to calculate the base plane. If you haven't filled your sandbox with sand, you can calculate the base plane using a region on the floor of your sandbox.

We already filled ours with sand and so we will use a piece of poster board (or a similar flat surface) on top of the sandbox to calculate the base plane.

From a terminal, launch the RawKinectViewer:

RawKinectViewer -compress 0

Maximize the window so it's easier to see your sandbox.

Press and hold the right mouse button, move your cursor over Average Frames, then release the mouse button.

Press and hold the 1 key, move your cursor over Extract Planes, then release the 1 key.

You now need to draw a rectangle that fits within the interior of your flat surface (in the depth view). You want some space between your rectangle and the edges of your flat surface.

Start with your cursor near the top-left corner of your flat surface. Press and hold the 1 key, drag out a rectangle toward the bottom-right corner, then release the 1 key (see the second image? That's what it should kinda look like).

Hit Esc to close the RawKinectViewer.

Step 6: Calculate Base Plane - Part 2

In the terminal you'll see two lines printed. Select the portion of the 2nd line after the asterisk (should be 3 coordinates in parentheses, an equal sign and another number), then right click and select Copy.

Edit the BoxLayout.txt file by running this command from the terminal:

gedit /etc/SARndbox-1.6/BoxLayout.txt

Paste the line you just copied over the first line in the BoxLayout.txt file.

You also need to edit this line, replacing the = (equal sign) with a , (comma).

You should end up with a first line that looks similar to the image above.

Save the file and close gedit.

Step 7: Measure 3D Extents of Sand Surface - Part 1

This step requires that you have filled your sandbox with sand. You want the surface to be as level as possible, but it doesn't need to be perfectly level. If you placed a piece of poster board (or another flat surface) on top of your sandbox previously, you'll need to remove it for this step.

From the terminal, again launch the RawKinectViewer:

RawKinectViewer -compress 0

Maximize the window so it's easier to see your sandbox.

Press and hold the right mouse button, move your cursor over Average Frames, then release the mouse button.

Press and hold the 1 key, move your cursor over Measure 3D Positions, then release the 1 key. You'll now measure the 3D extents of the interior of the sandbox.

Use the color-coded depth map to make sure you position your cursor over the sand surface and not over the sides of your sandbox. Position your cursor so that there's a small amount of space between the sand surface and the sides of your sandbox.

First, move your cursor to the lower-left interior corner, then press the 1 key.

• Note that there is no feedback from the UI when you press the 1 key.

Repeat this step with the lower-right, upper-left, and upper-right interior corners.

Hit Esc to exit RawKinectViewer.

Step 8: Measure 3D Extents of Sand Surface - Part 2

In the terminal you'll see four lines printed. You need to highlight these four lines. They should be 3 sets of 3 coordinates with brackets on either side. Highlight all 9 numbers and their parentheses, then right click and select Copy.

Edit the BoxLayout.txt file by running this command from the terminal:

gedit /etc/SARndboc-1.6/BoxLayout.txt

Replace the bottom four lines in the BoxLayout.txt with the four lines you just copied, so you end up with something like the image above.

Save the file and close gedit.

Step 9: Position Projector

If you haven't already, turn on your projector and plug it into your computer.

If you have a dedicated display in addition to the project, you'll find it easier if you set up the two displays to be mirrored.

As the full-screen hotkey for the Augmented Reality Sandbox currently doesn't work under Utility, we also recommend that you set the Ubuntu launcher to auto-hide (Settings --> Appearance --> Behavior).

Launch the XBackgroud tool from the terminal:

XBackground

You'll need to position the projector so that it fills the interior of your sandbox. It's okay if it overlaps slightly.

After you're done positioning your projector, hit Esc to close the XBackground application.

Step 10: Calibrate AR Sandbox - Part 1

By now your sandbox should already have sand it in (as noted in previous steps). You want the sand surface to be as level as possible but it doesn't need to be perfectly level.

To calibrate your sandbox, you'll need an alignment target and several spacers to help you perform the alignment at multiple heights.

The easiest way to build an alignment target is to create a target cross-hair on a piece of paper and tape it to the top of an old CD or DVD. The cross hairs should be at right angles to one another and be centered on the CD or DVD. A few rolls of tape make great spacers. The tape should be smaller in diameter than the CD or DVD you're suing for the alignment target.

At each height, you'll capture 12 tie-points. For a decent calibration, you'll need to capture tie-points at at least two heights (24 total tie-points). For an optimal calibration, we recommend capturing tie-points at three different heights (36 total tie-points).

Step 11: Calibrate AR Sandbox - Part 2

Launch CalibrateProjector from the terminal:

CalibrateProjector -s WIDTHHEIGHT

Replacing WIDTH and HEIGHT with the settings for your projector. The Kinect itself has a 4:3 aspect ration, so a resolution of 1024x768 or 1600x1200 is best.

To match the resolution of the laptop we used in this tutorial, we set our projector to 1920x1080, but our particular BenQ projector allows us to force the 4:3 aspect ration even when the resolution is a 16:9 aspect ration. So in our case, we launched CalibrateProjector like this:

CalibrateProjector -s 1920 1080

Then maximize the window so it fills your entire sandbox.

Press and hold the 1 key, move your cursor over Capture, then release the 1 key.

Press the 2 key when you see the dialog that reads: Please press the button to assign a tool function Capture Background Press 1 again to cancel.

Press the 2 key again to capture the background image, after which you'll briefly see a uniform red color projected onto your sandbox.

After this is completed, you can proceed with the calibration.

Step 12: Calibrate AR Sandbox - Part 3

Next, you will capture tie-points at the lowest height. Using your shortest spacer, line-up your alignment target under the white cross-hairs projected onto the sand surface. Then press the 1 key to capture this tie-point. After a brief moment, the software will automatically move the projected cross-hairs to the next tie point.

Repeat this process for the remaining 11 tie-points at this depth. Once the white projected cross-hairs are back at their original position, you're ready to switch to a taller spacer.

You will now capture tie-points at the middle height using the same method. Using a taller spacer, line-up your alignment target under the white cross-hairs projected onto the sand surface. Then press the 1 key to capture this tie-point.

Repeat this process for the remaining 11 tie-points at this depth and when the white projected cross-hairs are back at their original position, you're ready to switch to a taller spacer.

Finally you'll capture tie-points at the highest height. You can stack two spacers if you need to like we did in the picture above. Repeat the steps as before until you have completed all 12 tie-points at this level.

Once you've completed the calibration, hit Esc to close the CalibrateProjector application.

Hitting Esc will automatically write the calibration file in /etc/SARndbox-1.6/ProjectorMatrix.dat.

Step 13: Adjust the "Sea Level"

Launch the main SARndbox application by searching for "sandbox" in the Ubuntu dash. Then maximize the application so it fills your entire sandbox.

If you calculated the base plane with a piece of poster board (or a similar flat surface) on top of your sandbox, your sea level will be too high, and you'll see something like the first image above.

On the other hand, if you calculated the base plane when your sandbox was empty, the sea level will be tool ow. Either way, you'll want to adjust the sea level so it's just below the surface of your sand when it's more or less flattened out.

Hit Esc to close the AR Sandbox application.

To adjust the sea level, edit the /etc/SARndbox-1.6/BoxLayout.txt by running this command from the terminal:

gedit/etc/SARndbox-1.6/BoxLayout.txt

The value outside of the parentheses on the very first line is the one that controls the sea-level. You'll want to adjust the final value in the first line to be roughly the same as the smallest of the final value on the four lines at the end. You can see by our example in the second image that the smallest number is -106.927 so we would replace -95.9598 with -107.

Save the file and close gedit.

Re-launch the AR Sandbox application from the Ubuntu dash, maximize the window, and you'll see something similar to the final image above.

You can experiment with different values for the sea-level to suit your preferences, but our recommendations here will give you a good starting point.

Note that you can adjust the sea-level at any time in the future without re-calibrating your sandbox.

Step 14: Use the Sandbox!

Whew! You've made it through the setup and calibration!

As long as you don't change the physical setup of your sandbox (in particular, the position of the Kinect, the projector, and the sandbox relative to each other), you never need to calibrate it again. Although now that you have the hang of it, you might want to go through the calibration again in order to achieve a more accurate setup.

If you hold your hand above the sand surface (like a cloud), you can "make it rain" like in the image above.

As a convenience, the System76 packaging for Ubuntu includes a launcher you can search for in the Ubuntu dash. This is the shortcut to launch SARndbox with the following arguments (which you can also do directly from the terminal):

SARndbox -uhm -fpv -evr -0.01

The -evr -0.01 bit of the above command sets the evaporation rate. Without any evaporation rate set, the sandbox will steadily fill with more and more water as you "make it rain", which eventually will cause the real-time fluid simulation to become slow.

So you typically want to set an evaporation rate, especially when deploying the Augmented Reality Sandbox in schools or museums.

If our recommended evaporation rate seems too fast for your tastes, try:

SARndbox -uhm -fpv -evr -0.005

If our recommended evaporation rate seems too slow for your tastes, try:

SARndbox -uhm -fpv -evr -0.02

Thanks again to Oliver Kreylos for developing such an incredible, inspiring piece of software!