Introduction: Portals: a Networked Mixed-Reality Playground
Portals is a prototype of a tangible and networked system/interface for interacting with a virtual world.
A portal, according to Wikipedia, is a magical or technological doorway that connects two distant locations separated by spacetime. Places that a portal will link to include a different spot in the same universe, a parallel world, the past or the future, and other planes of existence.
There are basically 3 main features of these specific portals:
1. To interact with the portal, you reach inside the box, going behind the screen instead of gesturing in front of it or touching it.
2. The two portals are networked so you can collaboratively play with someone in a shared space, even if they're physically far away.
3. The system recognizes and responds when special objects are inside, tracking the id, position, and rotation.
Videos
It makes more sense when you see it in action, so here's an edited version of a longer scenario video I made for the project which just highlights the main features.
Also, a short video of the Portals in action at 4 Hours Solid, my grad show:
The project was created in 2011-12 as part of an MFA Thesis in Graduate Media Design, at Art Center College of Design. If you want more details about the project visit Portals (and friends). One of my main goals of the project was to create something that other people could build upon and expand. So.... without further ado, let's get on to the How To part of things!
Note: my process of building these boxes wasn’t this straight forward. It was a super iterative process that involved lots of small tests. Even though I’m going to try to explain it in a more logical step based way, I would still recommend testing out parts of the hardware + software before diving in and building the entire thing for real.
“The visible world is no longer a reality and the unseen world is no longer a dream.”-Arthur Symons
A portal, according to Wikipedia, is a magical or technological doorway that connects two distant locations separated by spacetime. Places that a portal will link to include a different spot in the same universe, a parallel world, the past or the future, and other planes of existence.
There are basically 3 main features of these specific portals:
1. To interact with the portal, you reach inside the box, going behind the screen instead of gesturing in front of it or touching it.
2. The two portals are networked so you can collaboratively play with someone in a shared space, even if they're physically far away.
3. The system recognizes and responds when special objects are inside, tracking the id, position, and rotation.
Videos
It makes more sense when you see it in action, so here's an edited version of a longer scenario video I made for the project which just highlights the main features.
Also, a short video of the Portals in action at 4 Hours Solid, my grad show:
The project was created in 2011-12 as part of an MFA Thesis in Graduate Media Design, at Art Center College of Design. If you want more details about the project visit Portals (and friends). One of my main goals of the project was to create something that other people could build upon and expand. So.... without further ado, let's get on to the How To part of things!
Note: my process of building these boxes wasn’t this straight forward. It was a super iterative process that involved lots of small tests. Even though I’m going to try to explain it in a more logical step based way, I would still recommend testing out parts of the hardware + software before diving in and building the entire thing for real.
“The visible world is no longer a reality and the unseen world is no longer a dream.”-Arthur Symons
Step 1: Supplies
1. MATERIALS (for a pair of Portals)
What you use to build the physical enclosures/structures is up to you really. And how much of it sort of depends on how big your box is.
• 1x sheet of drafting velum To diffuse the light box light.
• 3x 1/4” white acrylic (18” x 32”) To build the light boxes + poppets
• 2x 3/8” plywood (36” x 48”) The main cases
• 1x 1/8” plywood (18” x 24”) For the camera + light mounts
• 4 tiny hinges. For the box lids.
• 2 sheets of clear address label stickers Avery 1” x 2 5/8”
• Misc. adhesive + cutting supplies. A roll of velcro, white tape, elmer’s glue, acrylic cement, small screws, x-actos, laser cutter, woodshop, etc...
2. ELECTRONICS (for a pair of Portals)
They're not exactly cheap..
• 2x Newer iMacs. Running OSX 10.6.8. a newer/faster one is probably better.
• 2x flat-screen LCD monitor. I’m using 17inch Acer monitors with removable stands
• 2x display port to VGA adapters To hook up the monitors to the computers
• 2x HD webcam I’m using Logitech c910. Color and quality is great but there’s no manual control for anything which sucks. But the HD cams result in a pretty clean chroma key. I am definitely still inter- ested in an acceptable alternative to these cameras.
• 2x board camera with remote CCD Unibrain’s Fire-i Digital Board Camera with Remote CCD. These are Firewire and have a manual focus as well as an interchangeable lens.
• 2x lens for the Fire-i cameras 2.1mm seems to work alright for this distance.
• 2x firewire 400 to 800 adapters For the firewire cameras
• 2x set of ultra thin LED puck lights I have some by Amertac. One set contains 3 lights.
• 4x set of Ultra Thin LED strip lights Amertac. One set is 3 strips, you need 12 strips total.
• 1x power strip/surge protector
3. SOFTWARE + FILES
I’ll go into more detail about each of these later, but we’re basically going to be connecting all of these things together.
• Reactivision (http://reactivision.sourceforge.net/) Open source computer vision framework
• Osculator (http://www.osculator.net/) Interprets OSC messages
• NETLab Toolkit + HUB (http://www.netlabtoolkit.org/) Integrates tangible interaction + Flash
• Adobe Flash CS5+ (http://www.adobe.com/downloads/) You know. Flash.
• iChat (comes with your mac) Handles the video chat side of things
• CamTwist (http://camtwiststudio.com/) Layers all the video feeds together.
• Files (http://portals-and-friends.com/share/Portals_Files.zip) Includes program files, laser cutting files, and artwork
What you use to build the physical enclosures/structures is up to you really. And how much of it sort of depends on how big your box is.
• 1x sheet of drafting velum To diffuse the light box light.
• 3x 1/4” white acrylic (18” x 32”) To build the light boxes + poppets
• 2x 3/8” plywood (36” x 48”) The main cases
• 1x 1/8” plywood (18” x 24”) For the camera + light mounts
• 4 tiny hinges. For the box lids.
• 2 sheets of clear address label stickers Avery 1” x 2 5/8”
• Misc. adhesive + cutting supplies. A roll of velcro, white tape, elmer’s glue, acrylic cement, small screws, x-actos, laser cutter, woodshop, etc...
2. ELECTRONICS (for a pair of Portals)
They're not exactly cheap..
• 2x Newer iMacs. Running OSX 10.6.8. a newer/faster one is probably better.
• 2x flat-screen LCD monitor. I’m using 17inch Acer monitors with removable stands
• 2x display port to VGA adapters To hook up the monitors to the computers
• 2x HD webcam I’m using Logitech c910. Color and quality is great but there’s no manual control for anything which sucks. But the HD cams result in a pretty clean chroma key. I am definitely still inter- ested in an acceptable alternative to these cameras.
• 2x board camera with remote CCD Unibrain’s Fire-i Digital Board Camera with Remote CCD. These are Firewire and have a manual focus as well as an interchangeable lens.
• 2x lens for the Fire-i cameras 2.1mm seems to work alright for this distance.
• 2x firewire 400 to 800 adapters For the firewire cameras
• 2x set of ultra thin LED puck lights I have some by Amertac. One set contains 3 lights.
• 4x set of Ultra Thin LED strip lights Amertac. One set is 3 strips, you need 12 strips total.
• 1x power strip/surge protector
3. SOFTWARE + FILES
I’ll go into more detail about each of these later, but we’re basically going to be connecting all of these things together.
• Reactivision (http://reactivision.sourceforge.net/) Open source computer vision framework
• Osculator (http://www.osculator.net/) Interprets OSC messages
• NETLab Toolkit + HUB (http://www.netlabtoolkit.org/) Integrates tangible interaction + Flash
• Adobe Flash CS5+ (http://www.adobe.com/downloads/) You know. Flash.
• iChat (comes with your mac) Handles the video chat side of things
• CamTwist (http://camtwiststudio.com/) Layers all the video feeds together.
• Files (http://portals-and-friends.com/share/Portals_Files.zip) Includes program files, laser cutting files, and artwork
Step 2: Building the Boxes
Making the physical Portal containers is probably the most straight forward part of all.
1. Cut out the plywood pieces as shown. I used a 4 inch hole saw for the holes and the wood shop instructors helped me cut out the screen opening with the table saw by lowering the blade and cutting up into the wood from the bottom.
2. Assemble the box as shown. I used double sided tape (super thin, clear film like stuff) and an air nailer to put everything together. Wood glue would also work if you have the patience for that sort of thing.
3. Repeat first two steps for the second box.
4. Sand the boxes down, especially the holes. I did a pretty rough job sanding with 60 grit. I’m told I should have used like 220 or something, but I get impatient with sanding.
5. Finish the boxes with Danish Oil. I like the Danish Oil because you just rub it on with a rag and then rub it off a few minutes later. Easy and fast, and I think it looks nicer than a stain. I just used the natural color.
6. Take the stands off the back of the monitor, attach the various cables, and insert the monitor into the opening from the inside. Should be pretty snug, but I also put a strip of double sided foam tape along the top to keep it in there tighter, otherwise the screen has a tendency to pop out when being transported.
7. Attach the lid to the box using the hinges.
1. Cut out the plywood pieces as shown. I used a 4 inch hole saw for the holes and the wood shop instructors helped me cut out the screen opening with the table saw by lowering the blade and cutting up into the wood from the bottom.
2. Assemble the box as shown. I used double sided tape (super thin, clear film like stuff) and an air nailer to put everything together. Wood glue would also work if you have the patience for that sort of thing.
3. Repeat first two steps for the second box.
4. Sand the boxes down, especially the holes. I did a pretty rough job sanding with 60 grit. I’m told I should have used like 220 or something, but I get impatient with sanding.
5. Finish the boxes with Danish Oil. I like the Danish Oil because you just rub it on with a rag and then rub it off a few minutes later. Easy and fast, and I think it looks nicer than a stain. I just used the natural color.
6. Take the stands off the back of the monitor, attach the various cables, and insert the monitor into the opening from the inside. Should be pretty snug, but I also put a strip of double sided foam tape along the top to keep it in there tighter, otherwise the screen has a tendency to pop out when being transported.
7. Attach the lid to the box using the hinges.
Step 3: Building the Light Boxes
1. Open lightbox_a.ai and lightbox_b.ai, and convert as required for your laser cutter. (The ones we used at school only took Corel Draw files. If you don’t have access to a laser cutter you can also cut the pieces out of white foam core, like the first two versions of my light boxes, but you’ll have to accommodate for the difference in material thickness.)
2. Cut out 2 sheets of acrylic using lightbox_a.ai . This will make the back, base, and front for two boxes. (and some bumpers). Save the scraps for making the markers later. Cut out 1 sheet of acrylic using lightbox_b.ai. This one sheet contains the rest of the pieces for both boxes
3. Assemble a light box as shown using the acrylic cement to attach the pieces.
4. Install the camera: Screw in the lens to the camera. Attach the thin cable thing between the lens part and the main camera electronics. Attach the lens to the lens mount and the camera to the cam mount with screws (pre-drill!). Attach the mounts to the inside of the box where the dotted lines in the template indicate. Attach the firewire cable.
5. String the power through the hole in the back of the box and attach all the strips together using the included attachments. Install the LEDs on the red dotted lines. The one attached to the power cable goes in the front of the base and the rest go up the back. I just used two little pieces of double stick tape per strip to attach the lights. The first one will overlap the camera cable thing. Make sure everything works! You can also tape down the wire attachments to make it look nicer.
6. Cut out a piece of velum (12” x 15.5”) and attach it to the face of the lightbox using the masking tape. Make sure there’s a slight bend at the joint, not a hard crease. Trim off any excess velum.
7. Repeat for the second light box.
8. Before putting the light boxes into the larger main boxes, be sure to focus the lens of the camera. Plug the firewire (with adapter) into the computer. Open up Reactivision, set it to source image, and put something on the top of the velum to focus on. When it looks good, go ahead and put it in the box, stringing the wires out the back hole.
2. Cut out 2 sheets of acrylic using lightbox_a.ai . This will make the back, base, and front for two boxes. (and some bumpers). Save the scraps for making the markers later. Cut out 1 sheet of acrylic using lightbox_b.ai. This one sheet contains the rest of the pieces for both boxes
3. Assemble a light box as shown using the acrylic cement to attach the pieces.
4. Install the camera: Screw in the lens to the camera. Attach the thin cable thing between the lens part and the main camera electronics. Attach the lens to the lens mount and the camera to the cam mount with screws (pre-drill!). Attach the mounts to the inside of the box where the dotted lines in the template indicate. Attach the firewire cable.
5. String the power through the hole in the back of the box and attach all the strips together using the included attachments. Install the LEDs on the red dotted lines. The one attached to the power cable goes in the front of the base and the rest go up the back. I just used two little pieces of double stick tape per strip to attach the lights. The first one will overlap the camera cable thing. Make sure everything works! You can also tape down the wire attachments to make it look nicer.
6. Cut out a piece of velum (12” x 15.5”) and attach it to the face of the lightbox using the masking tape. Make sure there’s a slight bend at the joint, not a hard crease. Trim off any excess velum.
7. Repeat for the second light box.
8. Before putting the light boxes into the larger main boxes, be sure to focus the lens of the camera. Plug the firewire (with adapter) into the computer. Open up Reactivision, set it to source image, and put something on the top of the velum to focus on. When it looks good, go ahead and put it in the box, stringing the wires out the back hole.
Step 4: Building Light & Camera Mounts
1. Open mounts.ai (again, convert as required for your laser cutter) and cut it out of the 1/8” plywood sheet.
2. Assemble the triangular light mount as shown, I just used tape to hold it all together.
3. Attach 3 of the LED puck lights to the triangular mount using some velcro (fluffy side attached to the back of the lights). Attach them to one of the sides that doesn’t have the extra tabs on the ends.
4. Attach the side mounts inside the box with double stick foam tape, about 4 inches from the front and set the tabs of the light mount into the side mounts.
5. Repeat steps 2-4 for the second puck light mount.
6. Take apart the HD webcams’ attachment arm thing, making sure to save the metal brackets and tiny screws because we’ll use that to attach it to the mount.
7. Attach the camera to the mount with the screws. The holes in the mount are a little too big for the screws, so I wrapped it with some tape to really hold it in place.
8. Assemble the camera mount as shown using some elmer’s glue. It was originally designed to hold 2 cameras, but you only need one now, so you can ignore the second set of holes.
9. Attach the camera mount to the back of the monitor (top, centered) using some double sided foam tape.
10. Repeat steps 7-9 for the second webcam.
11. Run all the wires along the side and back of the inside of the main box and out the back hole, taping things down as needed to make it look nicer and cleaner.
2. Assemble the triangular light mount as shown, I just used tape to hold it all together.
3. Attach 3 of the LED puck lights to the triangular mount using some velcro (fluffy side attached to the back of the lights). Attach them to one of the sides that doesn’t have the extra tabs on the ends.
4. Attach the side mounts inside the box with double stick foam tape, about 4 inches from the front and set the tabs of the light mount into the side mounts.
5. Repeat steps 2-4 for the second puck light mount.
6. Take apart the HD webcams’ attachment arm thing, making sure to save the metal brackets and tiny screws because we’ll use that to attach it to the mount.
7. Attach the camera to the mount with the screws. The holes in the mount are a little too big for the screws, so I wrapped it with some tape to really hold it in place.
8. Assemble the camera mount as shown using some elmer’s glue. It was originally designed to hold 2 cameras, but you only need one now, so you can ignore the second set of holes.
9. Attach the camera mount to the back of the monitor (top, centered) using some double sided foam tape.
10. Repeat steps 7-9 for the second webcam.
11. Run all the wires along the side and back of the inside of the main box and out the back hole, taping things down as needed to make it look nicer and cleaner.
Step 5: Making Poppets
You can obviously design your own. But you can also just use mine for now.
1. Using one of the scrap acrylic pieces from the light box, laser cut the poppet base squares (poppet_base.ai). Using the other scrap acrylic, laser cut the poppet character outlines (poppet_chars.ai).
2. Print out the characters (02_stickers_sm.jpg) on the clear sticker sheet.
3. Roughly cut out the characters (so there’s plenty of excess sticky) and attach them to the appropriate acrylic characters, trying your best to line up the edges of the drawing with the edges of the acrylic.
4. Flip it over and use an x-acto knife to trim off the excess sticker sheet on the edges.
5. Repeat steps 3 and 4 for the back side of the characters as well.
6. Print out the fiducial markers (01_markers_sm.jpg) on the clear sticker paper. (These are the same markers included with reactivision, just scaled down to poppet size and arranged to be printed on the stickers)
7. Roughly cut out the fiducial markers and align them to the square bases. I included the small text on the bottom so I would know what the ID of each marker was. Trim off the excess after its attached.
8. Attach the characters to their bases using some acrylic cement.
1. Using one of the scrap acrylic pieces from the light box, laser cut the poppet base squares (poppet_base.ai). Using the other scrap acrylic, laser cut the poppet character outlines (poppet_chars.ai).
2. Print out the characters (02_stickers_sm.jpg) on the clear sticker sheet.
3. Roughly cut out the characters (so there’s plenty of excess sticky) and attach them to the appropriate acrylic characters, trying your best to line up the edges of the drawing with the edges of the acrylic.
4. Flip it over and use an x-acto knife to trim off the excess sticker sheet on the edges.
5. Repeat steps 3 and 4 for the back side of the characters as well.
6. Print out the fiducial markers (01_markers_sm.jpg) on the clear sticker paper. (These are the same markers included with reactivision, just scaled down to poppet size and arranged to be printed on the stickers)
7. Roughly cut out the fiducial markers and align them to the square bases. I included the small text on the bottom so I would know what the ID of each marker was. Trim off the excess after its attached.
8. Attach the characters to their bases using some acrylic cement.
Step 6: Setting Up the Software
Obviously plug everything into its appropriate power + port before getting this part up and running.
1. Turn on the lightbox lights and calibrate reactivision to account for the camera distortion by using the rectangular calibration pdf included in reactivision. I scaled it down so it was about 8 inches wide when printed. Follow the directions on the reactivision site.
2. Calibrate the camera in reactivision by pressing O. Place a fiducial marker on the surface and adjust the brightness, contrast, etc until reactivision is able to detect the ID of the marker. (It will display the number it thinks it is in blue). After everything is set up in reactivision I turn off the image by pressing N to save some CPU power. (Pressing T brings it back).
3. Open my poppet Osculator file (0327_markers). When a marker is detected in reactivision the corresponding line in Osculator will have a green box next to it. I recommend paying for Osculator because the unregistered version pops up a message occasionally that cuts off the communication temporarily, which is annoying.
4. Install the NETLab Toolkit and HUB according to the directions on the NETLab Toolkit site.
5. Launch the HUB app first (making sure you checked the Run in 32 bit mode as mentioned in the install instructions!), and minimize it. It just needs to run in the background for the Flash app to communicate to Osculator.
6. Open the Flash file (0327_poppet.fla) and run the app. If every- thing is working properly you should see an animation appear when a fiducial marker is detected. Move this window to the top side of your screen.
7. Do the same thing on the second computer.
8. Open up iChat and start a video chat with the second computer. Make sure the right camera is selected (should be logitech, not unibrain). Hide the local video and move the window to the other side of the screen (not overlapping with the flash window).
1. Turn on the lightbox lights and calibrate reactivision to account for the camera distortion by using the rectangular calibration pdf included in reactivision. I scaled it down so it was about 8 inches wide when printed. Follow the directions on the reactivision site.
2. Calibrate the camera in reactivision by pressing O. Place a fiducial marker on the surface and adjust the brightness, contrast, etc until reactivision is able to detect the ID of the marker. (It will display the number it thinks it is in blue). After everything is set up in reactivision I turn off the image by pressing N to save some CPU power. (Pressing T brings it back).
3. Open my poppet Osculator file (0327_markers). When a marker is detected in reactivision the corresponding line in Osculator will have a green box next to it. I recommend paying for Osculator because the unregistered version pops up a message occasionally that cuts off the communication temporarily, which is annoying.
4. Install the NETLab Toolkit and HUB according to the directions on the NETLab Toolkit site.
5. Launch the HUB app first (making sure you checked the Run in 32 bit mode as mentioned in the install instructions!), and minimize it. It just needs to run in the background for the Flash app to communicate to Osculator.
6. Open the Flash file (0327_poppet.fla) and run the app. If every- thing is working properly you should see an animation appear when a fiducial marker is detected. Move this window to the top side of your screen.
7. Do the same thing on the second computer.
8. Open up iChat and start a video chat with the second computer. Make sure the right camera is selected (should be logitech, not unibrain). Hide the local video and move the window to the other side of the screen (not overlapping with the flash window).
Step 7: Setting Up CamTwist
CamTwist is the magic sauce that brings all this stuff together. It’s also the thing that’s most likely to freeze up so if it’s getting slow I would restart your computer.
1. Open up CamTwist. Go to Preferences > General. Change the video size to 640 x 480 and check “Disable QuickTime Component.” Disabling Quicktime Component really helps keep CamTwist from eating too much CPU and freezing up. Restart CamTwist for the changes to take effect.
2. Double click “Slide Show” to set a still image as the base layer. Drag the bg image (background.jpg) into the settings box.
3. Go to Tools > Full Screen > Acer to select the second monitor and see what you’re doing.
4. In the video source list single click Webcam and press the PIP button at the bottom.
5. In Webcam Settings, make sure it’s selected the Logitech cam.
6. In PIP Settings drag the Scale slider all the way to the right. so the image fills the screen. Check the Chroma Key box. Drag Threshold slider all the way to the right.
7. Click on the colored (most likely white at first) bar between Chroma Key and Threshold to bring up the color picker. Use the magnifying glass tool to select the greyish background in the webcam view. You’ll have to play around with the slider on the right to get the right shade of grey for a clean chroma key. It helps if you have something in the box while doing this.
8. Go back to Video Source and click Desktop and click PIP again. In Desktop Settings uncheck “Full screen” but make sure “Resizable selection” is still checked.
9. Press “Select Capture Area” and a rectangle with square handles will appear. Drag it to the running flash app and press “Done Selecting”
10. In PIP Settings, check the “Clear Background” box. Drag the Scale Slider all the way to the right. Check Chroma Key. Drag Threshold all the way to the right. And with the Chroma Key color picker select the blue background. The webcam image should now be overlaid with the animation from the flash app.
11. Repeat steps 8-10 for the webcam feed, except when you select your capture area drag it to the video chat window. And when you do the chroma key color selection you’ll have to select the background of the video chat.
12. Obviously, do the same for the other computer as well.
13. If everything is working properly, congrats, you’re Teleportaling!
1. Open up CamTwist. Go to Preferences > General. Change the video size to 640 x 480 and check “Disable QuickTime Component.” Disabling Quicktime Component really helps keep CamTwist from eating too much CPU and freezing up. Restart CamTwist for the changes to take effect.
2. Double click “Slide Show” to set a still image as the base layer. Drag the bg image (background.jpg) into the settings box.
3. Go to Tools > Full Screen > Acer to select the second monitor and see what you’re doing.
4. In the video source list single click Webcam and press the PIP button at the bottom.
5. In Webcam Settings, make sure it’s selected the Logitech cam.
6. In PIP Settings drag the Scale slider all the way to the right. so the image fills the screen. Check the Chroma Key box. Drag Threshold slider all the way to the right.
7. Click on the colored (most likely white at first) bar between Chroma Key and Threshold to bring up the color picker. Use the magnifying glass tool to select the greyish background in the webcam view. You’ll have to play around with the slider on the right to get the right shade of grey for a clean chroma key. It helps if you have something in the box while doing this.
8. Go back to Video Source and click Desktop and click PIP again. In Desktop Settings uncheck “Full screen” but make sure “Resizable selection” is still checked.
9. Press “Select Capture Area” and a rectangle with square handles will appear. Drag it to the running flash app and press “Done Selecting”
10. In PIP Settings, check the “Clear Background” box. Drag the Scale Slider all the way to the right. Check Chroma Key. Drag Threshold all the way to the right. And with the Chroma Key color picker select the blue background. The webcam image should now be overlaid with the animation from the flash app.
11. Repeat steps 8-10 for the webcam feed, except when you select your capture area drag it to the video chat window. And when you do the chroma key color selection you’ll have to select the background of the video chat.
12. Obviously, do the same for the other computer as well.
13. If everything is working properly, congrats, you’re Teleportaling!
Step 8: Bonus: Grabatar Customization
Customize your Grabatar, using pens, markers, paints, paper, etc! Have fun!