Sensory Extensions for Ratio and Proportion (PhET Interactive Simulations)

In this Instructable you'll learn how to create and program sensory extension devices that interface with Ratio and Proportion (a PhET Interactive Simulation). Sensory extensions are technologies that augment human senses beyond biological capabilities. They typically take in qualities of an object that humans have difficulties perceiving and transform that information into a perceptible format. The sensory extension devices described in this Instructable use force, distance, sound frequency, and magnetic field strength to explore different ratios in the simulation. Using sensory extensions as multimodal input for interactive simulations enables users to connect their physical/sensory experiences with simulated concepts!

This material is based upon work supported by the National Science Foundation under Award #2119303.

This project was developed with the Craft Tech Lab, PhET Interactive Simulations, and in the ATLAS Institute at The University of Colorado, Boulder.

Thanks to my collaborators: Sammie Crowder & Casey Hunt!

Supplies

All the components (w/ purchase links!) used in this Instructable can be found in the attached spreadsheet. Depending on how you choose to build the devices, you may not need every item on the list. For example, you can use regular jumper wire in place of the Adafruit cable listed for each device, but you will need the Grove cable pigtail.

Step 1: Communication

Each sensory extension device consists of a sensor (force, distance, sound frequency, magnetic field strength) that is connected to a microcontroller. Each microcontroller uses Bluetooth to communicate with the simulation. The simulation uses Chrome's Web Bluetooth API to read in the sensor values and reacts with multimodal outputs when creating ratios (text descriptions, sound, visuals). In this Instructable the Bluetooth communication code is included in each .ino file, so I'll you need to do is upload the code to the microcontroller and then pair the device to the simulation.

Step 2: Control Unit

You'll need two sensory extension devices (two control units & two sensors) to interact with the Ratio and Proportion sim. I 3D printed the housing for the control units in PETG, without any supports. Once you have the bottom plate, secure the microcontroller and expansion board to the substrate using four standoffs. I also recommend now placing the battery into its slot. Then add two more standoffs to the corners of the bottom plate. You now have your control unit, repeat this step again for the other side!

Step 3: Weight Sensor

The housing for the weight sensory extension is printed in five parts: a bottom plate, a button, the button platform, a top plate (remix of ixp's design), and an elastic band. The unit's top, bottom, and button files are printed in PETG, and the platform file is printed in TPU. The force sensor is glued to the bottom plate, and the elastic band is threaded through the top and bottom plates to secure everything together. The button is placed inside the top plate and then the platform is fit-pressed into the button.

To connect the sensor to the microcontroller, we used a standard hookup for a force-sensitive resistor and connected the sensor to the expansion board using around 500mm worth of wire. Upload the .ino code file for the corresponding control unit and look at the values in the serial monitor. If everything is working correctly you can then connect to the sim and make ratios by holding objects of different weight.

Step 4: Sound Frequency Sensor

The ear (a remix of printable_models' design) can be printed in either TPU or PETG (depending if you want a squishy feel) and the MEMs microphone is secured in the ear with hot glue and sticky-backed velcro. To be able to detect a higher range of frequencies (up to 21k) we need to connect the mic to an op-amp to amplify the signal. The mic and op-amp hookup can be seen in figure 3, I also recommend using at least 800 mm worth of wire so that the mic/ear can be worn on different parts of the body. After soldering, upload the code and check that your device is working properly by using a tone generator to measure the frequency in the serial monitor.

Step 5: Magnetic Field Sensor

Once you have the GMR sensor and PCB, solder them together using the same orientation as shown in figure 4. It's super important that the sensor is placed correctly! Then make the connections between the sensor/PCB and microcontroller using around 400mm of wire. The finger sleeve (a remix of ZackFreedman design) is printed in TPU, and the sensor/PCB slides inside. Test the device by opening the serial monitor and holding the sensor near an object with a magnetic field, the values should get higher as you get closer to the field.

Step 6: Distance Sensor

This is the most straightforward sensor to build and connect to a control unit. The rings are printed in PETG and designed so that the distance sensors can be snapped into them. Then use a cable with a STEMMA QT / Qwiic header to plug into the sensor and plug the other end into the expansion board. If the values get larger in the serial monitor as you move your hand away from a surface, the device works.

Step 7: Connecting to Ratio and Proportion

Once you have two sensory extension devices (two sensors & two microcontrollers) you're now ready to connect to Ratio and Proportion! First, select either the "BLE left device" or "BLE right device" and then a dialog box should open in the top left of the window, then select either the "nrf52L" or "nrf52R" and press "pair". Repeat the process for the other side, and now both sensory extension devices are connected to the sim. Have fun making ratios!