A Machine That Listens

If you think about it, machines and computers are like us in many ways — they have brains (processors), eyes (cameras), ears (microphones), etc. Machine Senses is a series of artifacts that aim to reverse the perspective on how we perceive technology. How do machines see and sense us? How do we humans feel about being seen and sensed by them? What if machines have personalities and feelings just like we do? The conceptual yet functional objects (Nose Smoke Detector, Ear Microphone, and Eye Stickers) serve as reminders that as we design and develop new technologies, we must not forget that machines can and should be as human as the ones interacting with them.

As a capstone piece of the Machine Senses series, ‘A Machine That Listens,’ is a conceptual interactive art piece that takes the naive personality of a child. In contrast to what we normally think of when we hear the terms “artificial intelligence,” “supercomputer,” or “machine learning,” A Machine That Listen takes audio input from humans and displays what it hears in both text and visuals found on the internet, similar to a child trying to learn a language.

This instructable details how one goes about designing a simple, playful, and interactive art object that remind us that machines are, in many ways, just like us humans.

Step 1: Screen & Computer

The first prototype of “A Machine That Listens” was built around a 24” Dell Monitor. It was immediately evident that the size of the piece was very un-child-like (as well as the black color). I had to find a way to reduce the size of the machine, but small flat screen monitors are hard to come by these days. I considered hacking apart a laptop, flipping the screen around itself like this MacBook hack, but this proved too technical. I considered buying a screen from Adafruit, but that proved to be too expensive (I'd rather not pay $200+ for a screen). Luckily, there is an iPad+Mac app called Duet Display that allows for iPads to be used as a display, and there happens to be an iPad and a Mac Mini laying around here at Autodesk Pier 9, so I decided to borrow and design the piece around them.

In this particular design, I used the first generation iPad Air, in combination with a Core i5 Mac Mini. The dimensions of the box in Step 2 are based off from these two pieces of hardware.

Step 2: Container Box

After a few trial-and-errors with the sizes of the box, I decided on the the W 9” x H 12” x D 4” dimensions, with a removable back cover. Using an Epilog laser cutter, I cut the front and back panels, along with appropriate pieces that would hold the iPad and the back cover nicely. Optionally, this can be done manually without the laser cutter.

As for the top and side pieces, I used 0.5” MDF (cut using a table saw) because of its thickness. I then glued the box together and sand the edges and the surfaces down using an orbital sander, ready for them to be primed and painted. I also had to drill 3 holes: one for the microphone holder (see Step 3), one for the microphone cable back into the box, and one for Mac Mini’s power cord in the back panel.

One minor thing: I had to design a twistable tab to lock the iPad in place (and to unlock it easily when I want to). I cut these tabs on a laser cutter and hold it onto the inside of the front panel by tiny screws.

The spray painting process was a bit of a pain and a major learning lesson for me. Long story short, I didn’t let the primer to fully dry, and sanded with an orbital sander (too powerful) instead of hand-sanding it. Eventually I got a fairly smooth surface to my liking, only after days and days of frustration. The mantra that is helpful to keep in mind is “Slow is smooth; smooth is fast” (sage advice passed on by Pier 9 workshop’s Trent Still).

Step 3: Microphone + Holder

Initially, I planned to use the "Ear Microphone" I previously built for this piece, but decided against it, because I wanted to keep it simple and focus to be on the machine and not the microphone.

I used a cheap-ish handheld microphone from RadioShack. I then took it apart down to the barebones (even disconnecting the cables inside) and designed a new handle for it. This was done purely for aesthetic reasons — there’s something sexy about a spherical-shaped handheld microphone. With this particular microphone I'm working with, I wanted to keep the top of the microphone bare and have the bottom rubbery part poking out. I measured the appropriate dimensions and used them when I design the 3D piece in Autodesk Fusion 360.

Another important feature/detail is the microphone holder that attaches the microphone to the machine. I tried two design variations: a snap fit (think lego man’s hand), and a magnetic attachment. While the magnetic attachment feels very nice, it doesn’t hold too much weight and didn’t work reliably in my design, so I decided to go with the snap fit design instead. Again, this was designed in Autoesk Fusion 360.

Both the microphone handle and the holder were printed on the Fortus FDM machine using ASA black material. During reassembly, I had to solder the microphone top to the cable coming from the bottom of the microphone cone.

Step 4: Code

I was initially planning to use a combination of Chrome speech API and Processing to run the simple sketch that would take in voice input and do something visual with the data. After poking around with websocket on Processing and failing to make it work, I googled around and found this p5.speech library. P5.js is a platform that I’m fairly comfortable with (thanks to this partial 100 days of P5.js project I did last fall), plus it already works on the web, so it can take advantage of the Chrome speech API.

The speech/word recognition works like magic. I then wanted to take the words and do a google image search with them (as if the child computer takes in the voice input, and looks it up on the internet). This can already be done on a normal Google Image Search using speech (the result would look like a typical Google Images search result / tiled), but require the user to click a button every time he/she wants to start a search. I seriously considered reconfiguring my system so that there’s a button on the microphone, but in the end decided to work with the code I already have. To do this, I needed try to find a way to make the computer listen continuously and work with Google’s Custom Search API.

I enlisted the help of my ex-coworker friend, Kevin Ho, to help me with this. The whole API, JSON, JQuery calls were a bit over my head. He wrote a super bare bone sketch that pulls one image based on the speech input. I then make little tweaks to make the query text go over the image result with a slightly opaque black background.

I had to get my own API key, which can be done through the Google Developer Console. This API key is used in SimpleSearch.js to grab the first image search result from Google.

Step 5: Putting It All Together

Attach the microphone holder to the box using a wood screw. To fit the microphone cable through the hole, I had to cut the cable and re-solder it.

Plug the microphone and the iPad into the Mac Mini. Fire up the machine and run the listener.html on Chrome (I put the file on dropbox and run the public link), with the iPad as a display. Finally, I put the entire assembly into the box, close the back cover up, and voila! — A Machine That Listens!