Save the World One Drop at a Time, Part 3: Networking Your Sensors

For this project I worked together with Will Buchanan - Parts 1 and 2 - to create a home automation system that tracks water usage across fixtures in the home, (sink, shower, toilet etc). Using visual queues (including light color, duration and intensity) at each fixture the system should inform the user how much water they are using, real time. In addition, we wanted to send usage information to the cloud where the data could be parsed and visualized as a "household usage" dashboard.

Will's Instructables (Part 1 and 2) cover the arduino code and sensor circuit design for the piezo electric flow sensor (to be used at the fixtures), and the hall effect sensor (to be used at the main meter).

We combine the readings from all these sensors to create a cohesive picture of water across the home.

This Instructable focuses on

1. the way we used Pinocc.io modules to share data across "scouts" (mesh connected modules),
2. the way we connected to the cloud through the "scout leader" (wifi shield), and
3. the way we got the data into our persistent data store (data.sparkfun.com) for further analysis

You Need

1) Pinocc.io Starter Kit. https://pinocc.io/shop/detail/1007/-/starter-kit

2) Python installed locally (with Requests library installed)

3) Arduino IDE (version 1.57)

Why Mesh Networking?

Why not just use XBee Series 1s and a wifi shield? Will and I chose a meshed network for the following reasons:

1. Due to the limited range of each device, we liked the fact that mesh can be used to relay messages between nodes that are, alone, too far apart to communicate. In a large home, this kind of feature is important.
2. As we continue to build on this prototype, we want the ability to have the products talk to each other, sometimes in strange combinations. Having the ability to message specific products based on use case is an important future feature for us.

Why Pinocc.io?

Simple. The platform takes care of all the complexity of mesh networking and wifi bridging for us, so that we can do fast work and feel smart! (Gosh that sounds like a paid advert :-)

Step 1: Install Arduino IDE and Load the Bootstrap Sketch Onto the Lead Scout.

The first step is to install the required software.

As per Pinocc.io's instructions at http://support.pinocc.io/hc/en-us/articles/2029587...

1. Download the latest Arduino IDE (1.5.7 or higher).
2. Download Pinoccio's firmware files (2.3MB .zip) and move the files in the libraries and hardware folders into your Arduino Sketchbook directory's own libraries and hardware folders.
3. Launch the Arduino IDE
4. Make sure that "Pinoccio Scout" is selected under Tools > Board
5. Make sure "/dev/tty.usbmodem#" is selected under Tools > Port
6. Plug in the lead scout (the module with the wifi shield) into the USB port and turn it on.
7. Load the Boostrap sketch file at File → Examples →Pinoccio → Bootstrap.
8. Once the sketch is uploaded,click the Serial monitor icon at the top-right corner of the sketch window : "Autoscroll" (should be checked), "Newline" (or really, any option other than "No line ending")
9. Set the baud rate to 115200 baud. <-- don't forget this step, or you will see nonsense in the Serial Monitor!

Note: this Instructable was completed using Mac OS X. For Windows users, you'll need to complete these steps.

If you have any trouble, these instructions can also be found here.

Step 2: Configure Your Lead Scout's Wifi Settings

In order to get our network talking to the cloud, we needed to configure our lead scout to talk to the wifi network.

In the serial monitor view, with the scout connected and the bootstrap uploaded and running, enter the following ScoutScript commands (ref: https://docs.pinocc.io/scoutcommands.html)

1. wifi.config("[yournetworkname]","yournetworkpassword"]"); // if your network takes no password, simply enter empty quotes for the password. (The reference is currently out of date. The password parameter is NOT optional).
2. wifi.dhcp
3. wifi.reassociate

Give the scout leader a moment to connect and then enter "wifi.status". You should see a status of Connected. If you fail, troubleshoot.

Once the wifi.status indicates connection, congratulations, your network is connected!

An important note on Pinoccio and Wifi: We noticed during testing that Pinoccio cannot handle wifi connections that require more than just the standard network/password pair. For example, a Starbucks which requires a web authorization checkbox, will not work with the Scout Lead.

Furthermore, the Scout Lead does not appear to connect through a tethered iPhone. (Android remains untested)

Step 3: Add Additional Scouts Through Pinoccio HQ

Once your lead scout is connected, check in at HQ to ensure that its responsive.

https://hq.pinocc.io/

Notice that you can turn on and off the LED lights, change their colors, assign pins and check the battery charge level.

Also notice that there is a text entry field. This allows you to load ScoutScript commands directly onto the scout. Enter "print millis()" into that field and hit enter. A number (indicating the number of milliseconds since the module was last switched on should display).

You are now ready to start using ScoutScript. ScoutScript is a bitlash-based scripting language that sits on top of the compiled code that ties the hardware and networking stack together. https://docs.pinocc.io/scoutcommands.htm

Once you have verified that ScoutScript is yielding good values, its time to add additional scouts.

A little about the imagery of Pinocc.io.

Scouts are formed together in "Troops". For the purposes of our project, we need only one troop, corresponding to one household. Members of a troop communicate with each other using a lightweight mesh networking library. Each troop needs a "Lead Scout" with a "Wifi backpack" (wifi shield) to communicate with HQ.

In the left menu of the web-based hq console, click "add scout" and simply follow instructions. When prompted to install update, agree. This will update your scout's firmware. In our project we called one scout "sink" another scout "main" (to indicate main meter) and another scout "scout leader" (which communicates with the network.)

Add a "Sink" scout and a "Main Meter" scout for the purposes of this project. If you only have the starter kit (a scout lead and a scout), then feel free to make the Scout Leader the Main Meter module as well.

Step 4: Generate Flow Events

We designed the system so that each scout will listen for "flow events" from its sensor and then communicate those events to Pinoccio HQ. The plan was then to listen for those reports using the API and put the data into some form of online storage.

Note: The specific Arduino code and sensor circuit that detects flow in either the fixture (piezo) or the main meter (hall effect) is covered in Part 4. This Instructable simply covers the basic use case: communicating flow events to Pinocc.io HQ, where we can grab it for our nefarious Cloud-based purposes.

To communicate a flow event to the cloud we used the ScoutScript "hq.report" function. (Linked)

To view the stream of events getting sent to HQ, we use the Pinoccio API. The API is actually pretty cool. Using the web (javascript, etc) you can send specific commands to specific scouts, specific pins on specific scouts. Likewise, you could use certain pin inputs as triggers for some web-based activity. This is extremely powerful and we only got to scratch the surface of what this allows.

The API function we will use is called sync. (reference.)

https://api.pinocc.io/v1/sync?token= [ your security token ]

To get your API token, follow the below steps.

• On the command line, enter this command:
  • curl -X POST -v --data '{"email":"youremail","password":"yourpassword"}' https://api.pinocc.io/v1/login
• Copy the reply. (Don't lose this): {"data":{"token":"man6on560fiio2289llqjo5sb1","account":27}}

Open the sync stream API call in your browser

To view the sync stream in your browser, simply enter the sync service url:

https://api.pinocc.io/v1/sync?token=

Generate an HQ custom report from each scout and ensure it appears in the sync stream.

In HQ command field with your "Sink" scout selected, or in the Serial Console (with a Bootstrap-enabled sketch uploaded to your Piezo sink module), enter:

hq.report("fixture-event", "Hello World") // Hit enter

Now, in the API (opened in a browser page above), view the sync stream and look for the "fixture-event" custom report. Verify that this shows up in the feed.

Now, switch over to the "Main Meter" scout if you have set one up. For the purposes of this Instructable, this could be any other Pinoccio scout, added to the troop. Enter an hq.report into the HQ command field, or the Serial Monitor (like above).

hq.report("gallons_at_main", "Why Hello Again!")

Check your API sync stream again. You should see this custom event appear in your feed now, as well.

Once you have verified that the HQ reports from the scouts are making it up to HQ, skip to the next step.

If the reports are not making it into your sync stream, its likely that your Lead Scout has disconnected.

Step 5: Format the HQ Custom Reports

Because we wanted to capture actual flow events (fixture-events and gallons at main), we came up with the following formats.

For any fixture (sink, toilet etc) the custom report looks like this:

hq.report("fixture-event", "duration-[milliseconds flow lasted]|avg_flow_rate-[rate of flow in gallons per minute detected at the sensor]")

We essential overstuffed the event value with two values, delimited by the pipe '|' character. We took this approach because, with latency, with dozens of events firing per minute, we wanted to make sure that the flow duration and flow rate remained bound together in a single report.

So, a sample might look like.

hq.report("fixture-event", "duration-12336|avg_flow_rate-.56")

For the main meter sensor (which uses a hall effect sensor - see Will's Instructable), the custom report looks like this:

hq.report("gallons_at_main", "[gallons used]")

So a sample might look like:

hq.report("gallons_at_main", "0.76")

If you have followed Will's instructions to the letter, each sensor sketch will periodically create these HQ reports, formatted in this way. For the purposes of this Instructable, we are manually generating these reports to allow for easy testing.

Summed up:

Simply enter

hq.report("fixture-event", "duration-12336|avg_flow_rate-.56")

and

hq.report("gallons_at_main", "0.76")

and watch them appear on your sync stream.

Step 6: Set Up a Stream to Store Your Data at Data.sparkfun.com

For the purposes of this project, we chose to use data.sparkfun.com to store our sensor data. We chose this location due to ease of use of the API, ability to download as a CSV file, and the open source spirit of sparkfun. Yay Sparkfun!

To set up a repository to store your data, create a new "stream" on the data.sparkfun.com site. Use the following column names.

average_flow_rate,duration,gallons,scout_id, troop_id and enter the rest of the information how you like.

Keep track of all the keys you will use. They will come in handy.

When your stream is set up, test it by entering the following url in the browser:

http://data.sparkfun.com/input/[YOUR PUBLIC_KEY]?private_key=[YOUR PRIVATE_KEY]&gallons=0.56&avg_flow_rate=0&duration=3067&scout_id=2&troop_id=4

This should insert a row in your new stream. You can always check the contents of your stream here:

https://data.sparkfun.com/streams/[Public Key]

Congrats!

So, now you have a source stream (Pinoccio sync stream) and a destination stream (data.sparkfun.com). The trick is to get the select data from one to the other. For that, we used Python. (Skip to next step)

Note: we chose to have the piezo (sink and fixtures) write exclusively to the flow rate and duration columns, and have the hall effect sensor (main meter) write exclusively to the gallons column. All non-exclusive columns are simply set to zero.

We were able to visually tare test the system by comparing the flow at the main - as reported on data.sparkfun.com - with the flow at the fixture - as also reported on sparkfun - and got 99% accuracy when correlating the two!

Step 7: Run the Python to Pull From Pinoccio Sync Stream and Place It Into the Sparkfun Stream.

Ordinarily one would want to run an ETL (Extract, Transform, Load) script like this in a place where it can run constantly. For the purposes of this project, we ran it on our own computers using Python and the Requests package.

This script pulls from this stream

https://api.pinocc.io/v1/sync?token= [ your security token ]

And pushes to

http://data.sparkfun.com/input/[YOUR PUBLIC_KEY]?private_key=[YOUR PRIVATE_KEY]&gallons=0.56&avg_flow_rate=0&duration=3067&scout_id=2&troop_id=4

It expects the format of the source stream hq reports to be exactly those generated by the following commands:

hq.report("fixture-event", "duration-12336|avg_flow_rate-.56")

hq.report("gallons_at_main", "0.76")

To run the script, download the file.

1. Remove the .txt as the extension. The extension should be ".py"
2. Open the terminal and navigate to the script.
3. In the terminal window enter "python [file name].py"
4. The output should remain active, displaying new events as they arise.
5. To test, enter a few hq commands.

If everything worked, entering a manual hq report should show up in the following places:

1. In the sync stream (https://api.pinocc.io/v1/sync?token=[pinoccio token]
2. In the python output
3. On the data.sparkfun.com stream. (https://data.sparkfun.com/streams/[Public Key])

Congrats. You now have events getting generated at each scout, and making it all the way to the sparkfun repository.

Step 8: Part 3 Conclusion, Next Steps.

What we found through testing is that we could "tare test" the flow at the fixtures against the flow detected at the main. We then calibrated both of those against an actual filling of a gallon jug. We found that our flow detection worked extremely well, yielding 99% accuracy before tare, and 100% accuracy after.

Now that we are getting data into Sparkfun, the next step is to use that data. Visualizations, benchmarking and dashboards would be a next obvious step. We are considering a streaming visualization such as Plotly's streaming API.

If you want to get involved, please let us know! We'd love your help taking this to the next level.

Next Step Part 4. Write and upload your Arduino Code.