Introduction: Raspberry Pi3 + XBee + XBMQ + MQTT + Node-Red IoT

By
About: Just a normal guy that enjoys the DIY stuff

Hello!

This instructable is comprised with the notes I took during the implementation of the project. What I have working at home now is a system managed by a Raspberry Pi that is controlling the well Water Pump and receives the signal from a cheap Water Level Sensor on the Water deposit.


Just so you know, the Water Deposit and the Water Pump are quite far away from each other, I could not use cheap wi-fi receivers as the signal is lost. So, my old XBee's that were gathering dust came in very handy for the project. Also, the Water Pump takes a long, long time to fill the deposit, so we had some incidents where people activated the pump and forgetting about it.

Following constrains were observed:

1. Open Source software that will be running on locally owned and managed hardware (I have suffered from google or other third parties shutting down, shrinking public available api / projects, causing some of my home diy projects not operational anymore)
2. Hardware to be supported and available to get this project going:
3. Raspberry Pi (old model A for testing)
4. Raspberry Pi3 for the live system,
5. XBee's S2 (I got mines from Sparkfun in 2012, I tested first with these)
6. XBee's PRO S1 (Also, got them from Sparkfun, the live system is running with these)
7. XBee Explorer (I have used the Parallax XBee Explorer, and the Adafruit XBee Explorer, in fact they seem to be the same)

Experiences with Google shutting down services, deprecating API's and shrinking access, made me wary about using any cloud-based service for locally needed services.

As usual, it is assumed you have common sense and exercise it. It is assumed you know at least the basics about electronics. It is assumed you will not connect power electric devices to "signal" level electronic devices unless you know what you are doing. I take no responsibility if you fry your stuff, nor if you damage property or beings.

ToDo's

-Add security to the MQTT Broker.

-Detail the node-red part (if requested by 10 folks or more ja! dreaming..)

-Use a nice fontend GUI like the ones NetBeast can build. Too bad my NetBeast install went so wrong I had to reinstall Raspbian again. Alternatives, anyone ?

Step 1: Get Raspbian Running on the Raspberry Pi3

If you already have a working Raspbian installation, just take a quick look.

Installation of Raspbian OS

Of course, there are a lot of far better tutorials at how to install the Raspbian OS on the Raspberry Pi3, the only way for me to get the Raspbian image on a SD Card, was by means of my Windows laptop, so what I did was:


1. Go to the Raspbian download page, and get the full desktop image as a .zip file
2. Extract the "2016-05-27-raspbian-jessie.img" file from the downloaded .zip file. The current version will have a different date/name, according to the release date.
3. Get the Win32diskimager software, and write the "2016-05-27-raspbian-jessie.img" (or the one you downloaded) to an SD Card. CAREFUL: this will erase whatever information stored on the SD card and replace it with a brand new Raspbian installation, it will take a while (mine took 5 minutes or so).

After this you will have a SD Card with Raspbian ready to be inserted on your Raspberry Pi3. So:

1. Plug the SD Card into your Raspberry Pi3
2. Make sure you have your HDMI monitor/tv hooked up. Also a USB Keyboard / mouse if you are going to work directly on the Raspberry Pi3
3. Plug power onto your Raspberry Pi3, cross fingers and watch it boot (it is like operating the automated garage door for the first time)

If everything went ok, you will end up watching a beautiful RaspBerry on your output monitor/tv. That is the Raspbian Desktop.

Minimal Customization

Now let's make a minimal customization on your Pi. Open a Terminal window so you can input some commands. There are a lot of ways to do this:

Using the Raspbian Desktop:

You can use the mouse to click the small blackish rectangle on the top panel.

Or go to the Menu -> Accessories -> Terminal

If you go with this method you will not be asked your login and password, as you will be logged locally.

Using putty from a Windows laptop (the one I use)

Just download and execute putty, input the assigned IP address and click "Open"

Using ssh from another Linux Terminal (my preferred)

ssh pi@assigned_ip_address

After you have connected to your Pi, you will be asked for the login and password.


The default user is "pi" and the default password is "raspberry"

Once you are connected or in the terminal, start by


sudo raspi-config



Expand Filesystem

This command will take you to the "Raspberry Pi Configuration Tool". Use the up/down arrows to select

Expand Filesystem, and the left/right arrows to be able to use and . After it is done, it will ask you for a reboot. Do it.When we used the Win32diskimager to get Raspbian into the micro SD Card, not all the available capacity of the memory card is available to be used by the Raspbian OS when it boots up. So this option will reclaim the entire SD Card spaceHostnameAfter changing the hostname it will ask for a reboot. Do it. If not, you can reboot your Pi from the Terminal with:

sudo reboot -n


That means "I am a privileged user, reboot the system now!!!!"I changed the hostname on my Raspberry Pi3 to chchbt. You change it to whatever you like, just remember: No spaces No special characters Keep it simpleDisable serial TerminalIf you know that will not be using a serial interface, disable it

Change hostname

Using the sudo raspi-config command, also change the hostname of your Raspberry Pi3. I changed mine to chchbt and its the hostname I will refer from now on.

Change Password

Using the sudo raspi-config command, also change your default password from the default "raspberry" to something else.

Step 2: Installation of MQTT Broker, Mosquitto

MQTT

Once you got your Raspbian up and running, now we need an MQTT Broker.

There are a lot of places that explain what a MQTT Broker is, examples:

Hive MQ explanation: These guys provide a public MQTT Broker (there is a free version). Remember, we will try to get everything up and running from home, on our own hardware, but the learning material from HIVE is quite good.

If you make me explain what a MQTT system is, I'd provide a neighborhood analogy.


• Imagine your neighborhood as an MQTT system, every home (Topic) at some point "Publishes" information and those ones "Susbscribed" to the "Topic" need to know the new gossip. The Sensors sometimes need to give information to other Topics or get information from other Sensors as well.
• There is this special lady or guy in the neighborhood, the "Gossip Lady or Lad" that has no life on its own, it's only purpose is to carry Gossip from one home to another. This individual is the "Broker". It just makes information available to the people interested in other people's life.
• Whenever a member of the 'hood wants to bring "information" to other members, they "Publish" the information or Gossip with the "Gossip Lady". She gives away the info to the proper Home (or "Topic") so all the people that is "Subscribed" to the "Topic" get the new gossip.


Simple huh? When I realized that MQTT was a system to deliver small bits of information on unreliable networks "Gossip Lady" and the 'hood came immediately to my mind.

Install Mosquitto

So first, we need the MQTT Broker installed on our Raspberry Pi3.

From the Raspberry.org forums this is the procedure:

A) Login to your Raspberry Pi3 Terminal

I am assuming you have recently made an apt-get update, if not, issue the command

sudo apt-get update

B) Issue the following command:

sudo apt-get install mosquitto

That's it. If no errors were shown, you have a Gossip Lady on you Pi3.

Test your Mosquitto installation

Now we need to know if we can deliver "Publish" messages and retrieve them whenever we are "Subscrbed" to a Topic.

From an Android device connected to the same network as your Raspberry Pi3

A) I downloaded the IoT MQTT Dashboard, is an easy to use MQTT client for android devices.

B) Configured the MQTT connection to my Raspberry Pi3 (Attached Photo).

C) Subscribed myself to the "Gossip" Topic.

D) Created a Publish push button that whenever is pushed, will send some fixed text to the "Gossip" Topic.

E) Tested it from another Android device, used a different text string to be sent when the button is pushed

From a Windows machine

A) Download an MQTT client like MQTT.fx or mqtt-spy (both are java-based)

B) Launch it and configure a connection to your MQTT Broker (photo attached for MQTT.fx)


1. Configure connection
2. Enter a friendly name of your MQTT connection
3. Use the IP address of your Raspberry Pi3
4. Save your connection
5. Subscribe to the Gossip topic
6. Publish something to the Gossip topic.
7. If you have another MQTT client, lets say an Android device, check the messages on that device also.

Step 3: Get Those XBee Modules Ready

For configuring the XBee's modules, there is this tool, XCTU, from Digi. I was used to the ugly-impossible non user friendly old interface that is called now the "Legacy XCTU". But good news for everyone! The new XCTU version with revamped GUI and visible options is just great. Is what should have been since 2008.

There are great tutorials at how to get started with XBee´s like the one in Sparkfun Exploring XBee,

I took some screenshots and commented them here on this step so you can have an overview of my configuration, note the following:

• All your XBee devices have to be the same value for ID : PAN_ID parameter
• I did set encryption on my devices, by enabling parameter EE, and used a private KY:Encription_Key
• I use the same Digital Out pins on my XBee's for simplicity's sake
• I use the same Digital IN pins on my XBee's for simplicity's sake

Very Important, write down the MAC address of your devices (you can retrieve those MAC addresses using XCTU), you will need them to access MQTT topics.

I want to use the WATERPUMP XBee to crank up a big motor, but the XBee's will only carry the signal, so that is why I am using the XBee Pin 12 out signal to power a small relay that powers up a good 'old Schneider Contactor, and this contactor know how to handle big inductive loads like WATERPUMPs.

Before you comment on the use of big SSR's instead of good 'old Contactors, please make sure you have the right information about SSR's and inductive loads.

Step 4: XBMQ-Java, Make Those XBee's Talk to MQTT

Now we need to bring the messages generated by XBee devices into MQTT.

There is a great tool for that (small jewel, even) it is called XBMQ-java. I use it to get the XBee digital pins (reads and writes) to MQTT Topics.

Get it onto your Raspberry Pi3, so log in into your Raspberry Pi3, once inside change to your Desktop.

cd Desktop

As we are going to clone the GitHub project, I want everything in the users Desktop, so even if I login locally I will have quick access to the GitHub cloned folders.

First, make sure we will have RXTX working ok

sudo apt-get install librxtx-java

And clone the git repository

git clone https://github.com/angryelectron/xbmq-java

Now get the XBMQ git project. And change to the created folder

cd xbmq-java/

You can always try a "List files inside the folder"

ls -l

And put the required .jar file (latest version)

wget https://github.com/angryelectron/xbmq-java/releas...

Now, in your /home/pi/Desktop/xbmq-java folder you will have something like:

ls -l

pi@chchbt:~/Desktop/xbmq-java $ ls -l
total 196 -rw-r--r-- 1 pi pi 4247 Aug 1 18:28 build.xml

drwxr-xr-x 5 pi pi 4096 Aug 1 18:28 lib

-rw-r--r-- 1 pi pi 82 Aug 1 18:28 manifest.mf

drwxr-xr-x 2 pi pi 4096 Aug 1 18:28 nbproject

-rw-r--r-- 1 pi pi 9260 Aug 1 18:28 README.md

drwxr-xr-x 2 pi pi 4096 Aug 1 18:28 resources

drwxr-xr-x 7 pi pi 4096 Aug 1 18:28 rxtx-2.2pre2-bins

drwxr-xr-x 3 pi pi 4096 Aug 1 18:28 src

drwxr-xr-x 3 pi pi 4096 Aug 1 18:28 test

-rw-r--r-- 1 pi pi 69769 Nov 2 2015 xbmq.jar

-rw-r--r-- 1 pi pi 88 Aug 1 18:28 xbmq.properties

Ok!!! Now lets connect your Coordinator XBee to your Raspberry Pi3. I connected mine using the XBee Explorer USB interface. After it has was connected I used the following command to see if it was detected:

ls -l /dev/ttyUSB*

That means "list me all serial devices connected through USB... please? "

And gladly I got:

pi@chchbt:~/Desktop/xbmq-java $ ls -l /dev/ttyUSB*
crw-rw---- 1 root dialout 188, 0 Aug 1 03:17 /dev/ttyUSB0

Now I know /dev/ttyUSB0 is my XBee explorer with my Coordinator "chchbt" XBee. If you are brave enough and want to skip the XBee explorer, you will connect to TX, RX, 3.3V, GND pins from your Raspberry Pi3 to the XBee module. and you will access it mostly from /dev/ttyAMA0 (i did it like that on my tests on the Raspberry Pi A, and got into some required tweaking, like disabling the serial interface and so on, that is why we did that on Step 1)

In order to XBMQ-Java talk to the XBee's, that port needs to be configured on the xbmq.proerties file, so edit the file

nano xbmq.properties

and

1. Edit the port to match yours, for me it is port = /dev/ttyUSB0
2. Edit the root topic, xbmq uses greenhouse, so I changed it rootTopic 0 greenhouse
3. And the broker is running locally, so you can do broker = tcp://localhost:1883

Once you finish you need to hit ctrl+x , after that nano will ask you to save the changes, so just hit "y" and "enter"

Now, get the xbmq executable and updater to the /home/pi/Desktop/xbmq-java folder:

cp resources/*.sh /home/pi/Desktop/xbmq-java/

...make xbmq.sh and xbm-update.sh executable

chmod +x xbmq.sh
chmod +x xbmq-update.sh

Now you can test by running ./xbmq.sh, and go for the TESTS. But lets just finish, and add the raspbian xbmqd Daemon to start every time your Raspberry Pi3 starts

Edit it to point to your xbmq-java files location

nano xbmqd

Change the "Daemon" location to the one we are using, that is

DAEMON="/home/pi/Desktop/xbmq-java/xbmq.sh"

Once you finish you need to hit ctrl+x , after that nano will ask you to save the changes, so just hit "y" and "enter"

cp resources/xbmqd /home/pi/Desktop/xbmq-java/

chmod +x xbmqdsudo

cp xbmqd /etc/init.d/

sudo update-rc.d xbmqd defaults

Are we ready? Let's reboot and find out:

sudo reboot -n

After rebooting, if the xbmq-java Daemon started correctly, we should be able to go to an MQTT client and look for the status of the pins in our XBee's

So, have at hand the

• Coordinator XBee MAC address
• Router XBee Devices MAC addresses

Step 5: TESTS MQTT + Xbmq-java + Node-red

If everything went good, we can see the Digital pin status on the WATERPUMP and WATERLEVEL XBee devices.

So log into your MQTT client (for the tests I used MQTT.fx, but you can check on your Androit MQTT Dashboard if you like)

Remember I told you to have at hand your XBee devices MAC addresses ?

You will need the {Coordinator} or gateway MAC Address, mine is called chchbt and has the 0013A200408C562C MAC

You will need the {Router} MAC Adress, I got two:

WATERPUMP 0013A200408C55D9
WATERLEVEL 0013A200408C56D7

So, let's say I want to know the status of the Digital Pin 12 on my WATERPUMP XBee, I will have to:

Subscribe to the following topics:

syntax is : greenhouse/{Coordinator}/{Router}/io/DIO12

replacing it I will subscribe to:

greenhouse/0013A200408C562C/0013A200408C55D9/io/DIO12

If I publish 1's or 0's to this topic, the DIO12 of my WATERPUMP Xbee will change. Use a led on yours if you want to test.


Know the status of the WATERLEVEL XBee PIN 4

Subscribe to the following topics:
syntax is : greenhouse/{Coordinator}/{Router}/io/DIO4_AD4 replacing it I will subscribe to:

greenhouse/0013A200408C562C/0013A200408C56D7/io/DIO4_AD4

So amazing! I can receive 0's and 1's !!!. Ok, now let's create a Switch that will change the status of the WATERPUMP XBee Pin12, so the relay engages and turns the PUMP On.

On the MQTT Dashboard app on your Android device, go to Publish > +

Create a type switch publication, use the topic that will allow you to set the Digital Pin 12 on your actuator XBee (mine is WATERPUMP) and publish 1's or 0's

Ok, now we need to integrate some logic to prevent the WATERPUMP to be activated if the WATERLEVEL sensor knows the Water Deposit is full.

Here comes greatest node-red, you can install it by

sudo apt-get install nodered

But it already comes with Raspbian Jessie, so you just need to launch it by issuing:

node-red-start

..Cross fingers, after it launches, you can go to the raspberry_ip_address:1880

Of course, we will have it starting up every time our Raspberry Pi3 boots up

sudo systemctl enable nodered.service

So as you can see in the screenshot, I am using a MQTT input to know the status of the pins on the XBee devices, and taking some decisions with that info, even sending me a tweet @Teachingbudha every time the WATERPUMP changes state.

So finally, there is a system that prevent that sloooow water pumping system to overflow and waste precious water. It's been working good for three months so far.