Setting Up an MQTT Broker. Part 2 : IoT, Home Automation

Foreword

Okay In 'Part 1 : IoT, Home Automation', you've set up and configured your 'Vanilla' Raspberry Pi (pardon the pun) and can connect to it remotely with PuTTY. Now we will set up a server which will establish your main household IoT infrastructure and test it works as expected.

Introduction

This Instructable documents how to set up an IoT MQTT Broker and verify it is working correctly by 'snooping' MQTT topics with MQTTSpy.

What Parts do I need?

A home network
Router with DCHP server
One Raspberry Pi B+ or better
One USB Hub, Keyboard, Mouse and LCD monitor to set up the Pi
One Win 7 PC

What skills do I need?

Top of the list :- Patience
Some understanding of TCP/IP
A very basic understanding of what client/server means
Small knowledge of Linux

IoT Devices

IoT devices rely upon an underlying protocol named 'Message Queuing Telemetry Transport' or 'MQTT' for short. This protocol is known as 'Publish/Subscribe' and is extremely simple and lightweight, being designed specifically for 'constrained devices' and low bandwidth systems. Which in normal language means a communications protocol so simple and small that even a microprocessor system like an ESP8266 or Arduino can support it. Once you couple this with a WiFi connection (ESP8266) or Ethernet link (Ethernet Shield) you have yourself the basis of an IoT device.

Publish/Subscribe

The second picture above is a generalised pictorial representation of a Publish/Subscribe protocol in action.

Here an IoT device or 'Client' has detected an unusually high temperature and 'Published' this 'Topic' to the MQTT 'Broker'. The MQTT Broker has then forwarded this information to all those 'Clients' who have chosen to 'Subscribe' to this news 'Topic'. In this case it could be an intelligent device which switches on an A/C unit or a passive data logging system or even an intelligent building management system which in its wisdom has chosen to open the windows.

Equally it could have been an engine management system publishing fuel consumption data or a smart meter publishing electricity consumption figures.

If this explanation is troubling you then think of it like YouTube. I upload a video to YouTube (the broker) and my many followers who have subscribed to my channel will get a notification of the new publication.

The Broker

This is the heart of the publish/subscribe protocol. The MQTT Broker is optimally designed to handle many thousands of concurrently connected MQTT clients.

It is primarily responsible for receiving all messages, filtering them, deciding who is interested in them and forwarding messages to all subscribed clients.

The Client

The MQTT Client, be it Subscriber or Publisher (or both in one device) is any device from small Microcontroller up to a fully fledged server, that has an MQTT library running and is connected to an MQTT Broker over any kind of network.

And that's all you need to know on MQTT. We won't be going into the protocol or how they plug in as there is simply no need at this point.

Series Links

To Part 1 : IoT - 'Intranet' of Things : Home Automation. Part 1 : IoT, Introduction

To Part 3 : Creating Your First IoT WiFi Device. Part 3 : IoT, Home Automation

Step 1: Installing Mosquitto MQTT Broker on Your Raspberry Pi

We will be using the Mosquitto MQTT Broker from Eclipse as its free and runs under Linux. It is possible to use a PC based MQTT Broker however I specifically wanted to use a Raspberry Pi as they are cheap, physically very small and pack a large punch where processing power is concerned, better still I have a load of them handy. I used a B+ Pi.

Preamble

I am assuming you have followed the steps in my earlier Instructable named 'IoT - Intranet of Things : Home Automation. Part 1 : IoT, Introduction' and have a Raspberry Pi already set up and ready to go.

All you need to do is change the host name from 'MYHOSTNAME' to 'MQTTSVR'. You can do this by using PuTTY to connect to the Pi and from the command line enter;

sudo raspi-config

Select (9) Advanced Options then (A2) Hostname.

Enter 'MQTTSVR' and save the changes. Exit then choose to reboot the Pi

You can now re-connect PuTTY to this Pi by using the new hostname 'MQTTSVR.local'. Create a PuTTY saved session named 'MQTTSvr', it will come in handy later.

Installation

The installation process is as follows;

Connect to your new MQTT server via PuTTY and remembering to copy from this Instructable and paste into PuTTY to avoid having to type any commands by hand.

Ensure you are in the home directory. If you are you will see the following prompt

pi@MQTTSVR:~ $

If you don't see this enter

cd ~

Now complete the following commands;

wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key

then;

sudo apt-key add mosquitto-repo.gpg.key

Now make this repository available to apt;

cd /etc/apt/sources.list.d/

As we are using the Raspberry Pi Jessie Debian image enter the following;

sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list

Now enter the proceeding commands, which may take some time to complete. Also answer Yes (Y) to any questions;

sudo apt-get update
sudo apt-get upgrade

To complete the install process enter the following, answering Yes (Y) to any questions;

sudo apt-get install mosquitto

Let the Pi do its thing and complete where you will see the command prompt.

pi@MQTTSVR:/etc/apt/sources.list.d $

That's it you're done. Mosquitto has now been installed as a service and will automatically start when you boot your Raspberry Pi.

Now we need to test it works.

Note : Before you remove power from your MQTT Broker enter the following command and wait for the Pi to shutdown.

sudo shutdown now

Hint

If you want to quickly connect to your Raspberry Pi server using PuTTY without having to type in username and password each time. Do the following, assuming you have already created a saved session named 'MQTTSvr' as in Step 7, 'Part 1 : IoT, Automation' and your MQTTSVR username and password are unchanged from the example given.

Locate your PuTTY install and do the following;

Create a short cut to it.
Right click on the short cut
Select the 'Shortcut' tab
In the text box named 'Target:' Enter the following after the text "...\putty.exe"
- -load MQTTSvr -l pi -pw raspberry
So in my case I installed to "C:\Program Files (x86)\PuTTY" the text will read;
- "C:\Program Files (x86)\PuTTY\putty.exe" -load MQTTSvr -l pi -pw raspberry
Now drop this short cut on your desktop and double click each time you want to open a PuTTY session.

Step 2: MQTT Topics

This is a good time to have a quick discussion regarding MQTT Topics.

What is a Topic?

A topic is a string made up of UTF-8 (Unicode Transformation Format 8-Bit) characters and is the data sent by a Publisher to a Broker to be forwarded on to any Subscribers. In short its a basic text string.

How are they constructed?

Topics are broken down into one or more topic level each with a minimum of at least 1 character. These levels are separated by a forward slash '/' and can include ' '(spaces).For example (not including the quote marks);

'this/is/a/six/level/topic example with spaces'

Though arguably not very useful topic, hopefully gets the point over. Topics are case sensitive so;

'test/topic'

is different to;

'Test/Topic'

This is a single topic

'x'

as are these;

' ' (space) and '/'

There is no special action necessary to create a topic, the act of subscription or publication does this for you.

And that's about it.

Is there a special format or best practice for constructing topics?

The short answer is 'no' and there are lots of views surrounding how you should make them up (and I will cover the topic in a later Instructable). However you would need to consider the following when creating your topics;

Keep them short. They are sent over the network and need to be parsed (read) by some device/broker/server, so the longer and more intricate you make them the harder they are to decode.

Where you can, use meaningful names. It makes debugging easier. As a human its a lot simpler to type, or read 'sensor1' than 'Wcp10X!3wdrka2'.

Make them have meaning to you. The examples I have seen tend to be like a kind of address such as;

'myhouse/first floor/bedroom one/ceiling/light'

This assumes a fixed location IoT device and for the most part this will be true, though this type of nomenclature fails when considering mobile or re-deployable IoT devices, such as a temperature humidity sensor, or switchable extension socket. Should you decide to move a switchable extension socket from the living room to the bedroom to replace a failed device, you will need to reprogram the IoT device to respond to its new location to be consistent.

For my series on IoT based home automation I will be using topics which refer to the device only such as;

'/EthernetDevice/Led1Command'

'/WiFiDevice/Temp1Status'

This is because I ultimately want to use a MySQL database to hold the configuration and IoT device information. Which I will hopefully cover later.

Oh, and one final thing, it is usually convention not to precede the first topic with a '/', but you can see I've ignored this. :-)

Are there any short cuts?

Yes, MQTT offers what is know as topic 'wild cards'. These are special characters a subscriber can place into a subscribing topic to reduce the need to subscribe to many items. There are two types of wildcard characters; '+' known as 'Single Level' and '#' for 'Multi Level'. The Picture above gives examples of both.

A word of caution. If you use the '#' multi level wild card and have a lot of IoT devices in your system, then be prepared for a lot of messages!

Are there any special topics?

Yes, topics beginning with '$' are reserved for internal statistics of the MQTT Broker. At the time of writing this Instructable there is no clear official standardisation of topics that must be published by a Broker. However it is common practice to prefix with '$SYS/' for this type of information. The mosquitto MQTT Broker conforms to this '$SYS/broker/#'. (see Linux mosquitto man page for full details. ie. man mosquitto)

So if you want to see all the mosquitto MQTT Broker statistics you would need to subscribe MQTTSpy to the following (see next steps for further details);

'$SYS/broker/#'

Step 3: Setting Up Your Test Environment

Preamble

To test our new MQTT Broker we will first install a client which can subscribe and publish to a given topic on the Raspberry Pi. We will then monitor this exchange via tool named MQTTSpy from the PC (just like in the first picture above).

Why, test the install? Well it could save you a lot of time in the long run. To correctly set up a robust home automation system you start to rely upon and to which in the fullness of time be adding lots of useful IoT clients to, you need to make sure that all the links in the chain are working correctly, that way if you observe an issue with your new IoT device you can quickly rule out as many systems as possible and zero in on the actual issue.

Installing Pub/Sub Clients (Raspberry Pi)

From a PuTTY window enter the following command on your MQTT Broker. Ensure you are in the home directory;

pi@MQTTSVR:~ $

if not enter;

cd ~

followed by;

sudo apt-get install mosquitto-clients

Installing MQTTSpy (PC)

In order to successfully run MQTTSpy you will need to have the Java Runtime Engine (JRE) greater than version 8 installed on your PC.

To check what version of the JRE you have installed, run up a 'dos in a box', ie a command line window (picture above).

From the Start button, enter into the search box with the grey text 'Search programs and files'

cmd

and hit enter. A dos in a box will appear into which you will need to type;

java -version

It will return 'java version "W.X.Y.ZZ"' and some other 'guff' (picture above) or an error. if you have anything other that greater than version 8 (actually 1.8.y.zz) you will need to follow these instructions;

Go to this Oracle downloads web page.

https://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html

Select the 'Accept Licence Agreement' radio button, download and install the 'Java SE Runtime Environment 8u92', jre-8u92-windows-x64.exe (as in the picture above).

Ok, now go to the MQTTSpy download page given below;

https://github.com/kamilfb/mqtt-spy/wiki/Downloads

Select and download mqtt-spy jar file (picture above). At the time of writing version 0.5.0 was the latest available.

Save your 'mqtt-spy-0.5.0-jar-with-dependancies.jar' file to any directory.

To get the software to run you can double click it. It this doesn't work you can enter the following from the command line, in the directory you saved the jar file.

java -jar mqtt-spy-0.5.0-jar-with-dependancies.jar

Alternatively you could unzip and drop the RunMQTTSpy.bat file (given below) into the same directory and double click this to run MQTTSpy each time.

Ok, you 've now installed all your test software, let's configure it and run a test.

Hint

To open a command prompt at a specific directory do the following;

Open windows explorer
Navigate to the directory of interest
Press and hold 'Alt' and 'Shift' keys
Right click on the directory of interest
From the dropdown menu select 'Open command window here'

RunMQTTSpy.zip
Download

Step 4: Configuring MQTTSpy and Testing Your MQTT Broker

Out of the box, the minimal configuration required by MQTTSpy is as follows;

Connecting to your MQTT Broker
Subscribe to a given topic
Optionally, publish to a given topic

Here the video above shows how to connect to our mosquitto MQTT Broker 'MQTTSVR' and add a wild card to view all test topics and test the MQTT Broker. The running order is as follows;

Open two auto login PuTTY sessions
Start MQTTSpy by double clicking the 'RunMQTTSpy.bat' file (I've done the video as if you have just used MQTTSpy for the first time, so you can see the initial configuration)
Create a standard configuration
Create a new MQTT Broker connection which we will connect to our Broker
Enter the Broker details 'MQTTSVR.local' and give it a client ID 'Win 71'
Then connect to the Broker 'Open connection' and save this new set up so we can use it later.
Select this Broker tab, which has gone 'GREEN' to show it has connected to MQTTSVR
Make a subscription to all test topics (with multi level wild card)
- '/test/#'
From a PuTTY session window make a subscription to the test topic (no wildcards).
- mosquitto_sub -t "/test/Topic"
From the other PuTTY session window publish a message to the test topic.
- mosquitto_pub -t "/test/Topic" -m "Hello World"
You will now see this Hello World message arrive at both our client subscribers (PuTTY and MQTTSpy)
Now to prove we can publish a message from a client not located on the MQTT Broker we copy the test topic to the publish box in MQTTSpy and send the message 'SteveQuinn' to the MQTT Broker
You can now see this message arrive both at the PuTTY subscribed client and the MQTTSpy client (here MQTTSpy has become both subscriber and publisher).
Finally we close the connection to the MQTT Broker from the control panel and quit MQTTSpy.
Followed by closing the two PuTTY sessions. Remember to use ^C (Ctrl C) to quit the subscription before quitting.

Ok, so now we are accomplished MQTT Broker testers. What does this give us?

If we are having any issues with our Home Automation set up, we can now quickly run up MQTTSpy and check to see if the Broker is healthy or any published messages are 'malformed' etc. We can even publish messages, meaning we can test any IoT device we create.

Step 5: There Are IOS MQTT Clients

Although in the examples I have given above I use PC based software to carryout my testing equally it is possible to use a smartphone app.

Pictured above is one such application named 'MQTTool' for the iOS. Although not as feature rich as MQTTSpy it is easy to use and allows you to do some basic testing.

Step 6: And Not to Leave Out Android Users

From the play store I could see quite a few free apps, I tried many out but found the app pictured above to be the best of them 'MQTT Dashboard'. Though you need to put in an IP Address not the host name 'MQTTSVR.local' it does have a better method to publish to topics, allowing you to create text boxes, switches, sliders etc.

I included a screen grab from the Pi detailing how to get the IP Address of the Pi. To do this, connect via putty and enter;

ifconfig

The Pi will respond with the screen shot above and I have highlighted the IP address you will need to enter into MQTT Dashboard. I have assumed you are connecting to your network via Ethernet. If you have connected via WiFi read 'wlan0' for 'eth0'

Step 7: So What's Next?

In the next three Instructables we will create three simple IoT devices for your Intranet which are based around the ESP8266-01 and the Arduino ATMega2560 with Ethernet Shield. The first of which we will revisit to improve the design and extend its use to show just how easy it is to make your own IoT devices.