Set Up an ESP8266 Automatic Update Server

Many people are now using the ESP8266 in its many guises (ESP-01S, Wemos D1, NodeMCU, Sonoff etc) for home automation systems. If you write your own code (as I do) updating each of these separately even via OTA (over the air) becomes a bit tedious.

My own system, for example has 8x ESP-01S, 6x Wemos D1, 4x Sonoff Basic 12x Sonoff S20, 2x Sonoff SV and a NodeMCU which share a common code base, so that's 33 devices in all to update when I make a simple code change.

But there is an easier way: An "update server". The excellent Arduino IDE + ESP8266 core has a library to do most of the work (ESP8266httpUpdate), but you need to know how to set up your own server to make it work.

This Instructable shows you how using a NODE-RED server, but the same logic applies to any server technology of your choice e.g. Apache + PHP etc

Step 1: What You Need

Arduino IDE
ESP8266 core
Any ESP8266 dev board with 1M or more flash RAM
A Web Server (even a humble raspberry Pi will do - It's what I use)
(optional) mkspiffs tool if you want to auto-update a SPIFFS file system image

Step 2: Create a Repository to Hold Binary Firmwares

On my server, I have a folder called /home/pi/trucFirmware which holds the various device firmwares and SPIFFS images

I maintain a separate binary for each hardware type (from a single source file with a few #defines) and when a new release is ready I use the Arduino IDE "sketch/Export compiled Binary" menu command for each target device.Note that even though there are 5 different hardware types, there are only two SPIFFS binaries: a 1M and a 4M version - constructed with the mkspiffs tool - since all the devices have either 1M or 4M flash.

Step 3: Create the Binaries

Using the Arduino IDE menu option sketch/Export Compiled Binary, create the firmware that will be uploaded to the device when it requests it from the update server.

If you need a SPIFFS binary you will need to install the mkspiffs tool.

Once you have it, building the SPIFFS binary is simple. I have a one-line batch file for the 1M version which takes the version number as a parameter (%1)

mkspiffs -c data/ spiffs_%1_1M.bin

and another for the 4M version:

mkspiffs -p 256 -b 8192 -s 0x0FB000 -c data/ spiffs_%1_4M.bin

I then copy all the compiled binaries and the SPIFFS .binary files over to the repository

Step 4: Create the Server Flow

I'm using NODE-RED, but the simple logic will be the same on any server technology / language.

a) Define a url which will listen for the ESP8266httpUpdate request. My raspberryPi serevr is on 192.168.1.4 and listens on port 1880 for /update with the hardware type appended. So if I'm going to request a binary for a Wemos D1 Mini, the url ends up as:

http://192.168.1.4:1880/update/d1_mini

b) Create code to handle the following logic:

ESP8266: "Hi, I'm running firmware version a.b.c, do you have a newer version?"
Server: "Let me see...ah yes I have a.b.d - here it comes..."

If a newer version does exists the server just sends it as a load of binary data in the http reply. The ESP8266httpUpdate class does the tricky part of copying the binary into memory, changing the firmware boot address to the new code than (if requested) rebooting the device to run the new code.

If on the other hand there is no higher version, it replies with a http 304 error which effectively says: "I have nothing for you" and your code continues to run as normal.

Step 5: Add the Server Logic

The first node in the flow "listens" for an http request to url http://192.168.1.4:1880/update with the device type appended. It passes this to "Construct search path" function node which has the following javascript code:

msg.type=msg.req.params.type;
var h=msg.req.headers;
msg.version=h["x-esp8266-version"];
msg.mode=h["x-esp8266-mode"];
if(msg.mode=="sketch"){
    msg.payload="/home/pi/trucFirmware/*.ino."+msg.type+".bin";
}
else {
    var sz=h['x-esp8266-chip-size'];
    msg.payload="/home/pi/trucFirmware/spiffs_*_"+(sz/1048576)+"M.bin";
}
return msg;

This just sets up the appropriate path with wildcard for the sys function which follows, which simply runs

ls - r

The output is then fed to the "Compare versions" function node:

var f=msg.payload.split("\n")[0];
msg.filename=f;
if(msg.mode=="sketch"){
    f=f.replace("/home/pi/trucFirmware/truc_","");
    f=f.replace(".ino."+msg.type+".bin","");
}
else {
    f=f.replace("/home/pi/trucFirmware/spiffs_","");
    f=f.replace(/_\dM\.bin/,"");
}
if(msg.version < f){
    node.warn("upgrade required");
    node.warn("will return "+msg.filename);
    return msg;
}
node.warn("no upgrade");
msg.statusCode=304;
msg.payload=[];
return msg;

The switch node then ensures that either the 304 "no update needed" message is sent or the actual new binary is returned and sent back to the device.

Step 6: Add Code to the Sketch to Request an Update

The sketch needs to have the following code included in it so that it will update automatically the next time you increase the version number:

#include<esp8266httpupdate.h>
...
#define TRUC_VERSION "0_4_99"
#define SPIFFS_VERSION "0_5_0"

// THIS_DEVICE is set earlier depending on various compile-time defines 
// which eventually define the hw type, e.g. #define THIS_DEVICE "d1_mini"
const char * updateUrl="http://192.168.1.4:1880/update/"THIS_DEVICE;
// this is my raspberry Pi server, the 1880 is the default NODE-RED port
// /update is the url I chose for the server to "listen" for, followed by the device type
...
bool actualUpdate(bool sketch=false){
    String msg;
    t_httpUpdate_return ret;
     
    ESPhttpUpdate.rebootOnUpdate(false);
    if(sketch){
      ret=ESPhttpUpdate.update(updateUrl,TRUC_VERSION);     // **************** This is the line that "does the business"   
    }
    else {
      ret=ESPhttpUpdate.updateSpiffs(updateUrl,SPIFFS_VERSION);
    }
    if(ret!=HTTP_UPDATE_NO_UPDATES){
      if(ret==HTTP_UPDATE_OK){
        Serial.printf("UPDATE SUCCEEDED");
        return true;
        }
      else {
        if(ret==HTTP_UPDATE_FAILED){
         Serial.printf("Upgrade Failed");
          }
        }
      }
  return false;
}

Step 7: Finally, Initiate the Update

At boot time, or perhaps in response to an MQTT message (as I do) run the following code:

if(_actualUpdate(true)) ESP.restart();
// or for SPIFFS...
if(_actualUpdate(false)) ESP.restart();

The device will update itself and reboot running the latest code from the server. It's a lot simpler for me than manually updating 33 devices!

Lots more useful information about Home Automation, IOT and programming the ESP8266 can be found on My Blog

Introduction: Set Up an ESP8266 Automatic Update Server