Dynamically Updating Firmware from Arbitrary Sources on an ESP32

• Download source code - 250.8 KB

Introduction

I recently needed to provide firmware update capabilities to an ESP32 but rather than get it from WiFi, my ESP32 received it over serial UART from another connected device, which itself fetched the firmware package from an MQTT server. Woo! OTA out of the box won't work for you there, but you can use the OTA API to update from your own data stream.

Prerequisites

• You'll need an ESP32. This is coded for the base ESP32, but you can use it with something like an ESP32-S3 by changing platformio.ini.
• You'll need VS Code and Platform IO installed.
• To rebake the binary, you'll need a C++ compiler and CMake installed locally so you can build the zipsx binary (I use VS Code CMake extensions and Visual Studio's C++ compiler). This isn't strictly required for the demo but is if you want to compress your firmware in the real world, or if you want to change the firmware provided with the demo.
• You'll need to be able to create zip files locally if you want to rebake the binary.

Preparing the Update Binary

This involves several steps:

1. Switch the build configuration in Platform IO to the revision B entry and build it.
2. Under the .pio build folder, find your firmware.bin for revision B.
3. Zip the firmware.bin to create firmware_rev_b.zip. It must be the only file in the zip archive.
4. Run zipsx over the zip to create firmware_rev_b.stream. This is the compressed binary data extracted from the zip file. Zip files cannot be streamed because their directory entries are at the end of the file. Therefore, we don't use a zip file, but rather, just the embedded compressed stream for the firmware file.
5. For the demo - not the real world - use my online header generator tool with the output type set to C/C++ in order to produce the firmware_rev_b.h and copy it under the include folder in the main project.

Using the Code

Note: Since this uses OTA, your ESP32 must have two app partitions of identical size in order to facilitate updating. Fortunately, this is the default configuration for ESP32s so you shouldn't have to alter your flash partitions.

Use the revision A configuration to build the main firmware w/ the updater in it. All the code in setup() facilitates updating, so you can adapt that code to your projects.

#include <Arduino.h>

// build the other project, and then compress
// and convert its firmware.bin to a header
#define FIRMWARE_REV_B_IMPLEMENTATION
#include <firmware_rev_b.h>
#include <htcw_zip.hpp>
#include "esp_ota_ops.h"
using namespace zip;
using namespace io;

esp_ota_handle_t handle = 0;
uint8_t write_buffer[8192];
size_t write_size = sizeof(write_buffer);
uint8_t* write_ptr = write_buffer;
void setup() {
  Serial.begin(115200);
  Serial.println("Hello from revision A");
  Serial.print("Unpacking and updating firmware (");
  Serial.print(sizeof(firmware_rev_b));
  Serial.println(" bytes)");
  io::const_buffer_stream in(firmware_rev_b,sizeof(firmware_rev_b));
  archive arch;
  esp_ota_begin(esp_ota_get_next_update_partition(NULL), OTA_SIZE_UNKNOWN, &handle);
  
  zip_result r=inflate(&in,[](const uint8_t* buffer,size_t size, void* state){
    if(size>write_size) {
      size_t sz = sizeof(write_buffer)-write_size;
      if(ESP_OK!=esp_ota_write(handle,write_buffer,sz)) {
        Serial.println("OTA write error");
        return (size_t)0;
      } else {
        Serial.print("OTA wrote ");
        Serial.print(sz);
        Serial.println(" bytes");
      }
      write_size = sizeof(write_buffer);
      write_ptr = write_buffer;
    }
    memcpy(write_ptr,buffer,size);
    write_ptr+=size;
    write_size-=size;
    return size;
  },NULL,sizeof(firmware_rev_b));
  if(zip_result::success==r) {
    if(write_size<sizeof(write_buffer)) {
      size_t sz = sizeof(write_buffer)-write_size;
      if(ESP_OK!=esp_ota_write(handle,write_buffer,sz)) {
        Serial.println("OTA write error");
        while(1);
      } else {
        Serial.print("OTA wrote ");
        Serial.print(sz);
        Serial.println(" bytes");
      }
    }
    if (ESP_OK == esp_ota_set_boot_partition(esp_ota_get_next_update_partition(NULL))) {
      Serial.println("Updated. Restarting");
      esp_restart();
    } else {
      Serial.println("OTA unable to set boot partition");
      Serial.println("Update error");
    }
  } else {
    if(r!=zip_result::success) {
      Serial.print("Unpacking error: ");
      Serial.println((int)r);
    }
    Serial.println("Update error");
  }
}

void loop() {

}

The first thing we do is set up the includes. Most of it is boilerplate except for the OTA support, the firmware and the zip support. You'll note #define FIRMWARE_REV_B_IMPLEMENTATION. This provides the actual array data, and needs to be specified before the associated include or you'll get linking errors.

The htcw_zip.hpp file is zip extraction and Huffman decompression support. It is necessary in order to "inflate" (decompress) the "deflated" (compressed) firmware stream.

The esp_ota_ops.h is provided by the ESP-IDF and facilitates actually calling the OTA API.

After that, a couple of namespace imports - one for our stream support (we don't use the STL) and one for our zip/decompression support.

Next is a "handle" for the OTA update. Unlike most handles, it's not a pointer, but an integer so code it accordingly, for example by initializing it to 0 instead of NULL.

We use that handle to begin the OTA update on the next available OTA partition.

Next, we have code to facilitate a write buffer. The reason we need to buffer writes is because the Huffman decompression tends to write out data 1 byte at a time, and it's prohibitively slow to write the flash during the OTA update one byte at a time. Instead, we gather 8KB at a time from the decompression routine, and write that.

In setup(), we wrap our array in the firmware header file with a const_buffer_stream which gives us a cursor over the array. In the real world, you'd probably use file_stream to get it from something like an SD card, or arduino_stream to read from the serial port, or HTTP or something (the latter at least under Arduino.)

An anonymous function is declared to handle the inflate() callback wherein we fill the write_buffer and then write it out every time it gets full. The inflate() function will call its own callback every time it has produced more decompressed data, which is what we handle.

After that, on success, we write out the final remaining partial block, if there is one, or otherwise report an error. Once the final block is written, if everything is copacetic, we change the boot partition to the one we just wrote.

Finally, if all that works, we simply reboot.

If any of the above failed, we report errors.

Points of Interest

I originally tried to do this without extracting a stream from the zip, and just using the zip as is, even though if I would have thought about it I would have realized it wasn't possible because the central directory information is at the end of the file. That was quite a bit of time wasted. Silly me.

History

• 24^th March, 2024 - Initial submission