OTA Operator Guide

This is the operator procedure for Hanasu firmware updates: keys, release builds, signing, the first cable install, and day-to-day pushes over the mesh. For the design behind it (package format, checks, trial and rollback, threats), read Over-the-Air Updates first. Every command here matches the tools in the Hanasu repo: tools/keys/genkeys.sh, tools/mnpkg.py, tools/mcuboot_sign.py and tools/ota_push.py.

At a glance link

Three machines are involved, and only public keys and signed files cross between them:

There are two signatures:

Key setup link

Generate keys once, on the signing machine, never on a development box. genkeys.sh is self-contained shell and needs only openssl; copy it over.

  sh genkeys.sh release-keys/                  # Ed25519 OTA keys (OpenSSL >= 1.1.1)
sh genkeys.sh --ota-p256 release-keys/       # P-256 OTA keys (works with LibreSSL)
  

Ed25519 needs OpenSSL 1.1.1 or later. The stock macOS openssl is LibreSSL 3.3, which has no Ed25519; the script detects that and stops. Either install OpenSSL 3:

  brew install openssl@3
OPENSSL=$(brew --prefix openssl@3)/bin/openssl sh genkeys.sh release-keys/
  

or use --ota-p256. P-256 package signatures (sig_alg 1) are fully supported; Ed25519 is only preferred. The MCUboot key is always P-256.

The script writes three key pairs:

Private keys are PKCS#8 PEM, mode 600, and are never overwritten: the script refuses to run if any of them exists. It prints each OTA key’s key_id, the same value mnpkg.py and the nodes compute.

Back up the private files offline. Hand over only the three *.pub.pem files.

The Python tools can do the same if preferred:

  mnpkg.py keygen --alg ed25519 -o ota_release.pem --pub-out ota_release.pub.pem
imgtool keygen -t ecdsa-p256 -k mcuboot_release.pem
imgtool getpub -k mcuboot_release.pem -e pem > mcuboot_release.pub.pem
  

Build a release link

On the build machine, wire in the public halves:

  # OTA key store for both targets; refuses DEV keys
tools/mnpkg.py c-keys --release ota_release.pub.pem ota_release_next.pub.pem \
    > firmware-idf/components/magnet/include/magnet_pkg_keys_release.h
cp mcuboot_release.pub.pem tools/keys/        # public half only
  

ESP32-C6 link

Add -DMN_PKG_RELEASE_KEYS=1 to the OTA env’s build_flags (the OTA env today is esp32c6_xiao_ble_led_ota). Without the release header the build stops with #error, and a header that holds DEV keys stops it too. The env’s MN_FW_VER_MAJOR/MINOR/REV/BUILD flags are the version the image reports, and they must match the --version you give mnpkg.py.

XIAO MG24 link

Add release.conf (it sets CONFIG_MN_PKG_RELEASE_KEYS; a plain -D does not reach the app image under sysbuild), point MCUboot at the public key, and turn on offline signing:

  west build -p -b xiao_mg24 --sysbuild firmware-zephyr -d build/rel -- \
    -DEXTRA_CONF_FILE="ble.conf;ble-resident.conf;release.conf" \
    -DSB_CONFIG_BOOT_SIGNATURE_KEY_FILE=\"$PWD/tools/keys/mcuboot_release.pub.pem\" \
    -DMN_SIGN_OFFLINE=1
  

MCUboot is built with the public key embedded, and MN_SIGN_OFFLINE leaves the application unsigned instead of letting sysbuild sign it with that same file. The image version comes from firmware-zephyr/VERSION.

Then package the unsigned image for the signing machine:

  tools/mcuboot_sign.py job build/rel -o job/
  

job copies zephyr.bin and zephyr.hex and records the exact imgtool arguments the build would have used, plus a SHA-256 of each file. Carry job/ to the signing machine.

Sign a release link

The signing machine needs Python 3 with the cryptography package, imgtool (pip install imgtool), and copies of mnpkg.py and mcuboot_sign.py.

MG24: sign the MCUboot image, then wrap it in a package.

  python3 mcuboot_sign.py sign job/ --key mcuboot_release.pem -o signed/
python3 mnpkg.py build signed/zephyr.signed.bin --chip efr32mg24 --board xiao-mg24 \
    --variant <name> --version <X.Y.Z+N> --fw-version <tag> \
    --key ota_release.pem -o node.mnpkg
  

sign refuses a job whose files changed since job made it, signs both .bin and .hex, and runs imgtool verify against the key. mnpkg.py build refuses an MCUboot image whose embedded version differs from --version.

C6: package the IDF firmware.bin directly.

  python3 mnpkg.py build firmware.bin --chip esp32c6 --board xiao-esp32c6 \
    --variant <name> --version <X.Y.Z+N> --fw-version <tag> \
    --key ota_release.pem -o node.mnpkg
  

The variant must not start with dev for a release; mnpkg.py refuses a DEV key with any other variant. Optional flags: --min-running <version>, --clears-bundles, --requires-repartition, and --build-time <unix-seconds> for reproducible builds.

Before shipping, inspect and verify the package offline:

  python3 mnpkg.py inspect node.mnpkg                           # decode the header, check structure
python3 mnpkg.py verify node.mnpkg --pubkey ota_release.pub.pem   # checks 1-8 offline
  

First install by cable link

Every node needs one cable install of an OTA-capable release build. After that, updates go over the air.

ESP32-C6: flash the _ota env over USB. The OTA partition table keeps NVS at the same offset and size, so the node’s identity, name, channel and BLE bonds survive.

  cd firmware-idf && pio run -e esp32c6_xiao_ble_led_ota -t upload
  

MG24: copy the signed image into the build directory, then flash MCUboot and the signed app over SWD.

  tools/mcuboot_sign.py install signed/ build/rel
west flash -d build/rel
  

Stock OpenOCD has no MG24 support; pass the Silicon Labs Arduino core’s OpenOCD to west flash as described in XIAO MG24 on Zephyr.

Then check the node on its host link:

On a bench with several boards, pick serial ports by USB serial number, not by port name: ports renumber when re-plugged, and different Espressif chips enumerate with the same product name.

Admins link

A remote apply must be signed by a key the target allow-lists. The admin key is the sending node’s identity key.

All ADMIN and OTA verbs are privileged: they work over serial, or over BLE only on a bonded link (otherwise E_NOT_BONDED). A fingerprint’s first 4 bytes are the key owner’s device ID, which makes ADMIN LIST easy to read.

Push an update link

From the host attached to the sending node:

  python3 tools/ota_push.py <sender-port> <target-ml-eid> node.mnpkg --apply
  

The sender can be any Hanasu node on a host link; the target must allow-list the sender’s key. Options:

What you see:

  # package v0.7.0+2  chip 0x0002 board 0x0201  sha128 <32 hex>
# <bytes> B, <n> chunks, meta mnpkg:1
# sender: !XFER_SENT ... in <s>s
# apply: !OTA_RESULT <id> accepted v0.7.0+1
# report: !OTA_REPORT <id> confirmed v0.7.0+2 (was v0.7.0+1) bundles=1
  

!XFER_SENT means the target ran all nine package checks and staged the image. !OTA_RESULT … accepted means it armed the slot and is rebooting. The !OTA_REPORT arrives after the trial: confirmed, or rolled-back v<old> (failed v<new>). The tool waits up to 12 minutes for the report and exits non-zero unless it is confirmed.

Timing: about 90 ms per 336-byte chunk, set by the node and radio round trip. A 557 KB MG24 package took 137–169 s (about 3 minutes); a 1.17 MB C6 package took 385 s (about 6.5 minutes).

Update one node at a time: start with a canary and move to the next only after its confirmed report. Never apply to many nodes at once.

Package refusals link

A refused package reads !XFER_FAIL <xid> refused:<n>, and ota_push.py adds the name. Checks 1–6 refuse at chunk 0, before the payload is sent.

Apply results link

No reply at all means the target dropped the command: the sender’s key is not on the target’s allow-list, or the frame was lost. Check ADMIN LIST on the target.

Local apply link

Without --apply, apply on the target itself from a privileged link: FORTH, then ota-status to see what is staged, then ota-apply. A local apply always boots in TEST mode, and it sends no report because no admin asked for it.

Health and rollback link

After an apply the new image boots on a 10-minute trial. It is healthy, and confirms itself, once:

1. the node is READY (attached to Thread), and
2. every saved Forth role bundle re-applied at boot without error.

If not, the node reboots and the bootloader restores the previous image, which then reports rolled-back.

The report’s bundles= value:

A package built with --clears-bundles is for firmware that breaks the Forth vocabulary on purpose. During the trial the node holds its saved bundles back, so health is READY alone. On confirm, it forgets them and you redeploy bundles for the new vocabulary. On rollback they are untouched and the old image re-applies them.

--perm (PERMANENT) skips the trial entirely: the image is confirmed at its first boot and never rolls back on its own. Use it only when you mean it.

Rotation and compromise link

Rotating the OTA key link

The firmware’s store holds two OTA keys, current and next. To retire current:

1. On the signing machine, create a new spare. genkeys.sh will not overwrite existing files, so use openssl directly:

     openssl genpkey -algorithm ed25519 -out ota_release_new.pem && chmod 600 ota_release_new.pem
   openssl pkey -in ota_release_new.pem -pubout -out ota_release_new.pub.pem
   # P-256 instead:
   # openssl ecparam -name prime256v1 -genkey -noout | openssl pkcs8 -topk8 -nocrypt -out ota_release_new.pem
     

   Its key_id (use tail -c 65 for a P-256 key):

     openssl pkey -pubin -in ota_release_new.pub.pem -outform DER | tail -c 32 |
       openssl dgst -sha256 -binary | od -An -tx1 | tr -d ' \n' | cut -c1-16
     

2. On the build machine, generate a store of next plus the new spare and build a release:

     tools/mnpkg.py c-keys --release ota_release_next.pub.pem ota_release_new.pub.pem \
       > firmware-idf/components/magnet/include/magnet_pkg_keys_release.h
     

3. Sign that release’s package with ota_release_next.pem, which the fleet already trusts, and roll it out node by node.

4. Once every node runs it, destroy the old current private key. next is now current and the new spare is next.

The MCUboot key has no second slot: rotating it means reflashing the bootloader over SWD.

If a key leaks link

Troubleshooting link