Over-the-air firmware updates for IoT devices

An over-the-air (OTA) update installs new firmware on a device through its radio network. A meter or sensor stays in service for ten years or more, and in that time faults are likely to be found in its radio stack or its application code. On an estate of 400 devices across 20 sites, a fix applied by hand means 20 site visits. The same fix over the air is a planned campaign run from the gateway.

This guide is part of the networks and architecture path.

Why field devices need updates

The best-documented Zigbee case is the attack that Ronen, O'Flynn, Shamir and Weingarten published in 2016 against Philips Hue lamps. The lamps authenticated firmware with one AES-CCM key shared by every lamp of the same type. The researchers extracted that key by power analysis, with equipment costing a few hundred dollars. With the key, they could build firmware that every lamp of that type accepted as genuine, and a bug in the lamps' Zigbee Light Link code let the malicious image spread from lamp to lamp. Philips fixed the takeover bug and made the fix available as an OTA update. The key extraction showed why an image must carry a signature that the device checks with a public key.

Bugs matter as much as attacks. A fault that corrupts a pulse counter or makes a device leave the network after a parent router restarts must be fixed on every affected unit. NIST SP 800-82 Rev. 3, section 6.2.11, asks OT operators for a documented patch management process: how to learn of patches, how to test them, when to apply them, and which compensating controls to use while a patch is delayed. It also says to deploy patches during planned outages. The campaign steps below follow that pattern.

How Zigbee does it

Zigbee devices use the OTA Upgrade cluster, cluster ID 0x0019, defined in chapter 11 of the Zigbee Cluster Library (ZCL). The gateway is the OTA server and holds the image files. The device is the OTA client and drives the download.

Every OTA file starts with the identifier 0x0BEEF11E and a header that names the manufacturer code, the image type, the file version and the total image size. It can also give a minimum and maximum hardware version. The server matches the manufacturer code and image type against the device, so a file built for one product is never offered to another.

The exchange runs in this order:

1. The server can send Image Notify to tell a router that an image is available. Sleepy end devices do not hear it, so every client also polls with Query Next Image Request, giving its manufacturer code, image type and current file version.
2. The server answers with Query Next Image Response: the file version it offers and the image size.
3. The client sends Image Block Request with the file offset it needs and the largest block it can take. The server returns that data in Image Block Response. The client repeats this until it has the whole file. The client, not the server, tracks progress, so the server keeps little state per device.
4. The client checks the image and sends Upgrade End Request with a status of SUCCESS or INVALID_IMAGE.
5. The server answers with Upgrade End Response, which carries an upgrade time. The client switches to the new image at that time. A value of 0xFFFFFFFF tells it to wait for a later command, which lets the server switch a group of devices together.

The device keeps working while it downloads. The ZCL requires an application bootloader with enough memory to store the new image, so the running image is not overwritten until the download is complete. The interruption comes at the end, when the device restarts into the new image.

Zigbee update time

The block size makes Zigbee updates slow. The client sets a maximum data size in each request. The ZCL lets the server send less than that to allow for routing overhead when the client is several hops away, and common server implementations send 50 bytes of image data per block. A 200 kB image is then 4,000 blocks, and each block is a request and a response that cross every hop. If a round trip through two hops takes a quarter of a second, the transfer takes about 17 minutes. That agrees with the 10 to 20 minutes that Edge shows before an update.

Two things make it longer. The server can rate-limit a client through the MinimumBlockPeriod attribute, a minimum delay in milliseconds between block requests. The ZCL gives the example of limiting every client to one block every 500 ms while several download at once, which doubles the 17 minutes. And a sleepy end device only receives data when it polls its parent. The ZCL lets it request blocks more slowly than the server allows, to save its battery, so a battery sensor can take hours.

Image signing

The ZCL strongly encourages a signature made with the manufacturer's private key, which the device checks with the matching public key when the download is complete (clause 11.3.3.1). It does not require one. Each application standard sets its own minimum. Smart Energy can use image signatures together with network and APS encryption, and other standards may use network encryption only (clause 11.3.2). Without a signature, a hash of the image (clause 11.3.3.2) shows that the file arrived intact, but not who built it. Some manufacturers add their own check in the bootloader. Ask the manufacturer which method the device uses. A symmetric key is only as safe as the least protected device that holds it, as the lamp attack showed.

How LoRaWAN does it

LoRaWAN downlinks are small and scarce, so the LoRa Alliance built firmware update over the air (FUOTA) as a set of application-layer packages. The FUOTA process summary, TR002, puts them together:

• TS003, application layer clock synchronisation, gives every device the network's time, so that a multicast session can start at an agreed moment.
• TS005, remote multicast setup, gives devices a shared multicast address and keys, and schedules a Class C or Class B session. A device that only runs Class A cannot take part in a multicast session. It can receive fragments only by unicast, in the receive windows after its own uplinks.
• TS004, fragmented data block transport, splits the image into fragments and adds forward error correction. With 10% redundancy, a device can lose roughly 10% of the frames and still rebuild the file without asking for the missing ones. One session can carry at most 16,383 fragments.
• TS006, the firmware management protocol, reports and controls the firmware version on the device.

When a device has enough fragments, it rebuilds the image, checks its digital signature with the update server's public key, and checks that the header matches its hardware and current firmware. It marks the image ready and reboots. The bootloader then installs it, either into a second image area or page by page with its progress stored, so that a power cut during the write does not leave the device without firmware.

LoRaWAN update time

Take a 100 kB image sent by Class C multicast in EU868, on the default RX2 frequency of 869.525 MHz at DR0 (SF12, 125 kHz):

• RP002 limits the application payload at DR0 to 51 bytes. The TS004 DataFragment command uses 3 of them, which leaves 48 bytes of image per frame.
• 100 kB is 2,084 fragments. With 10% redundancy, the server sends about 2,300 frames.
• Each frame is about 2.8 s of airtime at SF12.
• ERC Recommendation 70-03 limits the 869.40 to 869.65 MHz band to a 10% duty cycle. The gateway can transmit for 360 s an hour, which is about 129 frames an hour.

The session needs about 18 hours of airtime, and the gateway shares that 10% with every other downlink it sends on that channel. At DR5 (SF7), 239 bytes fit in each frame, and the same image needs about 460 frames of 0.39 s: about 30 minutes under the same limit. But every device in the group must receive SF7 reliably, and devices at the edge of coverage often do not. Unicast to a Class A device is slower still. At about one fragment for each uplink, a device that reports every 15 minutes needs more than 3 weeks for the same 2,300 frames.

Support varies by model. Check which FUOTA packages the device implements, whether it can open a Class B or Class C session, and whether it has the flash space to hold a second image.

Plan an update campaign

1. List each device with its model, hardware revision, manufacturer code, image type and current file version.
2. Read the release notes. Check whether the update changes a value, a unit, a scaling factor or a register that the system reads. A changed scale factor gives readings that look valid and are wrong.
3. Ask the manufacturer whether the image is signed, whether the device accepts an older file version, and whether its bootloader keeps the previous image.
4. Update a pilot group first. Include the oldest hardware revision, a device at the far edge of the mesh, and a battery device if the site has any.
5. Update the rest in small groups, for example five devices at a time on one Gateway, with a delay between them. Watch the devices that are not being updated. Late or missing reports from them mean the mesh is saturated, so reduce the group size.
6. After each device finishes, read the version back from the device. On Zigbee this is the CurrentFileVersion attribute of the OTA cluster, or the version reported through the Basic cluster. Then check that the device reports on schedule, and compare its readings with the readings from before the update.

When an update fails

The ZCL has no rollback command. It allows the server to offer a file version lower than the one installed, but the device's firmware decides whether to accept it. The recovery method is left to the manufacturer (clause 11.18). The examples in the specification are a bootloader that swaps the new image with the previous one, and a button press at power-up that returns to the previous image. If the device has neither, a bad image means a site visit.

The gap in data comes at the restart, not during the download. A cumulative counter, such as an energy register, recovers the total after the gap. An instantaneous value, such as power or temperature, does not. The stale data guide explains how to show the gap.

Device firmware and gateway software are separate

The gateway is the OTA server. It holds the images and answers every block request, so a gateway software update that restarts it during a campaign stops every transfer in progress. Finish or pause a device campaign before you update the gateway. Start the next campaign only when the updated gateway is stable and every device reports again.

Firmware updates with Edge

Edge on the ZGW-20 Gateway keeps a catalogue of firmware images on the Gateway. It takes three file types, up to 5 MB each: Zigbee OTA files (.ota) for third-party devices, EBL images (.ebl) for EpiSensor devices, and binary images (.bin) for the IO Board. Edge validates each file when it is uploaded and lists it with its manufacturer, model, version, size and validity.

An update can start at once or at a scheduled time. It can target one device or a selection of devices. For a selection, Edge sends the command to each device in turn with a delay of up to 60 s between them, so the transfers do not all start together. Edge warns that an update can take 10 to 20 minutes per device and that the device may restart, and it shows each update as running, completed or failed. Edge itself is updated through its snap channel or the desktop app, not from this catalogue.