SDK 0.29 on a box: every carrier on, sessions through a middle box, and a way to prove it

TL;DR

In SDK 0.29.0 the Python service starts every enabled carrier: LAN peer streams, Bluetooth LE and the internet relay. Devices that only reach each other through a third box can now start an encrypted session, and offline-protocol-verify shows what the network did.

A box on a site, such as a gateway, a controller or a small Linux computer in a cabinet, is useful to a mesh when it stays on, keeps its queue on disk and reaches its neighbours by whatever link works today. Version 0.29.0 makes the Python service do that: it starts every carrier its configuration enables, it can now start an encrypted session with a device it only reaches through another box, and a new command reports what the network actually did.

Machines on a site keep working together over the links they have, and traffic between them stays local instead of going out to a cloud and back.

Offline Protocol service transports on a box

In 0.29 the service starts every carrier its configuration enables: peer streams on the LAN, Bluetooth LE in both roles, and the internet relay. Until 0.29, offline-protocol-service started the peer stream and the gateway client and nothing else. A configuration with ble_enabled built both Bluetooth LE managers and never ran them, and there was no way to name a relay. Now the service starts:

• Peer streams on the LAN. TCP streams between boxes, on port 7878 in the SDK’s container image. --lan advertises the box and finds the others over DNS-SD (it needs the lan extra), so no --peer list is needed on one network.
• Bluetooth LE in both roles. The central and the peripheral, where the platform has a backend: BlueZ on Linux, so phones can find the box, and a box can reach another box over Bluetooth when the cable is gone.
• The internet relay, once --relay URL names one, with the token from an environment variable.

offline-protocol-service --config config.json \
    --mls-root /var/lib/op/keys --state-root /var/lib/op/state \
    --listen 0.0.0.0:7878 --lan \
    --relay wss://relay.example.com/ws --relay-token-env OFFLINE_PROTOCOL_RELAY_TOKEN

with "wifi_direct_enabled": true, "ble_enabled": true and "internet_enabled": true in config.json. A transport that fails to start is logged and left stopped while the others run, and the service fails only if none starts. A Bluetooth adapter held by another process no longer takes the LAN down with it. A flag the configuration cannot honour now stops the service at start with the field named: --relay without internet_enabled says --relay needs internet_enabled in the config instead of running a service that reaches nobody.

Several Bluetooth fixes make the Linux box a better peer. The Python peripheral now knows which central wrote each fragment on BlueZ, so it has a route back to a phone that connected to it. The two roles no longer split one message’s fragments across two links. A link that has gone dead is dropped within about 20 seconds. The phones post covers the phone side.

Encrypted sessions through a relay box

From 0.29, two devices that only reach each other through a third box can start an encrypted session. While a message waits for a peer no carrier reaches directly, the sender hands its key package to its neighbours to carry, and the Welcome comes back the same way. The box in the middle carries frames it cannot read.

With encryption required, every frame after the session handshake is sealed. Before 0.29 the two handshake frames, the key package and the Welcome, were the only ones that never crossed the mesh. So a message to a device the sender had never been next to, reachable only through a third device, waited in the pending queue for its whole lifetime. Related fixes close the gaps around it:

• A resend is offered to the mesh when no carrier reaches the recipient, not only the first send. A message whose first attempt went into a link that had already died used to be retried down direct links for its whole life.
• A message the mesh carried now settles with message_delivered when the recipient answers. It used to stay in the sender’s outbox until message_failed seven days later, even though it had arrived.
• A forwarder carries the sender’s own retransmission again. It used to suppress every later copy of an id for ten minutes, so a sender whose acknowledgement was lost while it was off retried into the box in between until the window lapsed.

The threat model records the new reach as R24: a session can now be started from anywhere the mesh reaches.

Verifying message delivery

The new offline-protocol-verify command drives the service on the box it runs on and waits for the engine’s own events, because saying a message was held and delivered is easy and showing it is harder:

offline-protocol-verify state                            # address, carriers, queues, sessions
offline-protocol-verify pair off1qxk... --timeout 60     # wait for a confirmed session
offline-protocol-verify send off1qy8... "held while you were off" --await --timeout 120

Each command prints JSON lines and exits 0 when what it waited for happened, 2 when its time ran out, and 1 when the engine gave the message up, so a script can assert on it. ping sends on a cadence and reports the carrier each message arrived over, and watch prints every event across restarts of the service.

examples/offline-first uses it to run four scenarios on three containers, where the middle one is the only way between the other two: store and forward, a sender killed and restarted while its message is queued, a message carried through the middle box to a device the sender has never met, and a message the middle box delivers while the sender is off. We ran it against the 0.29.0 package from PyPI on one laptop, with fresh identities, and all four passed, each in under a second. Its README is also the runbook for the legs that need real hardware.

Raspberry Pi hardware tests

Containers on one machine prove the logic, not the links, so the service was also tested on hardware in one lab setup. Before the release, the SDK team ran the same scenarios and the HTTP front on two Raspberry Pi 4s, each plugged into a laptop over USB networking. The Pis had no route to each other, so the laptop was the only path between them. An Android phone running the demo app joined over Bluetooth LE. In that test:

Check	Result
First session on fresh identities	2.8 seconds; 1 to 6 seconds over three rounds
HTTP call from one Pi to the other, through the laptop	Passed, 5.2 seconds
HTTP call between the two Pis over Bluetooth LE alone	Passed
Store and forward; sender restarted; through the middle box	Passed
Carried by the middle box while the sender was off	Passed, 3 of 3 runs
One encrypted group of four: both Pis, the laptop and the phone	12 of 12 messages delivered, every device to every other

Two of these were failing before fixes that went into the release. The first session between a Pi and the laptop took 14 to 17 seconds, and once more than 90, because a peer that connected while the Pi was still starting its LAN discovery got no key package until an application sent something. And the carried-while-off scenario left the sender’s queue undrained for over ten minutes, because of the forwarder suppression described above. These are reported results from one lab setup, not guarantees for yours. Bluetooth LE from inside a container, host networking between two hosts, and the relay leg of the runbook have not been run on hardware yet.

Upgrading a box to SDK 0.29

• --listen, --peer and --lan now require wifi_direct_enabled, and --relay requires internet_enabled and a ws:// or wss:// URL. Each used to be ignored or accepted silently.
• Each transport has 30 seconds to start before it counts as failed and is stopped.
• A device with allowRelay on now also carries the handshake frames of sessions it is not part of. Turning it off declines them too.

The rest is in UPGRADING §27 and the 0.29.0 changelog. The HTTP front post covers calling services between boxes with plain HTTP, and the phones post covers Android and iPhone.

Get the package from PyPI, read the source on GitHub, or talk to us about a box per site.