Delivery and store-and-forward

How does store-and-forward messaging work?

Store-and-forward messaging works by having each device or server that receives a message write it to storage, hold it while there is no onward path, and send it to the next hop when one appears. Each holder keeps its copy until the next one confirms it has taken responsibility, or until the message expires.

TL;DR

Store-and-forward moves a message without a complete path, provided no holder deletes its copy before the next confirms it. Holding times range from router queues to days for email, so each system needs a limit, deduplication, and plans for storage pressure and lost ordering. Status labels should name the outcome they show, from stored locally to accepted by the recipient.

Learning objectives

After reading this article you will be able to:

  • Describe the accept, store, wait and hand-over cycle each store-and-forward node runs
  • Compare how long routers, email servers, brokers and DTN nodes hold messages
  • Explain why responsibility passes only when the next holder confirms it

Hop-by-hop delivery without a complete path

Store-and-forward moves a message one hop at a time, and between hops it waits in storage on whichever device or server currently holds it. A live call or a web request needs a complete path from sender to receiver at the moment it runs, and store-and-forward drops that requirement. If the next hop is reachable, the wait is short. If it is not, the message stays put until it is. The sender does not need the recipient to be online, and no single link has to be up for the whole journey.

Store-and-forward cycle at each hop

Every node in a store-and-forward system runs the same cycle: accept the message, store it, wait for a path, then forward it and hand it over.

  1. Accept. The node receives the message and checks that it is well formed, that it is addressed somewhere this node can help with, and that there is room to keep it.
  2. Store. The node writes the message to storage. For long waits this needs to be persistent: the Delay-Tolerant Networking architecture (RFC 4838) expects most nodes to keep bundles on disk or flash so that they survive a restart.
  3. Wait for a path. The node holds the message until a suitable next hop is available. RFC 4838 calls each window in which a link can carry data a “contact”.
  4. Forward and hand over. The node sends the message on. It keeps its own copy until the next node confirms it has taken the message, usually with an acknowledgment, and then it can delete its copy.

On the very first hop, the sending app usually does the same thing locally: it writes the message to an outbox before trying to send it, so that nothing is lost if the app closes.

Message holding times by system

Store-and-forward holding time ranges from about as long as queuing and transmission take in an internet router, to a give-up time for email that generally needs to be at least 4-5 days, to much longer in delay-tolerant networks. The defining question for any store-and-forward system is how long a node is prepared to hold a message.

  • Internet routers store and forward packets too, but RFC 4838 points out that the storage is only expected to last about as long as queuing and transmission delays.
  • Email waits much longer. SMTP (RFC 5321) requires mail that cannot be sent immediately to be queued and retried. It says the retry interval should generally be at least 30 minutes and that the give-up time generally needs to be at least 4-5 days.
  • Message brokers hold messages for clients that are offline. In MQTT 5.0 a broker keeps session state for a disconnected client, including messages pending transmission to it, until the Session Expiry Interval runs out.
  • Delay-tolerant networks may hold data for much longer, across links that only exist at scheduled or unpredictable moments. The delay-tolerant networking article covers this case in depth.

Whatever the system, the message carries or is given a limit. Past that point it is deleted and, ideally, the sender is told. Without a limit, messages for recipients that never return would fill storage forever.

Responsibility handover between nodes

Responsibility for a message passes when another node accepts it, not when the message is sent, and that handover is the important moment in store-and-forward. SMTP makes this explicit: once a receiving server replies with a success code at the end of the message data, it must either deliver the message or report the failure. From that point the sending server is no longer responsible for it.

RFC 4838 describes an equivalent for delay-tolerant networks called custody transfer, in which nodes that agree to take on reliable delivery become “custodians” and retransmit if needed. The pattern is the same in both: the old holder keeps the message until the new holder confirms, so there is never a moment when nobody has it.

The flip side is that a confirmation can be lost. If the receiving node took the message but its confirmation never arrived, the sender sends it again and the receiver ends up with two copies. Many systems therefore give each message an identifier and drop copies they have already seen, a technique covered in message deduplication.

Store-and-forward failure modes

Holding messages for a long time turns storage into a resource that has to be managed. RFC 4838 notes that long-term storage brings problems of congestion management and denial-of-service mitigation, because a node that keeps everything it is given can be filled up.

Order is another casualty. RFC 4838 says the relative order of messages might not be preserved: two messages can wait at different nodes or take different paths, and the later one can arrive first. Applications that care about order need their own sequence numbers or timestamps.

Finally, “the message left my device” says little about what happened next. A store-and-forward system can report several distinct outcomes: stored locally, handed to the next hop, delivered to the recipient, and accepted by the recipient’s application. Good interfaces show which of these the user is looking at.

Store-and-forward messaging use cases

Email is the classic store-and-forward system, relaying through a chain of servers. Message brokers do it for devices with intermittent connections. NASA describes delay/disruption tolerant networking as a store-and-forward approach in which each node stores data until the next node becomes available, and it now runs as an operational service on NASA’s networks. Phone meshes use it too: a phone holds a message until a neighbour comes into range, and the message moves closer through multi-hop relay as devices meet. Machines on a site do the same: in Offline Protocol on WendyOS, a Raspberry Pi and a Dragonwing IQ9 hold their messages for a Jetson while its network cable is out, and deliver every one when it is plugged back in.

Frequently asked questions

Does every hop need to store the message on disk?

Not always. Internet routers hold packets only for about as long as queuing and transmission take. Systems that expect long waits, such as delay-tolerant networks, keep messages in persistent storage so that they survive a restart.

How long does email keep retrying a message it cannot deliver?

SMTP (RFC 5321) requires mail that cannot be sent immediately to be queued and retried. It says the retry interval should generally be at least 30 minutes and that the give-up time generally needs to be at least 4-5 days.

How long does an MQTT broker hold messages for an offline client?

In MQTT 5.0 a broker keeps session state for a disconnected client, including messages pending transmission to it, until the Session Expiry Interval runs out.

Can store-and-forward deliver messages out of order?

Yes. RFC 4838 says the relative order of messages might not be preserved, because two messages can wait at different nodes or take different paths. Applications that care about order need their own sequence numbers or timestamps.

Why does store-and-forward sometimes deliver a message twice?

If the receiving node took the message but its confirmation never arrived, the sender sends it again and the receiver ends up with two copies. Many systems therefore give each message an identifier and drop copies they have already seen.

Sources

Build it with Offline Protocol

The Offline Protocol configuration reference lists the mesh SDK's outbox, acknowledgment, retry and deduplication defaults for v0.33.0. It also notes that retention bounds must cover the longest disconnection you expect.

Read the configuration reference