Encryption and identity

How does group encryption work?

Group encryption lets a set of people exchange messages that only current members can read. The simplest method encrypts every message separately for each member; sender keys let each sender encrypt once with a key shared in advance; and MLS (RFC 9420) arranges keys in a tree so the group can agree new secrets, with costs that scale as the log of the group size.

TL;DR

Pairwise fan-out keeps membership changes simple but costs one encryption per member per message, while sender keys make sending cheap but removals and compromise recovery expensive. MLS encrypts each message once and changes keys through Commits whose cost scales as the log of the group size. Without a central server, members must still agree which Commit wins each epoch.

Learning objectives

After reading this article you will be able to:

  • Compare pairwise fan-out, sender keys and MLS trees on sending and membership costs
  • Explain why removing a member is expensive with sender keys
  • Describe how MLS Commits and Welcome messages change group keys through the ratchet tree

Group encryption vs two-party encryption

A group adds two problems to end-to-end encryption, which between two people is a well studied problem: the same message has to reach several readers, and membership changes. RFC 9420, the MLS standard, notes that for two parties the Double Ratchet has emerged as a common solution, with fine-grained forward secrecy and post-compromise security that is still efficient enough for heavy use over low-bandwidth networks.

Each reader must be able to decrypt the message, and membership keeps changing: people join, leave and lose devices, and each change should cut off anyone who is no longer a member. The three main designs make different trade-offs between the cost of sending a message and the cost of changing who is in the group.

Pairwise fan-out group encryption

Pairwise fan-out, the simplest approach, reuses one-to-one encryption. The sender keeps a separate encrypted session with every other member and sends each one its own copy of the message. Every copy gets the full protection of the underlying session, including the forward secrecy and post-compromise security of a protocol like the Double Ratchet.

Membership changes are simple: to add someone, set up a session with them; to remove someone, stop sending them copies. The cost falls on every message instead. The sender encrypts and transmits once per member, so the work and the traffic for each message grow with the size of the group.

Sender keys in group encryption

Sender keys move the cost from each message to set-up. WhatsApp’s encryption white paper describes the scheme as it deploys it:

  1. The first time a member sends to a group, they generate a random chain key and a signature key pair, combine them into a sender key, and encrypt that sender key to each member individually over the existing pairwise sessions.
  2. For every later message, the sender derives a message key from the chain key, advances the chain key, encrypts the message once, signs it, and sends the single ciphertext to the server, which fans it out to the group.

The hash ratchet on the chain key provides forward secrecy. Removals are the expensive part: the white paper states that whenever a member leaves, all group participants clear their sender keys and start over, which means every sender distributes a new key to every remaining member.

RFC 9420 summarises the trade-off. Sender keys improve efficiency compared with pairwise messages, but post-compromise security is hard to achieve, requiring key update messages that scale as the square of the group size. An attacker who learns one sender key can often keep reading that member’s messages passively, and replacing it costs work that scales linearly with the group.

MLS tree-based group keys

MLS takes a third route to group encryption. Its ratchet tree places each member at a leaf. Each node above the leaves holds a public key for the subgroup beneath it, and the tree invariant says a member knows a node’s private key only if the member’s leaf is under that node. The root is therefore known to every member and to no one else.

Every change to the group happens through a Commit, which starts a new epoch. The member who commits generates fresh secrets for the nodes on the path from its leaf to the root, and encrypts each one to the subtrees that need it rather than to every member individually. New members receive a Welcome message with what they need to join the current epoch. The RFC states that this lets members derive and update shared keys with costs that scale as the log of the group size. Messages themselves are encrypted once for the group, as with sender keys. How MLS works goes through epochs and the key schedule.

Group encryption join and removal costs

With pairwise fan-out, removing a member means you stop sending to them; with sender keys, every sender issues a new key to every remaining member; with MLS, it takes one Commit whose cost scales as the log of the group size.

ApproachSending one messageAdding a memberRemoving a member
Pairwise fan-outOne encryption per memberSet up one new sessionStop sending to them
Sender keysOne encryption, fanned out by a serverThe newcomer needs each sender’s key, sent over pairwise sessionsEvery sender issues a new key to every remaining member
MLSOne encryption for the groupA Commit to the group and a Welcome to the newcomerOne Commit; cost scales as the log of the group size

MLS changes the guarantees as well as the costs. Its Commit gives fresh entropy only to members of the new epoch, so the epoch secret stays confidential to current members. RFC 9750 adds that compromise of a removed member does not affect messages sent after the removal.

Group encryption without a central server

Group encryption can work without a central server. RFC 9750 describes a delivery service that stores key packages and routes messages, but it also says that service can be completely abstract in a peer-to-peer system, provided clients implement the delivery properties themselves. One of those properties is agreement on order: each epoch needs every member to process the same Commit, and the RFC describes how groups can end up in forked views when members process different ones. A mesh or offline deployment has to settle which Commit wins without a server to decide.

Offline Protocol’s mesh SDK is one documented example of MLS groups without a central server: inviting a member sends a Welcome to the invitee and a Commit to existing members, and the default configuration supports groups of up to 256 members.

Frequently asked questions

Can a new member read messages sent before they joined?

Not in MLS. RFC 9420 says a new member can read and send new messages, but messages sent before they were added are not accessible to them.

Does a removed member keep access to the group?

In MLS, no. The Commit that removes them gives fresh entropy only to the remaining members, and RFC 9750 notes that messages sent after a removal are protected with keys the removed client cannot access.

Why is removing a member expensive with sender keys?

WhatsApp's encryption white paper states that whenever a member leaves, all group participants clear their sender keys and start over. Every sender then distributes a new key to every remaining member.

How does WhatsApp encrypt group messages?

It uses sender keys. The first time a member sends to a group, they encrypt a new sender key to each member over the existing pairwise sessions; after that, each message is encrypted once, signed and sent as a single ciphertext to the server, which fans it out to the group.

Do sender keys provide post-compromise security?

Only at a high cost. RFC 9420 says post-compromise security is hard to achieve with sender keys, requiring key update messages that scale as the square of the group size, and an attacker who learns one sender key can often keep reading that member's messages passively.

How many members can an MLS group have in the Offline Protocol mesh SDK?

The default configuration supports groups of up to 256 members. Inviting a member sends a Welcome to the invitee and a Commit to existing members.

Sources

Build it with Offline Protocol

The groups API reference lists the mesh SDK calls to create an MLS group, invite and remove members and send group messages, and notes that an invitation sends a Welcome to the invitee and a Commit to existing members.

Read the groups API reference