Two different starting points
The Signal Protocol is a family of specifications published by Signal at signal.org/docs. They include X3DH and its post-quantum extension PQXDH for agreeing a key, the Double Ratchet for protecting the messages that follow, Sesame for managing sessions across a user’s devices, and XEdDSA for signatures. The key agreement and ratchet specifications describe communication between two parties; none of them defines a group key agreement.
MLS, the Messaging Layer Security protocol, is an IETF standard published as RFC 9420 in July 2023, with its architecture in RFC 9750. It defines key agreement for a group, which can be as small as two members or as large as thousands.
Both aim at end-to-end encryption that works when people are not online at the same time, with forward secrecy and recovery after a compromise. The difference is the unit they are built around: a pair of people, or a group.
Starting a conversation
Both use keys published in advance so a sender can start without waiting for the recipient.
In X3DH, the recipient uploads an identity key, a signed prekey and a set of one-time prekeys to a server. The sender fetches a prekey bundle, performs several Diffie-Hellman calculations against those keys, and derives a shared secret it can use to encrypt a first message straight away. PQXDH extends this with a post-quantum key encapsulation mechanism, giving post-quantum forward secrecy, while mutual authentication still relies on the hardness of the discrete log problem.
In MLS, each client publishes KeyPackages through the Delivery Service. A member who wants to add someone uses their KeyPackage to add them to the group and sends them a Welcome message with the group’s current state. How does MLS work? goes through this in detail.
Keeping keys fresh
The Double Ratchet derives a new key for every message, so earlier keys cannot be calculated from later ones. Each message header also carries the sender’s current ratchet public key, and the parties take turns replacing those keys and mixing the new Diffie-Hellman outputs into their chains. That lets a session recover after an attacker briefly learns a party’s keys, which the specification calls break-in recovery.
MLS refreshes keys per epoch rather than per exchange. A member gains forward secrecy within an epoch by deleting each message key after use, and across epochs by deleting old tree keys. Recovery after a compromise comes when a member updates its leaf key in a Commit. RFC 9420 suggests doing this on the order of hours or days, not with every message.
Groups are where they differ most
With only two-party sessions, a group has two options, both described in the introduction to RFC 9420. Members can encrypt each message separately for every other member over pairwise channels. Or each member can distribute a symmetric sender key over those channels and encrypt group messages once with it. Sender keys are efficient and give forward secrecy with a hash ratchet, but RFC 9420 says recovering from a compromise with them is difficult: it needs a number of key update messages that grows with the square of the group size, and an attacker who learns a sender key can often keep reading that member’s messages.
MLS places members in a ratchet tree instead, which lets the group derive and update its shared keys at a cost that scales with the logarithm of the group size. The RFC 9420 introduction gives this scaling problem as the motivation for the design.
| Signal specifications | MLS (RFC 9420) | |
|---|---|---|
| Unit of protection | A session between two parties | A group of two or more members |
| Key refresh | A new key per message, and a Diffie-Hellman step when the other side replies | Each epoch, started by a Commit |
| Group key updates | Not defined in the published specifications | Ratchet tree, logarithmic in group size |
| Ordering needed | Out-of-order messages handled with skipped keys | Members must agree on one Commit per epoch |
| Deniability | Stated goal of X3DH and PQXDH | No claims made |
Other trade-offs
Ordering. The Double Ratchet tolerates lost and out-of-order messages by numbering them and keeping skipped keys for a while. MLS also tolerates out-of-order application messages, but the group has a linear history of epochs, so everyone must apply the same Commit. RFC 9750 explains that the Delivery Service usually orders Commits, and that two members committing at once is a realistic risk. It frames the choice of ordering strategy in terms of consistency and partition tolerance, which matters on links that are slow or often split.
Deniability. X3DH is designed so that neither party gets a publishable cryptographic proof of what was said or that they talked. RFC 9750 says MLS makes no claims about deniability and supports deployments where a recipient can prove who sent a message, for example to report abuse.
Post-quantum work. PQXDH adds post-quantum forward secrecy to the initial key agreement, and the current revision of the Double Ratchet specification adds a Sparse Post-Quantum Ratchet and a Triple Ratchet that combines it with the original. The cipher suites defined in RFC 9420 use elliptic-curve Diffie-Hellman key encapsulation.
Standardisation. MLS is an IETF Standards Track protocol with a cipher suite registry for interoperable implementations. The Signal specifications are published by Signal and placed in the public domain.