Offline-first and sync

What is a sync engine?

A sync engine is software that keeps a local copy of an app's data in step with other copies, usually a server database and sometimes other devices. The app reads and writes the local copy, and the engine moves changes in both directions, tracks what each side has seen, and applies the rules for combining changes.

Learning objectives

After reading this article you will be able to:

  • List the jobs a sync engine handles between a local store and other copies
  • Compare server-authoritative, read-path-only and peer-to-peer sync engine designs
  • Identify the questions to ask before choosing a sync engine

The job it does

An app that works offline keeps a local copy of its data, and that copy has to be kept in step with everyone else’s. Doing this by hand means writing code to queue local changes, send them, fetch other people’s changes, apply them without losing pending work, and redraw the screen. A sync engine is a component that does this as one system.

The 2019 essay Local-first software by Ink and Switch described Firebase as essentially a local on-device database combined with a cloud database service and data synchronization between the two, which is a fair description of the pattern. The essay also suggested a market for a “Firebase for CRDTs”, a sync service built on CRDTs, data types that merge without a central coordinator.

What a sync engine handles

Most sync engines cover the same set of jobs:

  • A local store. An embedded database or key-value store the app reads and writes directly, so screens do not wait for a network.
  • Capturing local changes. Each write is recorded in a persistent queue so it survives restarts and can be sent later.
  • Sending changes up. The queue is drained to a server or peer, with retries.
  • Bringing changes down. The engine fetches or receives what changed elsewhere, usually as deltas rather than full copies.
  • Combining the two. Incoming changes are merged with pending local ones using a defined rule.
  • Telling the interface. Screens subscribe to queries and redraw when results change, whatever caused the change.
  • Choosing what syncs. Each device gets only the data it should see, which is both a size limit and a permission boundary.

The main designs

Sync engines differ most in where authority lives and how writes travel. The examples below are described from each vendor’s own documentation.

DesignExampleHow it works
Server-authoritative, replay on the clientReplicacheThe client runs changes locally as speculative results, pushes them to the server, which runs them again and is authoritative, then pulls a patch and replays any still-pending changes on top.
Server database replicated to the devicePowerSyncThe PowerSync Service replicates data from your backend database, partitions it by what each user should receive, and streams it to SQLite on the device. Local writes go into an upload queue that your own backend API applies.
Read path onlyElectricElectric syncs data out of Postgres into local clients and does not do write-path sync; its guide describes patterns for sending writes through your API.
Peer-to-peer with CRDTsDittoEach device has a local database and syncs directly with nearby devices over Bluetooth, peer-to-peer Wi-Fi, or a local network, merging concurrent edits with CRDTs. Sync does not rely on Ditto’s server, which adds cloud sync and integration.

The first three keep the server as the judge. A device can write offline, but a change becomes final only when the backend accepts it, and the engine has to handle the case where it does not. PowerSync’s guidance is a good illustration of what that involves: return a success response even for validation errors and pass the error back to the client, because an error response would block the upload queue.

The peer-to-peer design lets devices agree with each other without any server, which is what peer-to-peer sync vs cloud sync compares. The trade-off moves elsewhere: there is no single authority to reject a change, so the data types themselves have to make every merge acceptable.

Questions to ask before choosing one

  • Which database is the source of truth? Some engines replicate a particular server database; others bring their own.
  • How do writes reach the backend? Through the engine, or through your existing API with its validation and permissions?
  • What happens when a write is rejected? The person who made it needs to find out, possibly long after they saw it succeed as an optimistic update.
  • How is the data scoped? Partial sync should follow your access rules, not just reduce volume.
  • Must devices sync without a server? If people work together where nobody has a connection, the engine needs a peer-to-peer path.

What a sync engine does not do

A sync engine makes copies agree. It does not make the agreed result correct for the business. If two people each claim the last item while offline, both claims can sync cleanly into a state that breaks a rule. Offline Protocol’s shared state guide says the same of its own replicated documents: convergence does not enforce inventory limits, exclusive assignments, or other business invariants, so the application must put those rules somewhere with the authority to decide. How does offline sync work? explains the convergence part in more detail.

Frequently asked questions

Is a sync engine the same as a database?

Not quite. Most sync engines include or wrap a local database on the device, but the engine is the part that moves changes between that database and other copies. Some work alongside a server database you already run.

Do I need a sync engine to build an offline-first app?

No. A local database, a queue of pending changes, and an API that accepts them are enough for many apps. A sync engine packages those parts, and is most useful when data is shared, changes often, or must merge without a server.

Sources

Build it with Offline Protocol

Offline Protocol's replicated documents let nearby devices edit shared state and merge changes when replicas communicate, while your application keeps its existing database for records outside that shared scope. The shared state guide covers merge rules, scoping, and convergence tests.

Read the shared state guide