What is challenge-response authentication?

NIST SP 800-63B-4 defines a challenge-response protocol as one in which the verifier sends the claimant a challenge, such as a random value or nonce, which the claimant combines with a secret to produce a response. The verifier checks the response independently and so establishes that the claimant possesses and controls the secret. The secret itself never crosses the wire.

The exchange has three steps:

1. Challenge. The verifier generates a fresh value and sends it.
2. Response. The claimant computes a response from the challenge and its secret.
3. Check. The verifier confirms the response is correct for that challenge and that key, and that the challenge has not been used before.

There are two families, depending on the kind of secret:

• Shared secret. Both sides hold the same key, and the response is a keyed hash of the challenge. The verifier recomputes it and compares. RFC 6287 (OCRA) standardises this, building on the HOTP algorithm. The drawback is that the verifier holds a copy of the secret, so stealing it from the verifier lets an attacker answer challenges.
• Public key. The claimant signs the challenge with a private key and the verifier checks the digital signature with the public key. The verifier holds nothing secret. NIST describes cryptographic authenticators this way: the verifier generates a challenge nonce and the authenticator’s output is generally some type of signed message.

The point of the challenge is replay resistance. NIST explains that protocols using nonces or challenges to prove freshness resist replay because old messages will not contain the right nonce, so a response recorded yesterday is useless today.

That only holds if challenges never repeat and cannot be guessed:

• NIST requires the challenge nonce used with a single-factor cryptographic authenticator to be at least 64 bits and either unique over the authenticator’s lifetime or generated by an approved random bit generator.
• The W3C Web Authentication specification says challenges must be generated randomly by the relying party in an environment it trusts, must match on return, and should be at least 16 bytes. It also says the relying party should store the challenge until the operation completes rather than trusting the client to echo it.

NIST also draws a distinction: a nonce is a value never repeated with the same key, but it is not necessarily unpredictable. A predictable counter meets the definition of a nonce, but an attacker who can predict the next challenge may be able to obtain a response to it in advance.

• Passkeys and security keys. Web Authentication is built on it: the authenticator signs its own data together with a hash of the client data, which carries the relying party’s challenge, using a key scoped to that relying party.
• SSH. During key exchange in RFC 4253, the server signs a hash of the exchange, which includes values both sides derived from fresh random numbers, with its host key. The client checks the signature to authenticate the server.
• Mutual authentication. RFC 6287 describes a mutual mode in which each side sends a challenge and checks the other’s response, so client and server both authenticate.
• Credentials. A credential can name a public key, and the holder proves it is theirs by answering a challenge. This is one way an issuer-signed credential is tied to the person presenting it.

One-time password codes are a close relative. HOTP (RFC 4226) computes codes from a counter and TOTP (RFC 6238) from the time, with a recommended default step of 30 seconds. Neither needs a challenge from the verifier, which makes them simple, but NIST does not count manually entered codes as phishing-resistant, because a fake verifier can pass the code on to the real one.

Nothing in the exchange needs the internet. Two devices on a local link can run it in both directions, so each knows the other holds its key. That is the basis of verifying someone with no internet. Offline Protocol’s identity guide uses the same pattern for account binding: the app verifies a signature by the device’s Ed25519 identity key over a fresh server-issued nonce, checks the derived off1 address, then stores the binding.

Three limits are worth keeping in mind:

• It proves possession, not identity. A correct response shows the claimant holds a particular key. Whose key it is comes from somewhere else: a pinned key, an enrolment step or a credential.
• It does not stop a live relay on its own. An attacker in the middle can pass a challenge to the real key holder and the answer back. NIST’s phishing resistance requirements address this by binding the response to the communication channel or to the verifier’s name.
• The verifier must keep state. It has to remember which challenges it issued and refuse any it has already seen.