---
title: "Run the Open Responses conformance suite"
description: "Mint a key the Open Responses conformance CLI can use, point it at api.quiver.ai, and read the one result that needs a production key."
icon: "check-check"
---

The [Open Responses](https://www.openresponses.org/) project publishes a conformance CLI in the
`openresponses/openresponses` repository. It calls `POST /v1/responses` on any host that implements
the protocol and reports, test by test, what the host got right. This guide covers what a QuiverAI
key needs before that suite can run, which of its tests apply, and the one test that needs a
production key.

## Create the key

Create the key in the API Platform at [API Keys](https://platform.quiver.ai/api-keys). Two of
its authority choices decide whether the suite runs at all:

- **Capabilities.** Grant **Responses**, the capability `/v1/responses` requires. Nothing else in
  the suite needs a second capability.
- **Models.** Grant `arrow-2`, or all current and future models. The suite sends one model on every
  request and `arrow-2` is the model to send.

Capabilities and models are fixed when the key is created. A key minted without **Responses**
cannot be repaired — create a second key rather than trying to widen the first.

## Choose the environment

The environment is the other choice made once, at creation:

- An `sk_test_` key runs the suite against the sandbox. Nothing is billed, nothing reaches a model,
  and every response is built from the same published schema a production call returns. See
  [Sandbox and test keys](/developers/guides/sandbox-and-test-keys).
- An `sk_live_` key runs the suite against the real model. Requests are billed as ordinary token
  usage.

Start on a test key: it proves your host wiring, the request shape, and the streaming event
sequence without spending anything.

## Run the suite

```bash
bun run bin/compliance-test.ts \
  --base-url https://api.quiver.ai/v1 \
  --api-key "$QUIVERAI_API_KEY" \
  --model arrow-2
```

`--base-url` is the `/v1` prefix, not the endpoint: the CLI appends `/responses` itself. There is no
separate sandbox host — a test key and a production key both call `https://api.quiver.ai`.

The suite also ships seven `websocket-*` tests and two `compaction-*` tests. `/v1/responses` is a
stateless HTTP endpoint and serves neither by design, so those nine fail on every key; use the
CLI's `--filter` option to run the HTTP tests only.

## Run `tool-calling` against a production key

Run the `tool-calling` test on an `sk_live_` key. The sandbox returns a `function_call` output item
only for the file-writing tool shape, so the suite's own tool declaration comes back as an assistant
message and the test reports a failure that says nothing about the API. The remaining HTTP tests
are answered the same way by both environments.

## If a request is refused

A key that was minted without **Responses** is refused with `403 unauthorized`, and the message
names the missing capability and the remedy:

```json
{
  "status": 403,
  "code": "unauthorized",
  "message": "This API key is not allowed to use Responses. A key's capabilities are fixed when it is created, so create a new key with the Responses capability in the API Platform at https://platform.quiver.ai/api-keys, then retry.",
  "request_id": "550e8400-e29b-41d4-a716-446655440000"
}
```

A `404 model_not_found` on `arrow-2` is a different problem: the model is not in the catalog this
key can reach. Confirm it with `GET /v1/models` on the same key. See
[Errors and debugging](/developers/guides/errors-and-debugging) for the full status and code matrix.