Skip to content
Esc
↑↓navigate↵open⌘Jpreview
On this page

Run the Open Responses conformance suite

Mint a key the Open Responses conformance CLI can use, point it at api.quiver.ai, and read the one result that needs a production key.

The Open Responses project publishes a conformance CLI in the openresponses/openresponses repository. It calls POST /v1/responses on any host that implements the protocol and reports, test by test, what the host got right. This guide covers what a QuiverAI key needs before that suite can run, which of its tests apply, and the one test that needs a production key.

Create the key

Create the key in the API Platform at API Keys. Two of its authority choices decide whether the suite runs at all:

  • Capabilities. Grant Responses, the capability /v1/responses requires. Nothing else in the suite needs a second capability.
  • Models. Grant arrow-2, or all current and future models. The suite sends one model on every request and arrow-2 is the model to send.

Capabilities and models are fixed when the key is created. A key minted without Responses cannot be repaired — create a second key rather than trying to widen the first.

Choose the environment

The environment is the other choice made once, at creation:

  • An sk_test_ key runs the suite against the sandbox. Nothing is billed, nothing reaches a model, and every response is built from the same published schema a production call returns. See Sandbox and test keys.
  • An sk_live_ key runs the suite against the real model. Requests are billed as ordinary token usage.

Start on a test key: it proves your host wiring, the request shape, and the streaming event sequence without spending anything.

Run the suite

bun run bin/compliance-test.ts \
  --base-url https://api.quiver.ai/v1 \
  --api-key "$QUIVERAI_API_KEY" \
  --model arrow-2

--base-url is the /v1 prefix, not the endpoint: the CLI appends /responses itself. There is no separate sandbox host — a test key and a production key both call https://api.quiver.ai.

The suite also ships seven websocket-* tests and two compaction-* tests. /v1/responses is a stateless HTTP endpoint and serves neither by design, so those nine fail on every key; use the CLI’s --filter option to run the HTTP tests only.

Run tool-calling against a production key

Run the tool-calling test on an sk_live_ key. The sandbox returns a function_call output item only for the file-writing tool shape, so the suite’s own tool declaration comes back as an assistant message and the test reports a failure that says nothing about the API. The remaining HTTP tests are answered the same way by both environments.

If a request is refused

A key that was minted without Responses is refused with 403 unauthorized, and the message names the missing capability and the remedy:

{
  "status": 403,
  "code": "unauthorized",
  "message": "This API key is not allowed to use Responses. A key's capabilities are fixed when it is created, so create a new key with the Responses capability in the API Platform at https://platform.quiver.ai/api-keys, then retry.",
  "request_id": "550e8400-e29b-41d4-a716-446655440000"
}

A 404 model_not_found on arrow-2 is a different problem: the model is not in the catalog this key can reach. Confirm it with GET /v1/models on the same key. See Errors and debugging for the full status and code matrix.

Was this page helpful?