Fact check

Nimble fact check takes a document and a list of claims. For each claim, it returns the probability that the document supports the claim. Send the request to POST /v1/systemone with one of the fact check models:

Put the document in state, as a string.
Add one noul question for each claim, with the claim in instructions.
The question IDs are your own. The answers use the same IDs.

Make a request

curl https://api.bespokelabs.ai/v1/systemone \
  -H "Authorization: Bearer $BESPOKE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nimble-factcheck",
    "state": "The board moved the vote to Tuesday because the CFO was travelling.",
    "questions": {
      "moved": { "type": "noul", "instructions": "The vote was moved to Tuesday." },
      "ceo": { "type": "noul", "instructions": "The CEO was travelling." }
    }
  }'

import os

from typesafe_sdk import Noul, TypeSafeClient

client = TypeSafeClient(
    api_key=os.environ["BESPOKE_API_KEY"],
    base_url="https://api.bespokelabs.ai",
    model="nimble-factcheck",
)

response = client.system_one(
    "The board moved the vote to Tuesday because the CFO was travelling.",
    {
        "moved": Noul(instructions="The vote was moved to Tuesday."),
        "ceo": Noul(instructions="The CEO was travelling."),
    },
)
for claim_id, answer in response.answers.items():
    print(claim_id, answer.noul)

Read the answer

{
  "model": "nimble-factcheck",
  "answers": {
    "moved": { "type": "noul", "noul": 0.99 },
    "ceo": { "type": "noul", "noul": 0.002 }
  },
  "usage": { "input_tokens": 25, "output_tokens": 0 },
  "request_id": "6b57f82e-6905-48a1-bc44-fe0af138d8c5"
}

noul is the probability that the document supports every part of the claim.
A claim counts as supported when noul is above 0.5. You can use a higher threshold, e.g., 0.8, when a wrong “supported” costs you more than a wrong “not supported”.
A claim that the document contradicts and a claim that the document does not mention are both not supported.

Models

Model	How it answers
`nimble-factcheck-lite`	A small model scores every claim. This is the fastest and the cheapest.
`nimble-factcheck`	The small model scores every claim. Then a larger model scores again the claims that the small model is unsure about. This is about one claim in five.
`nimble-factcheck-max`	The small model scores every claim. Then two of the largest models score again the claims that the small model is unsure about, and Nimble uses the average of their scores. This is the most accurate.

Accuracy

We measured the fact check models on the test split of LLM-AggreFact, a public benchmark of 11 datasets with 29,320 claims. No model was trained on LLM-AggreFact. Each number is the balanced accuracy at a threshold of 0.5. Balanced accuracy is the average of two shares. One is the share of supported claims that a model calls supported. The other is the share of unsupported claims that it calls unsupported. The table averages it over the 11 datasets, so each dataset counts the same. We also compared the models with Jev 1.13.0 from TypeSafe, on the same claims. Jev scores each claim whole, so the comparison uses whole claims. The last column gives each model’s difference from Jev in points, with a 95% interval from 1,000 resamples within each dataset.

Model	Split claims (the default)	Whole claims	Difference from Jev, whole claims
`nimble-factcheck-lite`	77.7	77.4	-1.3 (-2.2 to -0.3)
`nimble-factcheck`	78.7	78.2	-0.5 (-1.4 to 0.3)
`nimble-factcheck-max`	79.6	79.6	+0.8 (0.1 to 1.7)
Bespoke-MiniCheck-7B	77.1	76.9	-1.8 (-2.7 to -0.8)
Jev 1.13.0		78.7

On whole claims:

nimble-factcheck-lite is 1.3 points behind Jev.
nimble-factcheck is 0.5 points behind Jev. The interval includes zero, so the two are about even.
nimble-factcheck-max is 0.8 points ahead of Jev, and the whole interval is above zero.

Two notes on these numbers:

nimble-factcheck and nimble-factcheck-max send a claim to larger models when the small model scores it between 0.2 and 0.8. We chose that range on this test set, so their numbers are slightly optimistic.
We measured Jev through TypeSafe’s System One API on 2026-09-25. Jev returns probabilities with two decimals, so some claims score exactly 0.50. Those claims count as supported.

Results for each dataset, whole claims

Model	CNN	XSum	MediaSum	MeetingBank	Wice	Reveal	ClaimVerify	FactCheck-GPT	ExpertQA	LFQA	RAGTruth	Average
`nimble-factcheck-lite`	69.6	75.8	73.5	76.9	81.3	89.7	75.7	77.7	60.3	88.8	82.5	77.4
`nimble-factcheck`	67.2	77.3	73.4	81.3	80.7	90.8	78.3	77.2	60.5	89.2	83.8	78.2
`nimble-factcheck-max`	74.2	76.5	74.4	83.6	85.2	91.1	76.6	78.0	59.6	89.0	86.8	79.6
Bespoke-MiniCheck-7B	62.7	76.2	75.7	76.9	84.3	87.9	74.7	77.9	59.3	86.9	83.9	76.9
Jev 1.13.0	67.2	75.5	74.5	83.7	83.9	90.9	76.9	77.0	59.9	89.2	87.0	78.7

CNN and XSum are from AggreFact, and MediaSum and MeetingBank are from TofuEval. RAGTruth has the most claims, 16,371, and Wice has the fewest, 358.

Options

split_claims is true by default. Nimble then scores each sentence of a claim on its own and keeps the lowest score. Set it to false to score each claim whole. Put it at the top level of the request, next to model. With TypeSafe’s SDK, pass extra_body={"split_claims": False} to system_one.

{
  "model": "nimble-factcheck",
  "split_claims": false,
  "state": "The board moved the vote to Tuesday because the CFO was travelling.",
  "questions": {
    "moved": { "type": "noul", "instructions": "The vote was moved to Tuesday because the CFO was travelling." }
  }
}

Limits

Limit	Value
Claims in a request	1 to 64
Characters in one claim	4,000
Characters in the document	400,000

The questions must be noul questions with no criteria. Any other question gets 422. Nimble splits a long document into parts. A sentence counts as supported if any part supports it.

Get started

Guides

Make a request

Read the answer

Models

Accuracy

Options

Limits

​Make a request

​Read the answer

​Models

​Accuracy

​Options

​Limits

Make a request

Read the answer

Models

Accuracy

Options

Limits