Code quality

nimble-code answers questions about the quality of one unit of code, e.g., a function, a file or a diff. It takes the same three types of question as nimble-latest, which are noul, choice and score. Send the request to POST /v1/systemone with the model nimble-code:

Put the unit of code in state, as a JSON object. Describe the code lists its fields.
Add your questions about the code. The question IDs are your own, and the answers use the same IDs.

Make a request

curl https://api.bespokelabs.ai/v1/systemone \
  -H "Authorization: Bearer $BESPOKE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nimble-code",
    "state": {
      "repo": "example/shop",
      "commit": "4f2c9d1e8b7a6c5d4e3f2a1b0c9d8e7f6a5b4c3d",
      "language": "python",
      "unit": "function",
      "path": "shop/cart.py",
      "lines": [10, 16],
      "code": "  10 def apply_discount(cart, code):\n  11     discount = DISCOUNTS.get(code)\n  12     try:\n  13         cart.total = cart.total * (1 - discount.rate)\n  14     except Exception:\n  15         pass\n  16     return cart"
    },
    "questions": {
      "hides_errors": {
        "type": "noul",
        "instructions": "Does this function hide errors from its caller?",
        "criteria": { "true": "yes", "false": "no" }
      },
      "main_problem": {
        "type": "choice",
        "instructions": "What is the main reliability problem in this function, if any?",
        "criteria": {
          "unchecked value": "a value that can be missing is used without a check",
          "swallowed error": "an error is caught and then dropped",
          "no problem": null
        }
      },
      "error_handling": {
        "type": "score",
        "instructions": "How well does this function handle errors?",
        "criteria": [
          "It hides or ignores errors in normal use.",
          "It handles some errors and hides others.",
          "It handles or reports every error."
        ]
      }
    }
  }'

import os

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient(
    api_key=os.environ["BESPOKE_API_KEY"],
    base_url="https://api.bespokelabs.ai",
    model="nimble-code",
)

code = """  10 def apply_discount(cart, code):
  11     discount = DISCOUNTS.get(code)
  12     try:
  13         cart.total = cart.total * (1 - discount.rate)
  14     except Exception:
  15         pass
  16     return cart"""

state = {
    "repo": "example/shop",
    "commit": "4f2c9d1e8b7a6c5d4e3f2a1b0c9d8e7f6a5b4c3d",
    "language": "python",
    "unit": "function",
    "path": "shop/cart.py",
    "lines": [10, 16],
    "code": code,
}
questions = {
    "hides_errors": Noul(
        instructions="Does this function hide errors from its caller?",
        criteria={"true": "yes", "false": "no"},
    ),
    "main_problem": Choice(
        instructions="What is the main reliability problem in this function, if any?",
        criteria={
            "unchecked value": "a value that can be missing is used without a check",
            "swallowed error": "an error is caught and then dropped",
            "no problem": None,
        },
    ),
    "error_handling": Score(
        instructions="How well does this function handle errors?",
        criteria=[
            "It hides or ignores errors in normal use.",
            "It handles some errors and hides others.",
            "It handles or reports every error.",
        ],
    ),
}

response = client.system_one(state, questions)
print(response.answers["hides_errors"].noul)
print(response.answers["main_problem"].choice)
print(response.answers["error_handling"].score)

Read the answer

{
  "model": "nimble-code",
  "answers": {
    "hides_errors": { "type": "noul", "noul": 0.947 },
    "main_problem": {
      "type": "choice",
      "choice": "swallowed error",
      "probabilities": {
        "unchecked value": 0.05,
        "swallowed error": 0.893,
        "no problem": 0.057
      },
      "confidence": 0.622
    },
    "error_handling": {
      "type": "score",
      "score": 0.512,
      "legend": {
        "0": "It hides or ignores errors in normal use.",
        "1": "It handles some errors and hides others.",
        "2": "It handles or reports every error."
      },
      "probabilities": { "0": 0.629, "1": 0.231, "2": 0.14 },
      "confidence": 0.175
    }
  },
  "usage": { "input_tokens": 545, "output_tokens": 4 }
}

The noul of hides_errors is the probability that the function hides errors from its caller.
The choice of main_problem is the option with the highest probability. probabilities gives every option, and they add up to 1.
The score of error_handling is the expected level, counting from 0. Here the most likely level is 0, and the expected level is 0.512.
confidence is 1 when one option has all the probability, and 0 when every option is equally likely. It is not the chance that the answer is right.
usage.input_tokens is the number of input tokens that you pay for. Pricing explains how Nimble counts them.

Describe the code

Send one unit of code in each request. The state is a JSON object with these fields:

Field	What it holds
`repo`	The repository, e.g., `example/shop`.
`commit`	The commit that the code comes from.
`language`	The programming language, e.g., `python`.
`unit`	The kind of unit, e.g., `function`, `file` or `diff`.
`path`	The path of the file in the repository.
`lines`	The first and the last line of the unit in the file.
`code`	The code, with its line number at the start of each line, as in the example.

Nimble reads the state as JSON text. A state can also be a string, or another JSON object or list, but nimble-code was trained on states with these fields. Each question gets its own prompt, with the whole state in it. So a question’s answer does not depend on the other questions in the request, or on the question’s ID. The option keys of a choice question are part of its prompt, so renaming an option can change the answer.

Limits

Limit	Value
Questions in a request	1 to 64
Options in a `choice` question, or levels in a `score` question	2 to 26
Prompt tokens for one question, including the state	8,192

Nimble never cuts a prompt. A request gets 422 if the prompt for any of its questions is longer than 8,192 tokens.
When nimble-code is busy, a request gets 529. Retry it after about one second, with backoff.

The limits page lists the limits that apply to every model.

Get started

Guides

Make a request

Read the answer

Describe the code

Limits

​Make a request

​Read the answer

​Describe the code

​Limits

Make a request

Read the answer

Describe the code

Limits