Code execution

curator.CodeExecutor runs code for each row of a dataset and collects the output. This helps in cases like these.

You want only code that runs without errors in your training data. Open Thoughts used this method.
An LLM wrote code that draws a chart or renders a video, and you want the result.
You are building agents that use tools.

Example

from datasets import Dataset

from bespokelabs import curator


class HelloExecutor(curator.CodeExecutor):
    def code(self, row):
        return """location = input(); print(f"Hello {location}")"""

    def code_input(self, row):
        return row["location"]

    def code_output(self, row, execution_output):
        row["output"] = execution_output.stdout
        return row


locations = Dataset.from_list([{"location": "New York"}, {"location": "Tokyo"}])

hello_executor = HelloExecutor()
print(hello_executor(locations).to_pandas())
# Output:
#    location          output
# 0  New York  Hello New York
# 1     Tokyo     Hello Tokyo

A subclass of CodeExecutor has three methods.

code returns the Python code to run for a row. The code usually comes from the row, e.g., from a column that a curator.LLM step wrote.
code_input is optional. It returns the text that the code reads with input().
code_output gets the row and the result of the run, and returns the output row. The result has the attributes stdout, stderr, message, and error. The value of message is "success", "timeout", or "error".

Calling a CodeExecutor returns a Hugging Face Dataset. Like curator.LLM, the code executor caches its results. If a run stops, you can start it again without losing the work that finished. A progress display shows how the run is going.

Execution settings

Pass execution_params when you call the executor. timeout is the most seconds one run can take. The default is 10.

output = hello_executor(locations, execution_params={"timeout": 120})

Backends

The backend decides where the code runs. Curator uses the bespokelabs-sandbox package for this, and you choose the backend with the backend argument.

Backend	Where the code runs	Setup
`local`	On your machine, in a temporary directory. This is the default.	None.
`docker`	In a Docker container.	Docker, and `pip install "bespokelabs-sandbox[docker]"`.
`ray`	On a Ray cluster.	`pip install "bespokelabs-sandbox[ray]"`.
`e2b`	In an E2B cloud sandbox.	`pip install "bespokelabs-sandbox[e2b]"` and `E2B_API_KEY`.
`modal`	In a Modal sandbox.	`pip install "bespokelabs-sandbox[modal]"` and a Modal account.
`daytona`	In a Daytona sandbox.	`pip install "bespokelabs-sandbox[daytona]"` and `DAYTONA_API_KEY`.

The local backend runs the code directly on your machine. It is fast and needs no setup, but it is the least safe choice for code you do not trust. Use a container or cloud backend for that code.

The old name multiprocessing still works and means local.

Local

hello_executor = HelloExecutor(backend_params={"max_requests_per_minute": 1000})

Docker

Install Docker, e.g., Docker Desktop, and make sure it is running. Curator uses the python:3.12-slim image by default. To use a different image, set image in backend_params.

hello_executor = HelloExecutor(
    backend="docker",
    backend_params={"image": "andgineer/matplotlib"},
)

Ray

Use Ray when your dataset is too large to run on one machine. Set the RAY_ADDRESS environment variable to the address of your Ray cluster. If it is not set, Ray starts a local cluster.

hello_executor = HelloExecutor(backend="ray")

E2B

E2B is a paid service. Create an account on the E2B website, then set your key.

export E2B_API_KEY=<your-api-key>

hello_executor = HelloExecutor(backend="e2b")

Backend settings

Parameter	Default	Description
`max_requests_per_minute`	`10000`	The most runs Curator starts in one minute.
`max_retries`	`3`	The number of times Curator retries a failed run.
`seconds_to_pause_on_rate_limit`	`10`	Seconds to wait after hitting a rate limit.
`image`	None	The container image, for backends that use one.

More examples

The code execution examples on GitHub include a chart generator and a pipeline that renders math videos. If you have questions, ask in the Bespoke Labs Discord.

Get started

Core features

Guides

Recipes

Fine-tuning

Reference

Example

Execution settings

Backends

Local

Docker

Ray

E2B

Backend settings

More examples

​Example

​Execution settings

​Backends

​Local

​Docker

​Ray

​E2B

​Backend settings

​More examples

Example

Execution settings

Backends

Local

Docker

Ray

E2B

Backend settings

More examples