Bespoke Labs
  • Welcome
  • BESPOKE CURATOR
    • Getting Started
      • Quick Tour
      • Key Concepts
      • Visualize your dataset with the Bespoke Curator Viewer
      • Automatic recovery and caching
      • Structured Output
    • Save $$$ on LLM inference
      • Using OpenAI for batch inference
      • Using Anthropic for batch inference
      • Using Gemini for batch inference
      • Using Mistral for batch inference
      • Using kluster.ai for batch inference
    • How-to Guides
      • Using vLLM with Curator
      • Using Ollama with Curator
      • Using LiteLLM with curator
      • Handling Multimodal Data in Curator
      • Executing LLM-generated code
      • Using HuggingFace inference providers with Curator
    • Data Curation Recipes
      • Generating a diverse QA dataset
      • Using SimpleStrat block for generating diverse data
      • Curate Reasoning data with Claude-3.7 Sonnet
      • Synthetic Data for function calling
    • Finetuning Examples
      • Aspect based sentiment analysis
      • Finetuning a model to identify features of a product
    • API Reference
  • Models
    • Bespoke MiniCheck
      • Self-Hosting
      • Integrations
      • API Service
    • Bespoke MiniChart
    • OpenThinker
Powered by GitBook
On this page
  • Features:
  • Backends
  • Backend Setup and configuration options
  • Conclusion
  1. BESPOKE CURATOR
  2. How-to Guides

Executing LLM-generated code

PreviousHandling Multimodal Data in CuratorNextUsing HuggingFace inference providers with Curator

Last updated 2 months ago

We have built a code-executor that can be used to execute LLM-generated code. This is useful for many situations:

  1. You want to include error-free code in your training code. This method is used in .

  2. LLM generates some code to generate visualization etc.

  3. Agents and tool-use.

Here is a simple example of code execution in action:

from bespokelabs import curator
from datasets import Dataset

class HelloExecutor(curator.CodeExecutor):     
	def code(self, row):
		return """location = input();print(f"Hello {location}")"""     
		
	def code_input(self, row):        
		return row['location']     
	
	def code_output(self, row, execution_output):        
		row['output'] = execution_output.stdout 
		return row
		
locations = Dataset.from_list([{'location': 'New York'},{'location': 'Tokyo'}])

hello_executor = HelloExecutor() 

print(hello_executor(locations).to_pandas())

The inherited class contains three methods:

  1. code: This is the method that returns the piece of code to be run. This is usually part of the row (you can use curator.LLM to generate this code).

  2. code_input: This is optional, but can return a json that represents values to be passed to input() in the code.

  3. code_output: This is where you parse the output of the execution.

Features:

  1. Multiple code execution backends: We offer four backends: multiprocessing, docker, ray and E2B. These backends specify different locations where your code can be executed. You can easily switch the backend with a simple parameter change. For example, the hello world example can be run using the ray backend by simply initializing it with `HelloExecutor(backend=ray)`

  2. Progress monitoring using Rich Console:

Backends

We offer four backends for running your code:

  1. Multiprocessing: This is the default backend. This runs code locally and is therefore the least safe option, but is useful for quick execution as it does not require any dependencies.

  2. Docker: It is safer option than multiprocessing.

  3. Ray: If you have a ray cluster, you can use it by setting CodeExecutor(backend="ray"). This is useful when your code can take a long time to run.

Backend Setup and configuration options

Multiprocessing Backend:

This doesn't require any additional setup. You can configure backend params while initializing as follows:

```python
hello_executor = HelloExecutor(
    backend_params = {    
        "max_requests_per_minute": 1000
    }
)
```        

You can also configure execution parameters:

```python

output = hello_executor(dataset, 
    execution_params = {
      "timeout": 120 # in seconds  
      "memory_limit": 1024 ** 3 
   }
)
```

Docker

With docker, code can be executed in a secure containerized environment. You need docker installed and python's docker client installed on your machine:

  1. pip install docker

  2. In your terminal, run `docker pull python:3.11-slim`

  3. Run the HelloExecutor example with HelloExecutor(backend=docker)

  1. pip install docker

  2. In your terminal, run `docker pull python:3.11-slim`

  3. Run the HelloExecutor example with HelloExecutor(backend=docker)

With docker, you can specify a custom docker image to execute your code snippets:

```python
hello_executor = HelloExecutor(
    backend = "docker",
    backend_params = {    
        "image": "andgineer/matplotlib"
    }
)
```  

Ray

As the size of the dataset grows, it becomes harder to scale code execution requirements on a single machine. In such scenarios, one can use the ray backend.

Simply run pip install ray to install the dependencies required for ray backend.

```python
hello_executor = HelloExecutor(
    backend = "ray",
    backend_params = {    
        "base_url": "<url of ray cluster>"
    }
)
```  

E2B

We also add light support for e2b's hosted code execution backends. While not free, they are secure environments similar to docker environments and have more features.

  1. Run pip install e2b-code-interpreter to install the required dependencies.

  2. Create an account on e2b's website, get the API key and add it to your environment variables.

```python
hello_executor = HelloExecutor(
    backend = "e2b",
)
```  

Conclusion

Full caching and automatic recovery: Similar to , code executor also has inbuilt caching and automatic recovery. Any interrupted runs can be fully recovered and no computation is lost.

E2B: Code can also be run using . Use CodeExecutor(backend="e2b") .

Install

Install (recommended) or optionally just install the docker engine

You need to separately spin up a ray cluster and enter the base_url (). If base_url is not entered, then a local ray cluster is spun up.

Check out the to get started with code executor. If you have any questions, feel free to join our or .

Open Thoughts
Curator.LLM's caching feature
e2b.dev
Docker Desktop
docker desktop
Installation instructions
examples
Discord
send us an email