Execute LLM-generated code

We have built a code-executor that can be used to execute LLM-generated code. This is useful for many situations:

1. You want to include error-free code in your training code. This method is used in Open Thoughts.

2. LLM generates some code to generate visualization etc.

3. Agents and tool-use.

Here is a simple example of code execution in action:

from bespokelabs import curator
from datasets import Dataset

class HelloExecutor(curator.CodeExecutor):     
	def code(self, row):
		return """location = input();print(f"Hello {location}")"""     
		
	def code_input(self, row):        
		return row['location']     
	
	def code_output(self, row, execution_output):        
		row['output'] = execution_output.stdout 
		return row
		
locations = Dataset.from_list([{'location': 'New York'},{'location': 'Tokyo'}])

hello_executor = HelloExecutor() 

print(hello_executor(locations).to_pandas())

The inherited class contains three methods:

1. code: This is the method that returns the piece of code to be run. This is usually part of the row (you can use curator.LLM to generate this code).

2. code_input: This is optional, but can return a json that represents values to be passed to input() in the code.

3. code_output: This is where you parse the output of the execution.

We offer three methods of running the code:

1. Multiprocessing: This is the default backend (and can be activated as CodeExecutor(backend="multiprocessing") This runs code locally and is therefore the least safe option.

2. Docker: Use CodeExecutor(backend="docker")to use Docker to run the code. Safer option than multiprocessing.

3. Ray: If you have a ray cluster, you can use it by setting CodeExecutor(backend="ray"). This is useful when your code can take a long time to run.

4. E2B: Code can also be run using e2b.dev. Use CodeExecutor(backend="e2b") .