# Welcome ### New to Bespoke Curator or Bespoke-MiniCheck? Start here! ### Bespoke Curator Bespoke Curator makes it very easy to create high-quality synthetic data at scale, which you can use to finetune models or use for structured data extraction at scale. Bespoke Curator is an open-source project: 1. That comes with a rich Python based library for generating and curating synthetic data. 2. A Curator Viewer which makes it easy to view the datasets, thus aiding in the dataset creation. 3. We will also be releasing high-quality datasets that should move the needle on post-training. Start [here](/bespoke-curator/getting-started). ### Using Bespoke-MiniCheck Bespoke-MiniCheck is powered by Bespoke-MiniCheck-7B model, a best-in-class lightweight model that can be used to detect hallucinations. It tops the [LLM-AggreFact leaderboard](https://llm-aggrefact.github.io/)

From LLM-AggreFact Leaderboard

You can use Bespoke-MiniCheck either via: 1. [API service](/models/bespoke-minicheck/api) (easiest), or 2. [Host it yourself](/models/bespoke-minicheck/hosting). # Getting Started Bespoke Curator makes it easy to create synthetic data pipelines. Whether you are training a model or extracting structure, Curator will prepare high-quality data quickly and robustly. * Rich Python based library for generating and curating synthetic data. * Interactive viewer to monitor data while it is being generated * First class support for structured outputs * Built-in performance optimizations for asynchronous operations, caching, and fault recovery at every scale * Support for a wide range of inference options via LiteLLM, vLLM, and popular batch APIs
In addition, we are actively working on improving the library. Expect more changes to come in the future: 1. Verifiers: filter outputs to improve your data quality with models like [Broken mention](broken://pages/Zvukm7MAGL6ooitMTmpY), or with code executors. 2. MCTS: explore reasoning trajectories using Monte Carlo Tree Search. 3. Data versioning: version your data along with the code that generates it. 4. Diversity and data quality indicators: understand the quality of your data. 5. Curator viewer: visualize and explore your generated data. Next, let's take a [Quick Tour](/bespoke-curator/getting-started/quick-tour) of the Curator library! # Quick Tour ## Installation ``` pip install bespokelabs-curator ``` ## Hello World with LLM The `LLM`class provides a flexible interface to generate data with LLMs. Below is a minimal example of using `LLM`: we simply create an `LLM` object with a `model_name`, in this case `gpt-4o-mini`, and passing in a prompt. ```python from bespokelabs import curator llm = curator.LLM(model_name="gpt-4o-mini") poem = llm("Write a poem about the importance of data in AI.") print(poem.to_pandas()) # Output: # response # 0 In the realm where silence once held sway, \n... # Or you can pass a list of prompts to generate multiple responses. poems = llm(["Write a poem about the importance of data in AI.", "Write a haiku about the importance of data in AI."]) print(poems.dataset.to_pandas()) # Output: # response # 0 In the realm where silence once held sway, \n... # 1 Silent streams of truth, \nData shapes the le... ``` ### What's next? * Check out the key concepts of the library in [Key Concepts](/bespoke-curator/getting-started/key-concepts) * See important caching feature avail # Key Concepts ## Key Components of curator.LLM Conceptually, `curator.LLM` has two important methods, `prompt` and `parse`. ```python class Poem(BaseModel): poem: str = Field(description="A poem.") class Poems(BaseModel): poems_list: List[Poem] = Field(description="A list of poems.") class Poet(curator.LLM): response_format = Poems ​ def prompt(self, input: Dict) -> str: return f"Write two poems about {input['topic']}." ​ def parse(self, input: Dict, response: Poems) -> Dict: return [{"topic": input["topic"], "poem": p.poem} for p in response.poems] ``` ### prompt This calls an LLM on each row of the input dataset in parallel. 1. Takes a dataset row as input 2. Returns the prompt for the LLM. ### parse Converts LLM output into structured data by adding it back to the dataset. 1. Takes two arguments: * Input row (this was given to the LLM). * LLM's response (in response\_format --- string or Pydantic) 2. Returns new rows (in list of dictionaries) ### Returns A \`CuratorResponse\` instance which holds information about the run in python object. It consists of dataset, statistics (performance, token usage, cost), viewer link attributes. ## Data Flow Example Input Dataset: ``` Row A Row B ``` Processing by curator.LLM: ``` Row A → prompt(A) → Response R1 → parse(A, R1) → [C, D] Row B → prompt(B) → Response R2 → parse(B, R2) → [E, F] ``` Output Dataset: ``` Row C Row D Row E Row F ``` In this example: * The two input rows (A and B) are processed in parallel to prompt the LLM * Each generates a response (R1 and R2) * The parse function converts each response into (multiple) new rows (C, D, E, F) * The final dataset contains all generated rows You can chain `LLM` objects together to iteratively build up a dataset. # Visualize your dataset with the Bespoke Curator Viewer {% hint style="warning" %} IMPORTANT: We recently are retiring our local `curator-viewer` , now switching to a hosted Bespoke Curator Viewer. {% endhint %} The hosted Bespoke Curator Viewer is a rich interface to visualize data, and makes visually inspecting the data much easier. **Example**: Before your Curator run, set the `CURATOR_VIEWER` environment variable. Bash: ``` export CURATOR_VIEWER=1 ``` Python/colab: ``` import os os.environ["CURATOR_VIEWER"]="1" ``` With this enabled, as curator generates data, it gets uploaded and you can see the responses streaming in the viewer. The URL for the viewer is displayed right next to the rich progress. Then, run on any Bespoke Curator scripts, here we use `examples/poem-generation/poem.py` as an example, and

Example Bespoke Curator Rich CLI logs on synthetic poems

The line ``` Curator Viewer: ✨ Open Curator Viewer ✨ ``` contains a clickable link (the following is an example viewer link, the link changes for each new Curator generation run) which opens the hosted Bespoke Curator Viewer in a new browser tab: *

Example Bespoke Curator Viewer on synthetic poem topics link

#### Authenticate with a Bespoke Labs API key By default, datasets are accessible to anyone with the link. To keep your datasets private, you can associate them with a Bespoke Labs account. Doing so also allows you to: * Track all datasets associated with your account * Share datasets with collaborators * Analyze data generation costs over time You can enable authentication as follows: 1. [Sign up](https://curator.bespokelabs.ai/auth/signup) for a Bespoke Labs account. 2. Create an API key from the [API Key](https://curator.bespokelabs.ai/home/keys) page. {% embed url="" %} 3. Set the `BESPOKE_API_KEY` and `CURATOR_VIEWER` environment variables: ``` export BESPOKE_API_KEY= export CURATOR_VIEWER=1 ``` With these variables set, all your datasets will be streamed to the hosted viewer and linked to your Bespoke Labs account. {% embed url="" %} 4. You can visit the [Datasets](https://curator.bespokelabs.ai/home/datasets) page to see all the datasets generated with your API keys or shared with you by others. {% embed url="" %} 5. You can also visit the [Cost Report](https://curator.bespokelabs.ai/home/costs) page to see the data generation costs for a given period. {% embed url="" %}
# Automatic recovery and caching Curator automatically caches the output generated by the `LLM` class. This is very useful for: * Recovering from failures and interruption: During large data generation runs, you can run into unexpected failures or interruption. Caching partially completed responses from a data generation run allows you to recover from the latest completed output instead of starting from scratch when you restart your run. * Caching previous completed runs: When working with multi-stage pipelines, you might want to reuse earlier stages in the pipeline while iterating on the later stages. Caching previously completed runs from the earlier stages allows you to iterate quickly while saving time & money. To see caching in action, try running the Hello World example below twice. The second run should reuse the cached responses from the first run instead of making an LLM call. ```python from bespokelabs import curator llm = curator.LLM(model_name="gpt-4o-mini") poem = llm("Write a poem about the importance of data in AI.") print(poem.dataset.to_pandas()) ``` ## Disable caching To disable caching, you can simply set `CURATOR_DISABLE_CACHE=1` before generating your data. ## Custom cache directory By default, all cached datasets are saved to `~/.cache/curator`, but you can change it in two ways: 1. Setting the CURATOR\_CACHE\_DIR environmental variable to the desired directory 2. Passing the desired directory to the `working_dir` parameter when applying the `LLM` object on a dataset, e.g. `llm("Write a poem about the importance of data in AI.", working_dir="/path/to/my/poems")`. ## Cache Internals (subject to future changes) The cache directory contains the following: 1. metadata.db**:** a SQLite database containing metadata about data generation runs 2. cache directories of individual data generation runs: each directory is named after the fingerprint of the data generation run. ```bash >> ls ~/.cache/curator 032bc5ead2892f8f 6d1f31229726231d 137851647e75f9a7 91f4ad23d5821c9f 24b1d8917f7ef6f1 a2a3c8e5a58e3fc3 metadata.db ``` The fingerprint of a data generation run is based on the following: 1. The input dataset on which the `LLM` object is being applied. 2. The `prompt` function of the `LLM` object. 3. Whether or not the data generation is using batch mode 4. The response format of the `LLM` object. 5. The model name defined in the `LLM` object. 6. Generation parameters, e.g. temperature, top\_k, etc. ## Troubleshooting ### Corrupt or full working directory The cache directory can get too large or become corrupt due to unexpected errors. You can recover from these types of failures by deleting the cache directory: `rm -rf ~/.cache/curator`. Note that this will delete \*all\* cached responses. # Structured Output ### Structured Output for Data Generation with LLMs This example demonstrates how to use structured output with a custom LLM class to generate poems on different topics while maintaining a clean data structure: ```python from typing import Dict, List from datasets import Dataset from pydantic import BaseModel, Field from bespokelabs import curator # Define our structured output models class Poem(BaseModel): poem: str = Field(description="A poem.") class Poems(BaseModel): poems: List[Poem] = Field(description="A list of poems.") # Create a custom LLM class with specialized prompting and parsing class Poet(curator.LLM): response_format = Poems def prompt(self, input: Dict) -> str: return f"Write two poems about {input['topic']}." def parse(self, input: Dict, response: Poems) -> Dict: return [{"topic": input["topic"], "poem": p.poem} for p in response.poems] # Initialize our custom LLM poet = Poet(model_name="gpt-4o-mini") # Create a dataset of topics topics = Dataset.from_dict({ "topic": [ "Urban loneliness in a bustling city", "Beauty of Bespoke Labs's Curator library" ] }) # Generate poems poem = poet(topics) print(poem.dataset.to_pandas()) # Output: # topic poem # 0 Urban loneliness in a bustling city In the city's heart, where the lights never di... # 1 Urban loneliness in a bustling city Steps echo loudly, pavement slick with rain,\n... # 2 Beauty of Bespoke Labs's Curator library In the heart of Curation's realm,\nWhere art... # 3 Beauty of Bespoke Labs's Curator library Step within the library's embrace,\nA sanctu... ``` ### How This Works: 1. **Structured Models**: We define Pydantic models (`Poem` and `Poems`) that specify the expected structure of our LLM output. 2. **Custom Poet Class**: By inheriting from `curator.LLM`, we create a specialized class that: * Sets `response_format = Poems` to specify the output structure * Implements a `prompt()` method that formats our input into a proper prompt * Implements a `parse()` method that transforms the structured response into a list of dictionaries where each poem is a separate row with its associated topic 3. **Processing Pipeline**: When we call `poet(topics)`, our custom class: * Takes each topic from the dataset * Creates a prompt for each topic * Sends the prompt to the LLM * Parses the structured response * Returns a dataset where each row contains a topic and a single poem This approach gives us clean, structured data that's ready for analysis or further processing while maintaining the relationship between inputs (topics) and outputs (poems). ### Chaining LLM calls with structured output Using structured output along with custom prompting and parsing logic allows you to chain together multiple calls to the `LLM` class to create powerful data generation pipelines. Let's return to our example of generating poems. Suppose we want to also use LLMs to generate the topics of the poems. This can be accomplished by using another `LLM` object to generate the topics, as shown in the example below. ```python from typing import Dict, List from pydantic import BaseModel, Field from bespokelabs import curator class Topic(BaseModel): topic: str = Field(description="A topic.") class Topics(BaseModel): topics: List[Topic] = Field(description="A list of topics.") class Muse(curator.LLM): response_format = Topics def prompt(self, input: Dict) -> str: return "Generate ten evocative poetry topics." def parse(self, input: Dict, response: Topics) -> Dict: return [{"topic": topic.topic} for topic in response.topics] class Poem(BaseModel): poem: str = Field(description="A poem.") class Poems(BaseModel): poems: List[Poem] = Field(description="A list of poems.") class Poet(curator.LLM): response_format = Poems def prompt(self, input: Dict) -> str: return f"Write two poems about {input['topic']}." def parse(self, input: Dict, response: Poems) -> Dict: return [{"topic": input["topic"], "poem": p.poem} for p in response.poems] muse = Muse(model_name="gpt-4o-mini") topics = muse() print(topics.dataset.to_pandas()) poet = Poet(model_name="gpt-4o-mini") poem = poet(topics) print(poem.dataset.to_pandas()) # Output: # topic poem # 0 The fleeting beauty of autumn leaves In a whisper of wind, they dance and they sway... # 1 The fleeting beauty of autumn leaves Once vibrant with life, now a radiant fade,\nC... # 2 The whispers of an abandoned house In shadows deep where light won’t tread, \nAn... # 3 The whispers of an abandoned house Abandoned now, my heart does fade, \nOnce a h... # 4 The warmth of a forgotten summer day In the stillness of a memory's embrace, \nA w... # 5 The warmth of a forgotten summer day A gentle breeze delivers the trace \nOf a day... # ... ``` Chaining multiple `LLM`calls this way allows us to build powerful synthetic data pipelines that can create millions of examples. # Save $$$ on LLM inference Providers like OpenAI and Anthropic offer batch mode, which allows you to upload a bunch of prompts to be processed asynchronously, for lower costs (typically 50%). However, these APIs are often very cumbersome to manage: * You typically have to prepare your batch file, upload it, and poll for responses periodically. * Large datasets will typically not fit in a single batch due to batch size limits, and so you will need to split your dataset into mutiple smaller batches, increasing the complexity you need to manage. With Curator, you only need to toggle a single flag to save $$$, without any headache! ## Using batch mode Let's look at a simple example of reannotating instructions from the [WildChat](https://huggingface.co/datasets/allenai/WildChat) dataset with new responses from gpt-4o-mini. First, we need to load the WildChat dataset using HuggingFace: ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(3_000)) # Select a subset of 3,000 samples ``` We then create a new `LLM` class and apply to `dataset`. All you need to do to enable batching is setting `batch=True` when initializing your `LLM` object, and you're done! ```python from bespokelabs import curator class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format.""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} # Initialize the reannotator with batch processing reannotator = WildChatReannotator( model_name="gpt-4o-mini", batch=True, # Enable batch processing backend_params={"batch_size": 1_000}, # Specify batch size ) reannotated_dataset = reannotator(dataset).dataset ``` ## Supported Models Check out how-to guides for using batch mode with our supported providers: * [Using OpenAI for batch inference](/bespoke-curator/save-usdusdusd-on-llm-inference/using-openai-for-batch-inference) * [Using Anthropic for batch inference](/bespoke-curator/save-usdusdusd-on-llm-inference/using-anthropic-for-batch-inference) * [Using Gemini for batch inference](/bespoke-curator/save-usdusdusd-on-llm-inference/using-gemini-for-batch-inference) * [Using kluster.ai for batch inference](/bespoke-curator/save-usdusdusd-on-llm-inference/using-kluster.ai-for-batch-inference) Feel free to tell us which providers you want us to add support for, or send a PR if you want to [contribute](https://github.com/bespokelabsai/curator/blob/main/CONTRIBUTING.md)! # Using OpenAI for batch inference You can use **OpenAI** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **OpenAI:** OpenAI API key ## **Steps** #### **1. Setup environment vars** ```sh export OPENAI_API_KEY= ``` **2. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} ``` #### **3. Configure the OpenAI model** ```python distiller = WildChatReannotator(model_name="gpt-4o-mini", batch=True ) ``` #### **4. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset.dataset) print(distilled_dataset.dataset[0]) ``` ### **Example Output** Using the above example, the output might look like this: | instruction | new\_response | | ------------------------------------------------- | ----------------------------------------------- | | Write a very long, elaborate, descriptive and ... | Scene: Omelette Apocalypse\n\n\*\*INT. DINER... | | what are you? | I am a large language model, trained by OpenAI | ## **Batch Configuration** * Check out complete [batch configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) # Using Anthropic for batch inference You can use **Anthropic** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Anthropic:** Anthropic API key ## **Steps** #### **1. Setup environment vars** ```sh export ANTHROPIC_API_KEY= ``` **2. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} ``` #### **3. Configure the Anthropic model** ```python distiller = WildChatReannotator(model_name="claude-3-5-haiku-20241022", batch=True) ``` #### **4. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset.dataset) print(distilled_dataset.dataset[0]) ``` ### **Example Output** Using the above example, the output might look like this: | instruction | new\_response | | ------------------------------------------------- | ------------------------------------------------- | | Write a very long, elaborate, descriptive and ... | Scene: Omelette Apocalypse\n\n\*\*INT. DINER... | | what are you? | I am a large language model, trained by Anthropic | ## **Batch Configuration** * Check out complete [batch configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) # Using Gemini for batch inference You can use **Gemini** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Gemini (Vertex AI):** GCP account with Vertex AI enabled. * **Google Cloud Bucket**: Access to cloud storage. ## **Steps** #### **1. Setup environment vars** ```sh export GOOGLE_CLOUD_REGION=us-central1 export GOOGLE_CLOUD_PROJECT= export GEMINI_BUCKET_NAME= export GEMINI_API_KEY= ``` #### **2. ADC authentication** ```sh gcloud auth application-default login ``` #### **3. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} ``` #### **3. Configure the Gemini Backend** ```python distiller = WildChatReannotator(model_name="gemini-1.5-flash-002", backend="gemini", batch=True ) ``` #### **4. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset.dataset) print(distilled_dataset.dataset[0]) ``` ### **Example Output** Using the above example, the output might look like this: | instruction | new\_response | | ------------------------------------------------- | ----------------------------------------------- | | Write a very long, elaborate, descriptive and ... | Scene: Omelette Apocalypse\n\n\*\*INT. DINER... | | what are you? | I am a large language model, trained by Google | ## **Gemini Batch Configuration** * Check out complete [batch configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) * Check out Gemini [generation parameters](https://cloud.google.com/vertex-ai/generative-ai/docs/reference/python/latest/vertexai.generative_models.GenerationConfig) # Using Mistral for batch inference You can use **Mistral** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Mistral:** Mistral API key ## **Steps** #### **1. Setup environment vars** ```sh export MISTRAL_API_KEY= ``` **2. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} ``` #### **3. Configure the Anthropic model** ```python distiller = WildChatReannotator(model_name="mistral-tiny", batch=True) ``` #### **4. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset.dataset) print(distilled_dataset.dataset[0]) ``` ### **Example Output** Using the above example, the output might look like this: | instruction | new\_response | | ------------------------------------------------- | ----------------------------------------------- | | Write a very long, elaborate, descriptive and ... | Scene: Omelette Apocalypse\n\n\*\*INT. DINER... | | what are you? | I am a large language model, trained by Mistral | ## **Batch Configuration** * Check out complete [batch configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) # Using kluster.ai for batch inference You can use **kluster.ai** for batch inference in **Curator** to generate synthetic data. In this example, we will generate answers for GSM8K dataset, but the approach can be adapted for any data generation task. The following models are supported with pricing for different completion windows:
Model IDRealtime24h48h72h
meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8$0.20/$0.80$0.25$0.20$0.15
meta-llama/Llama-4-Scout-17B-16E-Instruct$0.08/$0.45$0.15$0.12$0.10
deepseek-ai/DeepSeek-V3-0324$0.70/$1.40$0.63$0.50$0.35
google/gemma-3-27b-it$0.35$0.30$0.25$0.20
deepseek-ai/DeepSeek-V3$1.25$0.63$0.50$0.35
deepseek-ai/DeepSeek-R1$3.00/$5.00$3.50$3.00$2.50
Qwen/Qwen2.5-VL-7B-Instruct$0.30$0.15$0.10$0.05
klusterai/Meta-Llama-3.1-405B-Instruct-Turbo$3.50$0.99$0.89$0.79
klusterai/Meta-Llama-3.3-70B-Instruct-Turbo$0.70$0.20$0.18$0.15
klusterai/Meta-Llama-3.1-8B-Instruct-Turbo$0.18$0.05$0.04$0.03
*Note: Prices shown as $ per 1M tokens. For Realtime, some models have different input/output prices shown as input/output. Please find the up to date map here:* ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **kluster.ai API key:** Get your key from ## **Steps** #### **1. Setup environment vars** ```sh export KLUSTERAI_API_KEY= ``` **2. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class Reasoner(curator.LLM): """Curator class for processing GSM8K dataset.""" def prompt(self, input): """Create a prompt for the LLM to reason about the problem.""" return f"Answer the following question: {input['question']}" def parse(self, input, response): """Parse the LLM response to extract reasoning and solution. The response format is expected to be 'reasoninganswer' """ full_response = response # Extract reasoning and answer using regex import re reasoning_pattern = r"(.*?)" reasoning_match = re.search(reasoning_pattern, full_response, re.DOTALL) reasoning = reasoning_match.group(1).strip() if reasoning_match else "" # Answer is everything after answer = re.sub(reasoning_pattern, "", full_response, flags=re.DOTALL).strip() return [ { "question": input["question"], "reasoning": reasoning, "deepseek_solution": answer, "gold_answer": input["answer"], } ] ``` #### **3. Configure Reasoner to use DeepSeek-R1 through kluster.ai** ```python reasoner = Reasoner(model_name="deepseek-ai/DeepSeek-R1", backend="klusterai", batch=True, backend_params={"max_retries": 1, "completion_window": "1h"}) ``` #### **4 Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python from datasets import load_dataset dataset = load_dataset("openai/gsm8k", name="main") dataset_to_use = dataset["train"].take(3) output = reasoner(dataset).dataset ``` ### **Example Output** Using the above example, the output might look like this: ```python from IPython.display import HTML, display, Markdown which = 0 question = output[which]['question'] gold_answer = output[which]['gold_answer'] model_answer = output[which]['deepseek_solution'] thought = output[which]['reasoning'] to_display_input = question.replace("\n", "
") to_display_output = model_answer.replace("\n", "
") display(Markdown( "

Question

" f"

{question}

" )) display(Markdown( "

Model answer

" f"

{model_answer}

" )) display(Markdown( "

Gold answer

" f"

{gold_answer}

" )) display(Markdown( "

Model Thought

" f"

{thought}

" )) ```
## **Batch Configuration** * Check out complete [batch configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) # How-to Guides This section contains various How-to Guides showcasing some of the features of Curator. Following are the available guides: * Curator with various backends: * [Using vLLM with Curator](/bespoke-curator/how-to-guides/using-vllm-with-curator) * [Using Ollama with Curator](/bespoke-curator/how-to-guides/using-ollama-with-curator) * [Using LiteLLM with curator](/bespoke-curator/how-to-guides/using-litellm-with-curator) * Multimodal Guide: * [Handling Multimodal Data in Curator](/bespoke-curator/how-to-guides/handling-multimodal-data-in-curator) * Code Execution: * [Executing LLM-generated code](/bespoke-curator/how-to-guides/executing-llm-generated-code) # Online Processing This guide demonstrates how to use **Curator** for **online processing** by generating diverse topics for poems and composing poems based on these topics. We'll focus on key parameters to manage rate limits effectively, ensuring smooth operation with LLMs. ## Prerequisites * Python 3.10+ * Curator: Install via `pip install bespokelabs-curator` * Access to an LLM provider (e.g., OpenAI or an equivalent API) ## Steps ### 1. Define the Response Formats **Code Example:** ```python from typing import List from pydantic import BaseModel, Field class Topics(BaseModel): """A list of topics.""" topics_list: List[str] = Field(description="A list of topics.") class Poems(BaseModel): """A list of poems.""" poems_list: List[str] = Field(description="A list of poems.") ``` *** ### 2. Create a Muse Define a subclass of `curator.LLM` to generate a list of topics for our poems. **Code Example:** ```python from bespokelabs import curator class Muse(curator.LLM): response_format = Topics def prompt(self, input: dict) -> str: return "Generate 10 diverse topics that are suitable for writing poems about." def parse(self, input: dict, response: Topics) -> dict: return [{"topic": t} for t in response.topics_list] # Instantiate the Topic Generator topic_generator = TopicGenerator(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100}) # Generate topics from datasets import Dataset topics: Dataset = topic_generator() print(topics["topic"]) # View the generated topics ``` *** ### 3. Create a Poet Define another subclass of `curator.LLM` to create poems based on the generated topics. **Code Example:** ```python class Poet(curator.LLM): """A poet that generates poems about given topics.""" response_format = Poems def prompt(self, input: dict) -> str: return f"Write two poems about {input['topic']}." def parse(self, input: dict, response: Poems) -> dict: return [{"topic": input["topic"], "poem": p} for p in response.poems_list] # Instantiate the Poet poet = Poet(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100}) # Generate poems based on topics poems = poet(topics) print(poems.to_pandas()) # View the poems in a tabular format ``` *** ### Example Output ```plaintext topic poem 0 Dreams vs. reality In the realm where dreams take flight,\nWhere ... 1 Dreams vs. reality Reality stands with open eyes,\nA weighty thro... 2 Urban loneliness in a bustling city In the city's heart where shadows blend,\nAmon... 3 Urban loneliness in a bustling city Among the crowds, I walk alone,\nA sea of face... ``` *** ## Online Processing Configuration Curator provides several configuration parameters to better control your API usage policies and improve efficiency. #### 1. **`max_requests_per_minute`** **Definition**: The maximum number of API requests allowed per minute. * **Use Case**: Controls the frequency of requests to prevent exceeding API quotas. * **Example**: Setting this to `60` ensures only 60 requests are sent per minute. **Example**: ```python backend_params={ "max_requests_per_minute": 60 # Sets maximum requests in a minute. } ``` **Note**: Retries will also count under the limit set. #### 2. **`max_tokens_per_minute`** **Definition**: The maximum number of tokens allowed to be processed per minute. * **Use Case**: Defines contraint on maximum usable tokens in a minute. * **Example**: Setting this to `30,000` ensures token limits are respected for responses. **Example**: ```python backend_params={ "max_tokens_per_minute": 1_000_000 # Sets maximum tokens in a minute. } ``` **Note**: Token limit will be counted including input and output tokens. #### 3. **`seconds_to_pause_on_rate_limit`** **Definition**: The duration (in seconds) to pause when a rate limit is hit. * **Use Case**: Automatically waits before retrying, ensuring compliance with API limits. * **Example**: Setting this to `60` pauses the processing for one minute if a rate limit error occurs. **Example**: ```python backend_params={ "seconds_to_pause_on_rate_limit": 60 # Sets seconds to wait before new request on rate limit. } ``` # Batch Processing This guide demonstrates how to use **Curator** for batch processing, specifically reannotating datasets. We'll walk through an example using the **WildChat** dataset to create new responses for its conversations. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * Access to an LLM provider (e.g., OpenAI or equivalent API) ## **Steps** ### **1. Load and Prepare the Dataset** Use the **Hugging Face Datasets** library to load the WildChat dataset and select a subset for reannotation. ```python from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(3_000)) # Select a subset of 3,000 samples ``` ### **2. Create a Curator.LLM Subclass** Define a subclass of `curator.LLM` to handle prompt generation and parsing. The subclass defines how the model processes inputs and outputs. ```python from bespokelabs import curator class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format.""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} ``` ### **3. Configurer Batch Processing** Set up the LLM reannotator with batch processing enabled. Specify the `batch_size` parameter to determine the number of samples processed in each batch. ```python import logging # Enable detailed logging for batch processing logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) # Initialize the reannotator with batch processing distiller = WildChatReannotator( model_name="gpt-4o-mini", batch=True, # Enable batch processing backend_params={"batch_size": 1_000}, # Specify batch size ) ``` ### **4. Process the Dataset** Run the distiller on the dataset to generate new annotations. ```python distilled_dataset = distiller(dataset) ``` ### **5. Inspect the Results** Print the distilled dataset to verify the new annotations. ```python print(distilled_dataset) print(distilled_dataset[0]) ``` Example Output: ```python {'instruction': 'What is the capital of France?', 'new_response': 'The capital of France is Paris.'} ``` ### Batch Processing Configuration: ### **1. Supported models** **`model_name`** Currently, we only support batch mode for Anthropic and OpenAI models. You can change the model by setting the `model_name` argument in the `LLM` constructor. ### **2. Batch Size** **`batch_size`** * **Description**: Maximum number of requests to process in a single batch. * **Best Practice**: * For large datasets, choose a value that balances efficiency and memory usage. * For LLM APIs refer the number of requests per batch allowed and set this accordingly. **Example**: ```python backend_params={ "batch_size": 1_000 # Process 1,000 requests per batch } ``` ### **3. Batch Check Interval** **`batch_check_interval`** * **Description**: Time in seconds between status checks for active batches. * **Best Practice**: * Set a low value (e.g., 5–10 seconds) for near real-time monitoring. * Increase the interval for long-running jobs to reduce overhead. **Example**: ```python backend_params={ "batch_check_interval": 10 # Check batch status every 10 seconds } ``` ### **4. Delete Successful Batch Files** **`delete_successful_batch_files`** * **Description**: Whether to delete batch files after successful processing to save storage. * **Best Practice**: * Enable (`True`) for production or disk-constrained environments. * Keep (`False`) for debugging or when an audit trail is needed. **Example**: ```python backend_params={ "delete_successful_batch_files": True # Automatically delete successful batch files } ``` ### **5. Delete Failed Batch Files** **`delete_failed_batch_files`** * **Description**: Whether to delete batch files after failed processing to free up space. * **Best Practice**: * Enable (`True`) if the failures are logged elsewhere or can be regenerated. * Disable (`False`) when debugging or troubleshooting errors. **Example**: ```python backend_params={ "delete_failed_batch_files": False # Retain failed batch files for debugging } ``` ### **Example Configuration** Below is an example combining all the options for optimized batch processing: ```python backend_params={ "batch_size": 1_000, # Process 1,000 requests per batch "batch_check_interval": 10, # Check batch status every 10 seconds "delete_successful_batch_files": True, # Delete files after successful processing "delete_failed_batch_files": False # Retain files after failed processing } ``` # Using vLLM with Curator You can use VLLM as a backend for Curator in two modes: offline (local) and online (server). This guide demonstrates both approaches using structured recipe generation as an example. ## Prerequisites * Python 3.10+ * Curator: Install via `pip install bespokelabs-curator` * VLLM: Install via `pip install vllm` ## Offline Mode (Local) In offline mode, VLLM runs locally on your machine, loading the model directly into memory. ### 1. Create Pydantic Models for Structured Output First, define your data structure using Pydantic models: ```python from pydantic import BaseModel, Field from typing import List class Recipe(BaseModel): title: str = Field(description="Title of the recipe") ingredients: List[str] = Field(description="List of ingredients needed") instructions: List[str] = Field(description="Step by step cooking instructions") prep_time: int = Field(description="Preparation time in minutes") cook_time: int = Field(description="Cooking time in minutes") servings: int = Field(description="Number of servings") ``` ### 2. Create a Curator LLM Subclass Create a class that inherits from `LLM` and implement two key methods: 
from bespokelabs import curator

class RecipeGenerator(curator.LLM): 
    response_format = Recipe
    
    def prompt(self, input: dict) -> str:
        return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic."
    
    def parse(self, input: dict, response: Recipe) -> dict:
        return {
            "title": response.title,
            "ingredients": response.ingredients,
            "instructions": response.instructions,
            "prep_time": response.prep_time,
            "cook_time": response.cook_time,
            "servings": response.servings,
        }
#### 3. Initialize and Use the Generator ```python # Initialize with a local model generator = RecipeGenerator( model_name="Qwen/Qwen2.5-3B-Instruct", backend="vllm", backend_params={ "tensor_parallel_size": 1, # Adjust based on GPU count "gpu_memory_utilization": 0.7 } ) # Create input dataset cuisines = [{"cuisine": c} for c in ["Italian", "Chinese", "Mexican"]] recipes = generator(cuisines) print(recipes.dataset.to_pandas()) ``` ## Online Mode (Server) In online mode, VLLM runs as a server that can handle multiple requests. ### 1. Start the VLLM Server Start the VLLM server with your chosen model: 
vllm serve Qwen/Qwen2.5-3B-Instruct \
    --host localhost \
    --port 8787 \
    --api-key token-abc123
### 2. Configure the Generator Use the same Pydantic models and LLM subclass as in offline mode, but initialize with server configuration: 
# Set API key if required
os.environ["HOSTED_VLLM_API_KEY"] = "token-abc123"

# Initialize with server connection
generator = RecipeGenerator( 
    model_name="hosted_vllm/Qwen/Qwen2.5-3B-Instruct", 
    backend="litellm", 
    backend_params={ 
        "base_url": "http://localhost:8787/v1", 
        "request_timeout": 30 
    } 
)

# Generate recipes
recipes = generator(cuisines)
print(recipes.dataset.to_pandas())
## Example Output The generated recipes will be returned as structured data like: ```json { "title": "Spicy Szechuan Noodles", "ingredients": [ "400g wheat noodles", "2 tbsp Szechuan peppercorns", "3 cloves garlic, minced", "2 tbsp soy sauce" ], "instructions": [ "Boil noodles according to package instructions", "Heat oil in a wok over medium-high heat", "Add peppercorns and garlic, stir-fry until fragrant", "Add noodles and soy sauce, toss to combine" ], "prep_time": 15, "cook_time": 20, "servings": 4 } ``` ## VLLM Offline Configuration #### Backend Parameters (for Offline Mode) * `tensor_parallel_size`: Number of GPUs for tensor parallelism (default: 1) * `gpu_memory_utilization`: GPU memory usage fraction between 0 and 1 (default: 0.95) * `max_model_length`: Maximum sequence length (default: 4096) * `max_tokens`: Maximum number of tokens to generate (default: 4096) * `min_tokens`: Minimum number of tokens to generate (default: 1) * `enforce_eager`: Whether to enforce eager execution (default: False) * `batch_size`: Size of batches for processing (default: 256) # Using Ollama with Curator You can use **Ollama** as a backend for **Curator** to generate structured synthetic data. In this example, we will generate a list of countries and their capitals, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Ollama:** Download via ## **Steps** #### **1. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python from bespokelabs import curator from pydantic import BaseModel, Field class Location(BaseModel): country: str = Field(description="The name of the country") capital: str = Field(description="The name of the capital city") class LocationList(BaseModel): locations: list[Location] = Field(description="A list of locations") class SimpleOllamaGenerator(curator.LLM): response_format = LocationList def prompt(self, input: dict) -> str: return "Return five countries and their capitals." def parse(self, input: dict, response: str) -> dict: return [{"country": output.country, "capital": output.capital} for output in response.locations] ``` ### **2. Configure the Ollama Backend** 1. Start Ollama server with `llama3.1:8b`model. ```bash ollama pull llama3.1:8b ollama serve ``` 2. Initialize your generator with Ollama configuration: ```python llm = SimpleOllamaGenerator( model_name="ollama/llama3.1:8b", # Ollama model identifier backend_params={"base_url": "http://localhost:11434"}, # Ollama instance ) ``` ### **3. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python locations = llm() print(locations.dataset.to_pandas()) ``` ### **Example Output** Using the above example, the output might look like this: | Country | Capital | | ------- | --------- | | France | Paris | | Japan | Tokyo | | Germany | Berlin | | India | New Delhi | | Brazil | Brasília | ## **Ollama Configuration** Use `base_url` in the `backend_params` to specify the connection URL. Example: ```python backend_params={"base_url": "http://localhost:11434"} ``` # Using LiteLLM with curator This guide demonstrates how to use LiteLLM as a backend for curator to generate synthetic data using various LLM providers. We'll walk through an example of generating synthetic recipes, but this approach can be adapted for any synthetic data generation task. ## Prerequisites * Python 3.10+ * Curator (`pip install bespokelabs-curator`) * Access to an LLM provider (e.g., Gemini API key) ## Steps ### 1. Create a curator.LLM Subclass First, create a class that inherits from `curator.LLM`. You'll need to implement two key methods: * `prompt()`: Generates the prompt for the LLM * `parse()`: Processes the LLM's response into your desired format ```python """Generate synthetic recipes for different cuisines using curator.""" from datasets import Dataset from bespokelabs import curator class RecipeGenerator(curator.LLM): """A recipe generator that generates recipes for different cuisines.""" def prompt(self, input: dict) -> str: """Generate a prompt using the template and cuisine.""" return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic." def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return { "recipe": response, "cuisine": input["cuisine"], } ``` ### 2. Set Up Your Seed Dataset Create a dataset of inputs using the HuggingFace `Dataset` class: ```python # List of cuisines to generate recipes for cuisines = [ {"cuisine": cuisine} for cuisine in [ "Chinese", "Italian", "Mexican", "French", "Japanese", "Indian", "Thai", "Korean", "Vietnamese", "Brazilian", ] ] cuisines = Dataset.from_list(cuisines) ``` ### 3. Configure LiteLLM Backend Initialise your generator with LiteLLM configuration: ```python recipe_generator = RecipeGenerator( model_name="gemini/gemini-1.5-flash", # LiteLLM model identifier backend="litellm", # Specify LiteLLM backend backend_params={ "max_requests_per_minute": 2_000, # Rate limit for requests "max_tokens_per_minute": 4_000_000 # Token usage limit }, ) ``` ### 4. Generate Data Generate your synthetic data: ```python recipes = recipe_generator(cuisines) print(recipes.dataset.to_pandas()) ``` ## LiteLLM Configuration ### API Keys and Environment Variables For Gemini: ```bash export GEMINI_API_KEY='your-api-key-here' # Get from https://aistudio.google.com/app/apikey ``` ## Curator Configuration ### Rate Limits Configure rate limit with backend parameters: ```python # Custom RPM/TPM configuration # By default, this is set to: # - max_requests_per_minute: 10 # - max_tokens_per_minute: 100_000 backend_params={ "max_requests_per_minute": 2_000, # 2K requests/minute "max_tokens_per_minute": 4_000_000 # 4M tokens/minute } ``` ## Providers and Models Here are a list of providers. This is not an exhaustive list. Please refer to the [litellm provider documentation](https://docs.litellm.ai/docs/providers). ### Together ``` export TOGETHER_API_KEY='your-api-key-here' ``` ```python recipe_generator = RecipeGenerator( model_name="together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo", backend="litellm", ) ``` Other common models ([more info](< https://api.together.ai/models?filter=serverless.>)): ```python together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo together_ai/meta-llama/Llama-3.3-70B-Instruct-Turbo together_ai/Qwen/Qwen2.5-7B-Instruct-Turbo # doesn't support structured outputs ``` ### DeepInfra ``` export DEEPINFRA_API_KEY='your-api-key-here' ``` ```python recipe_generator = RecipeGenerator( model_name="deepinfra/deepseek-ai/DeepSeek-R1-Turbo", backend="litellm", ) ``` Other common models ([all models](https://deepinfra.com/models) — use prefix deepinfra): ``` deepinfra/meta-llama/Llama-3.3-70B-Instruct deepinfra/meta-llama/Llama-3.3-70B-Instruct-Turbo deepinfra/Qwen/Qwen2.5-72B-Instruct ``` # Handling Multimodal Data in Curator You can easily run multimodal synthetic data generation using curator. This guide shows you how to define and run multimodal synthetic data generation using curator in three easy steps. Step 1: Creating a multimodal dataset ```python from datasets import Dataset ingredients = [ {"spice_level": ingredient[0], "image_url": ingredient[1]} for ingredient in [ ("very spicy", "https://cdn.tasteatlas.com//images/ingredients/fcee541cd2354ed8b68b50d1aa1acad8.jpeg"), ("not so spicy", "https://cdn.tasteatlas.com//images/dishes/da5fd425608f48b09555f5257a8d3a86.jpg"), ] ] ingredients = Dataset.from_list(ingredients) ``` Step 2: Writing the Curator Block ```python from bespokelabs import curator class RecipeGenerator(curator.LLM): """A recipe generator that generates recipes for different ingredient images.""" def prompt(self, input: dict) -> str: """Generate a prompt using the ingredients.""" prompt = f"Create me a {input['spice_level']} recipe from the ingredients image." return prompt, curator.types.Image(url=input["image_url"]) def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return { "recipe": response, } ``` Step 3: Running the curator block using gpt-4o-mini on the ingredients dataset 
recipe_generator = RecipeGenerator(
    model_name="gpt-4o-mini",
    backend="openai",
)

# Generate recipes for all ingredients
recipes = recipe_generator(ingredients)

# Print results
print(recipes.dataset.to_pandas())
# Executing LLM-generated code We have built a code-executor that can be used to execute LLM-generated code. This is useful for many situations: 1. You want to include error-free code in your training code. This method is used in [Open Thoughts](https://open-thoughts.ai). 2. LLM generates some code to generate visualization etc. 3. Agents and tool-use. Here is a simple example of code execution in action: ```python from bespokelabs import curator from datasets import Dataset class HelloExecutor(curator.CodeExecutor): def code(self, row): return """location = input();print(f"Hello {location}")""" def code_input(self, row): return row['location'] def code_output(self, row, execution_output): row['output'] = execution_output.stdout return row locations = Dataset.from_list([{'location': 'New York'},{'location': 'Tokyo'}]) hello_executor = HelloExecutor() print(hello_executor(locations).to_pandas()) ``` The inherited class contains three methods: 1. `code`: This is the method that returns the piece of code to be run. This is usually part of the row (you can use `curator.LLM` to generate this code). 2. `code_input`: This is optional, but can return a json that represents values to be passed to `input()` in the code. 3. `code_output`: This is where you parse the output of the execution. ### Features: 1. Full caching and automatic recovery: Similar to [Curator.LLM's caching feature](https://docs.bespokelabs.ai/bespoke-curator/tutorials/automatic-recovery-and-caching), code executor also has inbuilt caching and automatic recovery. Any interrupted runs can be fully recovered and no computation is lost. 2. Multiple code execution backends: We offer **four** backends: multiprocessing, docker, ray and E2B. These backends specify different locations where your code can be executed. You can easily switch the backend with a simple parameter change. For example, the hello world example can be run using the ray backend by simply initializing it with \``HelloExecutor(backend=ray)`\` 3. Progress monitoring using Rich Console:
### Backends We offer **four** backends for running your code: 1. Multiprocessing: This is the default backend. This runs code locally and is therefore the least safe option, but is useful for quick execution as it does not require any dependencies. 2. Docker: It is safer option than multiprocessing. 3. Ray: If you have a ray cluster, you can use it by setting `CodeExecutor(backend="ray")`. This is useful when your code can take a long time to run. 4. E2B: Code can also be run using [e2b.dev](https://e2b.dev/). Use `CodeExecutor(backend="e2b")` . ### Backend Setup and configuration options #### Multiprocessing Backend: This doesn't require any additional setup. You can configure `backend params` while initializing as follows: 
```python
hello_executor = HelloExecutor(
    backend_params = {    
        "max_requests_per_minute": 1000
    }
)
```
You can also configure execution parameters: ```` ```python output = hello_executor(dataset, execution_params = { "timeout": 120 # in seconds "memory_limit": 1024 ** 3 } ) ``` ```` #### Docker With docker, code can be executed in a secure containerized environment. You need docker installed and python's docker client installed on your machine: {% tabs %} {% tab title="Mac" %} 1. pip install docker 2. Install [Docker Desktop](https://www.docker.com/products/docker-desktop/) 3. In your terminal, run \`docker pull python:3.11-slim\` 4. Run the HelloExecutor example with HelloExecutor(backend=docker) {% endtab %} {% tab title="Linux" %} 1. pip install docker 2. Install [docker desktop](https://docs.docker.com/desktop/setup/install/linux/) (recommended) or optionally just install the docker engine 3. In your terminal, run \`docker pull python:3.11-slim\` 4. Run the HelloExecutor example with HelloExecutor(backend=docker) {% endtab %} {% endtabs %} With docker, you can specify a custom docker image to execute your code snippets: 
```python
hello_executor = HelloExecutor(
    backend = "docker",
    backend_params = {    
        "image": "andgineer/matplotlib"
    }
)
```
#### Ray As the size of the dataset grows, it becomes harder to scale code execution requirements on a single machine. In such scenarios, one can use the ray backend. Simply run `pip install ray` to install the dependencies required for ray backend. You need to separately spin up a ray cluster and enter the base\_url ([Installation instructions](https://docs.ray.io/en/latest/ray-core/starting-ray.html)). If base\_url is not entered, then a local ray cluster is spun up. 
```python
hello_executor = HelloExecutor(
    backend = "ray",
    backend_params = {    
        "base_url": "<url of ray cluster>"
    }
)
```
#### E2B We also add light support for e2b's hosted code execution backends. While not free, they are secure environments similar to docker environments and have more features. 1. Run `pip install e2b-code-interpreter` to install the required dependencies. 2. Create an account on e2b's website, get the API key and add it to your environment variables. 
```python
hello_executor = HelloExecutor(
    backend = "e2b",
)
```
### Conclusion Check out the [examples](https://github.com/bespokelabsai/curator/tree/main/examples/code-execution) to get started with code executor. If you have any questions, feel free to join our [Discord](https://discord.com/invite/KqpXvpzVBS) or [send us an email](mailto:company@bespokelabs.ai). # Using HuggingFace inference providers with Curator This guide demonstrates how to use Hugging Face Inference Providers with curator to generate synthetic data using various LLM providers available as Inference Providers on Hugging Face. We’ll walk through an example of generating synthetic recipes, but this approach can be adapted for any synthetic data generation task. ### What is Inference Providers Hugging Face’s [Inference Providers](https://huggingface.co/docs/inference-providers) give developers streamlined, unified access to hundreds of machine learning models, powered by Hugging Face’s serverless inference partners. Using Inference Providers gives you access to a wide range of state of the art models, with newly released models regularly added by providers. Since Inference Providers are available using OpenAI compatible APIs, you can use them as a drop in replacement for OpenAI in your project. This is the approach we’ll take in this guide. ### Setup 1. Ensure you have a Hugging Face account, you can sign up [here](https://huggingface.co/join). 2. Create a Hugging Face API key, you can create one [here](https://huggingface.co/settings/tokens). This will be used to authenticate your requests to the Inference Providers. 3. Ensure you have at least one Inference Provider enabled in your Hugging Face account. You can configure this in the [Inference Providers](https://huggingface.co/settings/inference-providers) page. First, install the necessary packages: ``` pip install bespokelabs-curator pip install huggingface_hub openai ``` ### Steps #### 1. Create a curator.LLM Subclass First, create a class that inherits from curator.LLM. You’ll need to implement two key methods: * `prompt()`: Generates the prompt for the LLM * `parse()`: Processes the LLM’s response into your desired format ```python """Generate synthetic recipes for different cuisines using curator.""" from datasets import Dataset from bespokelabs import curator class RecipeGenerator(curator.LLM): """A recipe generator that generates recipes for different cuisines.""" def prompt(self, input: dict) -> str: """Generate a prompt using the template and cuisine.""" return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic and make the recipe vegan" def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format.""" return { "recipe": response, "cuisine": input["cuisine"], } ``` #### 2. Set Up Your Seed Dataset Create a dataset of inputs using the HuggingFace Dataset class: ```python # List of cuisines to generate recipes for cuisines = [ {"cuisine": cuisine} for cuisine in [ "Chinese", "Italian", "Mexican", "French", "Japanese", "Indian", "Thai", "Korean", "Vietnamese", "Brazilian", ] ] cuisines = Dataset.from_list(cuisines) ``` #### 3. Configure the OpenAI Backend to use an Inference Provider Since Inference Providers are available using OpenAI compatible APIs, we can use them as a drop in replacement for OpenAI in our project. We just need to configure a few things: * the `base_url` * the `api_key` * the `model_name` You can find this infromation on the model page of the Inference Provider. For example, here’s the information for the Together Inference Provider: ```python recipe_generator = RecipeGenerator( model_name="meta-llama/Llama-4-Scout-17B-16E-Instruct", backend="openai", backend_params={ "base_url": "https://router.huggingface.co/together/v1", "api_key": HF_TOKEN, # your Hugging Face API key }, ) results = recipe_generator(cuisines).dataset print(results.to_pandas()) ``` #### 4. Push to Hub You can view the dataset in Curator Viewer or HuggingFace hub Curator Viewer ```python # We can visualize data using Curator viewer easily. import os os.environ['CURATOR_VIEWER']='1' from bespokelabs.curator.utils import push_to_viewer url = push_to_viewer(results) ``` OR since curator is using 🤗 datasets, you can push the results to the Hub just like any other dataset! ```python results.push_to_hub( "hf_username/llama-recipes", private=False, token=HF_TOKEN, ) ``` You can see what the dataset looks like on the Hub [here](https://huggingface.co/datasets/davanstrien/llama-recipes)! ### Tips Since Inference Providers are available via a standard protocol, it’s easy to swap out models on providers in your pipeline depending on the needs of your project. You can find a list of models that are available via a specific Inference Provider by using a Inference Provider filter on the Hub. For example, [here’s the list of models](< https://huggingface.co/models?inference_provider=together\&sort=trending>) that are available via the Together Inference Provider. # Data Curation Recipes Here are some simple data curation recipes to get you started with generating synthetic data at scale using Curator: * [Generating a diverse QA dataset](/bespoke-curator/data-curation-recipes/generating-a-diverse-qa-dataset) * [Curate Reasoning data with Claude-3.7 Sonnet](/bespoke-curator/data-curation-recipes/curate-reasoning-data-with-claude-3.7-sonnet) * [Using SimpleStrat block for generating diverse data](/bespoke-curator/data-curation-recipes/using-simplestrat-block-for-generating-diverse-data) * [Using SimpleStrat block for generating diverse data](/bespoke-curator/data-curation-recipes/using-simplestrat-block-for-generating-diverse-data) In addition to these examples, we also have the following larger examples in our github repo: | **Task** | **Link(s)** | **Goal** | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | **Reasoning dataset generation (Bespoke Stratos)** | [Code](https://github.com/bespokelabsai/curator/tree/main/examples/bespoke-stratos-data-generation) | Generate the Bespoke-Stratos-17k dataset, focusing on reasoning traces from math, coding, and problem-solving datasets. | | **Reasoning dataset generation (Open Thoughts)** | [Code](https://github.com/open-thoughts/open-thoughts) | Generate the Open-Thoughts-114k dataset, focusing on reasoning traces from math, coding, and problem-solving datasets. | | **3Blue1Brown video generation** | [Code](https://github.com/bespokelabsai/curator/tree/main/examples/code-execution/math-animation) | Generate videos similar to 3Blue1Brown and render them using code execution. |
# Generating a diverse QA dataset This tutorial will guide you through creating a hierarchical dataset of diverse question-answer pairs using a structured approach similar to the CAMEL dataset. We'll build a pipeline that generates subjects, subsubjects, and corresponding Q\&A pairs. ### Introduction In many AI training scenarios, having diverse question-answer pairs across multiple domains is valuable. This tutorial demonstrates how to create "ungrounded" Q\&A pairs - meaning they're generated through language models rather than extracted from existing texts. Let's begin! ### Step 1: Setting Up the Environment First, let's set up our environment and import the required libraries. ```python # Install required packages if not already installed # !pip install pydantic bespokelabs-curator # Import necessary libraries from typing import List from pydantic import BaseModel, Field from bespokelabs import curator import os # disable this if you don't want to use Curator Viewer os.environ["CURATOR_VIEWER"] = 1 ``` ### Step 2: Define Data Models We'll use Pydantic to create structured data models that will help validate our language model outputs. ```python class Subject(BaseModel): """A single subject.""" subject: str = Field(description="A subject") class Subjects(BaseModel): """A list of subjects.""" subjects: List[Subject] = Field(description="A list of subjects") class QA(BaseModel): """A question and answer pair.""" question: str = Field(description="A question") answer: str = Field(description="An answer") class QAs(BaseModel): """A list of question and answer pairs.""" qas: List[QA] = Field(description="A list of QAs") ``` These models ensure that our data maintains a consistent structure throughout the pipeline. The `Field` objects provide descriptions that can be useful for documentation and validation. ### Step 3: Create Subject Generator Now, let's build our first component - a generator for high-level subjects. ```python class SubjectGenerator(curator.LLM): """A subject generator that generates diverse subjects.""" response_format = Subjects def prompt(self, input: dict) -> str: """Generate a prompt for the subject generator.""" return "Generate a diverse list of 3 subjects. Keep it high-level (e.g. Math, Science)." def parse(self, input: dict, response: Subjects) -> dict: """Parse the model response into the desired output format.""" return response.subjects ``` This generator will produce high-level subjects like "Math," "History," or "Computer Science." The `response_format` tells the system what structure to expect from the language model's response. ### Step 4: Create Subsubject Generator Next, we'll create a generator for subsubjects that takes each subject and generates more specific topics within it. ```python class SubsubjectGenerator(curator.LLM): """A subsubject generator that generates diverse subsubjects for a given subject.""" response_format = Subjects def prompt(self, input: dict) -> str: """Generate a prompt for the subsubject generator.""" return f"For the given subject {input['subject']}. Generate 3 diverse subsubjects. No explanation." def parse(self, input: dict, response: Subjects) -> dict: """Parse the model response into the desired output format.""" return [{"subject": input["subject"], "subsubject": subsubject.subject} for subsubject in response.subjects] ``` For example, if the subject is "Math," this generator might produce subsubjects like "Calculus," "Algebra," and "Statistics." ### Step 5: Create QA Generator Now, we'll create our final generator component that produces question-answer pairs for each subsubject. ```python class QAGenerator(curator.LLM): """A QA generator that generates diverse questions and answers for a given subsubject.""" response_format = QAs def prompt(self, input: dict) -> str: """Generate a prompt for the QA generator.""" return f"For the given subsubject {input['subsubject']}. Generate 3 diverse questions and answers. No explanation." def parse(self, input: dict, response: QAs) -> dict: """Parse the model response into the desired output format.""" return [ { "subject": input["subject"], "subsubject": input["subsubject"], "question": qa.question, "answer": qa.answer, } for qa in response.qas ] ``` This generator takes a subsubject and creates Q\&A pairs relevant to that topic. It maintains the hierarchical structure by keeping track of both the subject and subsubject. ### Step 7: Run the Complete Pipeline Now let's run our complete pipeline and see the results: ```python # Step 1: Generate subjects subject_generator = SubjectGenerator(model_name="gpt-4o-mini") subject_dataset = subject_generator() # Step 2: Generate subsubjects for each subject subsubject_generator = SubsubjectGenerator(model_name="gpt-4o-mini") subsubject_dataset = subsubject_generator(subject_dataset.dataset) # Step 3: Generate Q&A pairs for each subsubject qa_generator = QAGenerator(model_name="gpt-4o-mini") qa_dataset = qa_generator(subsubject_dataset.dataset) # Clean up answers by stripping whitespace qa_dataset = qa_dataset.dataset.map(lambda row: {"answer": row["answer"].strip()}, num_proc=2) # Print the final dataset print(qa_dataset.to_pandas()) ``` ### Example Output When run, the code might produce output like this: ``` Generated Subjects: - Mathematics - History - Biology Generated Subsubjects: - Mathematics → Calculus - Mathematics → Linear Algebra - Mathematics → Number Theory - History → Ancient Civilizations - History → World War II - History → Renaissance Period - Biology → Genetics - Biology → Ecology - Biology → Cell Biology Sample of Generated Q&A Pairs: subject subsubject question answer 0 Mathematics Calculus What is the derivative of the function f(x) = e^x? The derivative of f(x) = e^x is also e^x. This is a special property of the exponential function. 1 Mathematics Calculus What is the fundamental theorem of calculus? The fundamental theorem of calculus states that differentiation and integration are inverse processes. It connects the concept of the derivative of a function with the concept of the definite integral. 2 Mathematics Calculus How do you find the area under a curve? To find the area under a curve, you can use a definite integral. First, identify the function and the bounds of integration. Then calculate the integral over that interval. 3 Mathematics Linear Algebra What are eigenvalues and eigenvectors? Eigenvalues are special scalars associated with a linear system of equations, while eigenvectors are the corresponding vectors that, when that linear transformation is applied, change only in scale (not direction). For a matrix A, if Av = λv, then λ is an eigenvalue and v is an eigenvector. ... ``` ### Customizing the Pipeline Now that you have the basic pipeline working, here are some ways you can customize it: ```python # Modify the subject generator to produce more subjects class CustomSubjectGenerator(SubjectGenerator): def prompt(self, input: dict) -> str: return "Generate a diverse list of 5 subjects spanning sciences, arts, and humanities." # Customize the Q&A generator to produce more complex questions class ComplexQAGenerator(QAGenerator): def prompt(self, input: dict) -> str: return f""" For the given subsubject {input['subsubject']}, generate 3 diverse questions and answers. Include at least one factual question, one conceptual question, and one application question. Make the questions challenging but clear. """ ``` ### Conclusion You've now built a complete pipeline for generating diverse, hierarchical question-answer datasets! This approach is similar to how the CAMEL dataset was created, though with a simplified implementation that focuses on educational utility. This technique is particularly useful for: * Creating training data for question-answering systems * Developing educational resources across diverse domains * Evaluating AI systems on breadth of knowledge * Generating prompts for further research or content creation You can extend this framework by adding question types, difficulty levels, or specialized domains based on your specific needs. # Using SimpleStrat block for generating diverse data ## StratifiedGenerator: Generate Balanced Question-Answer Pairs ### Overview The `StratifiedGenerator` is a powerful tool for creating high-quality question-answer (QA) pairs with balanced and diverse coverage across your input questions. It ensures your generated dataset avoids biases and provides comprehensive representation of your input space. > 📝 **Research Background**: For a comprehensive understanding of the methodology and theoretical foundation, see the paper: [Stratified Generation for Artificial Data in Question Answering](https://arxiv.org/pdf/2410.09038) ### Installation ```bash pip install bespokelabs-curator ``` ### Quick Start Example ```python from datasets import Dataset from bespokelabs.curator.blocks.simplestrat import StratifiedGenerator # Create a simple dataset of questions questions = Dataset.from_dict({"question": [f"{i}. Name a periodic element" for i in range(20)]}) # Initialize the generator with your preferred model generator = StratifiedGenerator(model_name="gpt-4o-mini") # Generate stratified QA pairs qa_pairs = generator(questions).dataset # Examine the results print(f"Generated {len(qa_pairs)} QA pairs") print(qa_pairs[0]) # View the first QA pair ``` ### How StratifiedGenerator Works 1. **Input**: A `Dataset` containing questions you want to generate answers for 2. **Stratification Process**: The algorithm: * Clusters similar questions together * Ensures balanced coverage across different question types * Prevents overrepresentation of common patterns 3. **Model Integration**: Uses your specified LLM to generate high-quality answers 4. **Output**: Returns a new dataset of QA pairs with well-distributed coverage ### Advanced Usage #### Customizing the Generator ```python # With custom parameters generator = StratifiedGenerator( model_name="gpt-4o", # Use a more powerful model generation_params={ "temperature":0.7, # Adjust creativity } ) ``` #### Saving and Loading Results ```python # Save your generated QA pairs qa_pairs.push_to_hub('hf_org/dataset_name') ``` ### Performance Considerations * **Model Selection**: Larger models (e.g., GPT-4) produce higher quality answers but cost more ### Common Applications * Creating balanced training datasets for QA systems * Generating diverse test sets for robustness evaluation * Augmenting existing datasets with additional QA pairs * Creating instruction-tuning datasets with varied coverage ### Troubleshooting **Q: My generated answers seem too similar across different questions.**\ A: Try increasing the temperature parameter or the number of clusters. **Q: I'm getting API errors during generation.**\ A: Try changing the model or backend params. You can check the API reference [here](broken://pages/VAxLcu9BxAB30omwvyzp). ### Additional Resources * [BespokeLabs Documentation](https://docs.bespokelabs.io/) Happy data generation!
# Curate Reasoning data with Claude-3.7 Sonnet You can use **Sonnet** reasoning model in **Curator** to generate synthetic data. In this example, we will answer some questions with reasoning traces from claude sonnet 3.7, but the approach can be adapted for any data generation task. ## **Prerequisites** * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Anthropic:** Anthropic API key ## **Steps** #### **1. Setup environment vars** ```sh export ANTHROPIC_API_KEY= ``` **2. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: ```python """Example of reasoning on simple questions using curator.""" import os from datasets import load_dataset from bespokelabs import curator class Reasoner(curator.LLM): return_completions_object = True def prompt(self, input): return input["question"] def parse(self, input, response): """Parse the LLM response to extract reasoning and solution.""" content = response["content"] thinking = "" text = "" for content_block in content: if content_block["type"] == "thinking": thinking = content_block["thinking"] elif content_block["type"] == "text": text = content_block["text"] elif content_block["type"] == "redacted_thinking": print("Redacted thinking block! (notifying you for fun)") input["claude_thinking_trajectory"] = thinking input["claude_attempt"] = text return input ``` #### **3. Configure the Anthropic model** 
llm = Reasoner(
    model_name="claude-3-7-sonnet-20250219",
    generation_params={"max_tokens": 20000, "thinking": {"type": "enabled", "budget_tokens": 18000}},
    batch=False,
    backend="anthropic",
    backend_params={"require_all_responses": False},
)
#### **4. Generate Data** Generate the structured data and output the results as a pandas DataFrame: ```python ds = llm([ {"question": "How to solve for world peace?"}, {"question": "What is the fifteenth prime number?"}, ]) print(ds.dataset) print(ds.dataset[0]) ``` ### **Example Output** Using the above example, the output might look like this: | question | claude\_thinking\_trajectory | claude\_attempt | | ----------------------------------- | ------------------------------------------------- | ------------------------------------------------- | | How to solve for world peace? | This is a question about solving for world pea... | The Path to World Peace\n\nWorld peace is on... | | What is the fifteenth prime number? | Let me list out the prime numbers in order to ... | The fifteenth prime number is 47.\n\nThe seque... | ## **Api Reference** * Check out complete [configuration ](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#online-mode-parameters) # Synthetic Data for function calling This step-by-step tutorial will guide you through creating a system that generates customized function calls using different parameters for each row in a dataset. We'll explore how to override default generation parameters at the row level when using language models. ### Introduction In this tutorial, we'll learn how to: 1. Create a function call generator using Curator 2. Define different function tools (APIs) 3. Configure different generation parameters for each row in a dataset 4. Handle both successful function calls and regular message responses Let's dive in! ### Step 1: Import Required Libraries First, let's set up our environment and import the necessary libraries: ```python # pip install bespokelabs-curator import json from typing import Dict from datasets import Dataset from bespokelabs import curator ``` ### Step 2: Define the Function Call Generator We'll create a custom LLM class that generates function calls based on user requests: ```python class FunctionCallGenerator(curator.LLM): """A simple function calling generator.""" return_completions_object = True def prompt(self, input: Dict) -> str: """The prompt is used to generate the function call.""" return f"""You are a function calling expert. Given the user request: {input['user_request']}. Generate a function call that can be used to satisfy the user request. """ def parse(self, input: Dict, response) -> Dict: """Parse the response to extract the function call or the message.""" if "tool_calls" in response["choices"][0]["message"]: input["function_call"] = str([tool_call["function"] for tool_call in response["choices"][0]["message"]["tool_calls"]]) else: # Handle the case where the model returns a string instead of a function call input["function_call"] = response["choices"][0]["message"]["content"] return input ``` This class does two main things: * Generates a prompt asking the model to create a function call based on a user request * Parses the response to extract either the function call or regular message ### Step 3: Define Function Tools Now, let's define two function tools that our model can use: ```python function_docs = [ { "type": "function", "function": { "name": "get_weather", "description": "Retrieves current weather for the given location.", "parameters": { "type": "object", "properties": { "location": {"type": "string", "description": "City and country e.g. Bogotá, Colombia"}, "units": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "Units the temperature will be returned in."}, }, "required": ["location", "units"], "additionalProperties": False, }, "strict": True, }, }, { "type": "function", "function": { "name": "get_local_time", "description": "Get the local time of a given location", "strict": True, "parameters": { "type": "object", "required": ["location", "timezone"], "properties": { "location": {"type": "string", "description": "The name or coordinates of the location for which to get the local time"}, "timezone": {"type": "string", "description": "The timezone of the location, defaults to the location's timezone if not provided"}, }, "additionalProperties": False, }, }, }, ] ``` These function definitions describe: * A weather API that requires location and units parameters * A local time API that requires location and timezone parameters ### Step 4: Create an LLM Instance with Default Parameters Let's instantiate our function call generator with default parameters: ```python llm = FunctionCallGenerator( model_name="gpt-4o-mini", # Default generation_params has both functions generation_params={"tools": function_docs}, backend_params={"max_retries": 1, "require_all_responses": False}, ) ``` This LLM instance has: * The "gpt-4o-mini" model * Both function tools available by default * Configuration for retries and response handling ### Step 5: Create a Dataset with Row-Level Parameters Now, let's create a dataset where each row has its own generation parameters: ```python dataset = Dataset.from_dict( { "user_request": ["What's the current temperature in New York?", "What time is it in Tokyo?"], # WARNING: The generation_params in Dataset must be a string otherwise the Dataset operation automatically expand dictionary keys # See https://github.com/bespokelabsai/curator/issues/325 for more detail # The generation_params from the row will override the default generation_params during inference "generation_params": [json.dumps({"tools": [function_docs[0]]}), json.dumps({"tools": [function_docs[1]]})], } ) ``` Important notes: * The first row only has access to the weather function * The second row only has access to the time function * The `generation_params` must be JSON strings to prevent dataset operations from expanding dictionary keys ### Step 6: Run the Generator and Display Results Let's run our function call generator on the dataset: ```python function_calls = llm(dataset) # The model is expected to return a function call for each row print(function_calls.dataset.to_pandas()) ``` This will: * Process each row with its specific generation parameters * Generate appropriate function calls for each user request * Display the results in a pandas DataFrame ### Practical Applications This technique is useful for: * Processing diverse user requests with specialized tools * A/B testing different function configurations * Creating targeted function call generators for specific domains * Building efficient pipelines that adapt to different input types ### Conclusion You've learned how to create a flexible function call generation system that can adapt to different rows in a dataset. This approach allows for more targeted and efficient use of language models when generating function calls, particularly when different requests require different tools or configurations. Remember to properly configure both default and row-level parameters, and to handle both function call and regular message responses in your parsing logic. # Finetuning Examples In these examples, we demonstrate how to use synthetically generated data to finetune small LLMs and get better performance than SOTA big LLMs. * [Aspect based sentiment analysis](/bespoke-curator/finetuning-examples/aspect-based-sentiment-analysis) * [Finetuning a model to identify features of a product](/bespoke-curator/finetuning-examples/finetuning-a-model-to-identify-features-of-a-product) # Aspect based sentiment analysis ## Introduction In this notebook, we will demonstrate how to use Curator to distill capabilities from a large language model to a much smaller 8B parameter model. We will use Yelp restaurant reviews dataset to train a sentiment analysis model. We will generate a synthetic dataset using curator and finetune a model using Together's finetuning API. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1W2jf6v-ZwQ7mku1pcfdIMOawKY99Shi9?usp=sharing) Example input: ``` The food was good, but the service was slow. ``` Example output: ```json { "food_sentiment": "Positive", "service_sentiment": "Negative" } ``` ### Installation ```python !pip install bespokelabs-curator datasets together ``` ### Imports ```python from bespokelabs import curator from datasets import load_dataset from together import Together import os import json ``` ```python import getpass os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ") os.environ["TOGETHER_API_KEY"] = getpass.getpass("Enter your Together API key: ") # We use curator viewer to visualize the data fast. # You can comment it out if you don't want to use it. os.environ['CURATOR_VIEWER']='1' ``` ### Dataset Curation The data curation process is pretty simple. We will use a prompt to instruct the model to analyze the review and output the sentiment for each aspect. Note that here we are not using structured outputs, since the same prompt/curator block will be used to evaluate the base model that we finetune below (Llama-3.1-8B-Instruct). We use json mode instead of structured outputs below since many small models don't support that ````python PROMPT ="""You are a sentiment analysis expert specializing in restaurant reviews. You need to analyze the sentiment of the given restaurant review. Analyze the review for the following specific aspects: 1. Food: Quality, taste, presentation, menu variety, etc. 2. Service: Staff behavior, responsiveness, professionalism, etc. 3. Ambience: Atmosphere, decor, comfort, noise level, etc. 4. Price: Value for money, affordability, etc. 5. Overall: General impression of the restaurant experience For each aspect, classify the sentiment as exactly one of the following: - Positive: The review expresses satisfaction or praise - Negative: The review expresses dissatisfaction or criticism - Neutral: The review is balanced or doesn't mention the aspect If an aspect is not mentioned in the review, classify it as Neutral. Output the sentiment for each aspect in the following format: ```json {{ "food_sentiment": "Positive", "service_sentiment": "Negative", "ambience_sentiment": "Neutral", "price_sentiment": "Positive", "overall_sentiment": "Negative" }}``` """ class AspectBasedSentimentCurator(curator.LLM): def prompt(self, input: dict) -> str: # we can also return a prompt string: return f"{PROMPT}\nThe review is {input['text']}" return [{"role": "system", "content": PROMPT}, {"role": "user", "content": f"The review is: {input['text']}"}] def parse(self, input: dict, raw_response: str) -> dict: response = raw_response.split("```json")[1].split("```")[0] try: response = json.loads(response) except: response = {} return { **input, "food_sentiment": response.get("food_sentiment", "None"), "service_sentiment": response.get("service_sentiment", "None"), "ambience_sentiment": response.get("ambience_sentiment", "None"), "price_sentiment": response.get("price_sentiment", "None"), "overall_sentiment": response.get("overall_sentiment", "None") } ```` We will run this curator on yelp restaurant reviews dataset to generate aspect based sentiment annotations for each review. ```python source_dataset = load_dataset("bespokelabs/yelp_restaurant_reviews", split="train") # We can visualize data using Curator viewer easily. from bespokelabs.curator.utils import push_to_viewer url = push_to_viewer(source_dataset) ``` ``` Curator Viewer: ✨ https://curator.bespokelabs.ai/datasets/249dcc5c831f4563b5e7565465252ed8 ``` {% embed url="" fullWidth="false" %} ```python annotated_dataset = AspectBasedSentimentCurator( "gpt-4o", generation_params = { "temperature": 0.0, } )(source_dataset).dataset ``` {% embed url="" %} ### Creating the finetuning dataset We will create a train test split and use the curated dataset to finetune a smaller model. ```python # Rename the column to _gt (note: gt stands for ground truth). annotated_dataset = annotated_dataset.rename_column("food_sentiment", "food_sentiment_gt") annotated_dataset = annotated_dataset.rename_column("service_sentiment", "service_sentiment_gt") annotated_dataset = annotated_dataset.rename_column("ambience_sentiment", "ambience_sentiment_gt") annotated_dataset = annotated_dataset.rename_column("price_sentiment", "price_sentiment_gt") annotated_dataset = annotated_dataset.rename_column("overall_sentiment", "overall_sentiment_gt") split = int(len(annotated_dataset) * 0.9) train_dataset = annotated_dataset.select(range(split)) test_dataset = annotated_dataset.select(range(split, len(annotated_dataset))) ``` ### Evaluating the base model ```python def evaluate_sentiment(dataset): """ Evaluates sentiment analysis models by comparing model output with ground truth. """ aspects = ['food_sentiment', 'service_sentiment', 'ambience_sentiment', 'price_sentiment', 'overall_sentiment'] # Calculate accuracy for each aspect aspect_accuracies = {} for aspect in aspects: correct_predictions = sum(1 for i in range(len(dataset)) if dataset[aspect][i] == dataset[f"{aspect}_gt"][i]) total_predictions = len(dataset) aspect_accuracies[aspect] = correct_predictions / total_predictions if total_predictions > 0 else 0 # Calculate overall accuracy (average of all aspects) overall_accuracy = sum(aspect_accuracies.values()) / len(aspect_accuracies) if aspect_accuracies else 0 return {"overall_accuracy": overall_accuracy, "aspect_accuracies": aspect_accuracies} ``` ```python small_model_output = AspectBasedSentimentCurator( "together_ai/meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", generation_params = { "temperature": 0.0, }, backend_params = { "max_tokens_per_minute": 100000000, }, backend="litellm", )(test_dataset).dataset base_eval = evaluate_sentiment(small_model_output) print(json.dumps(base_eval, indent=4)) ``` Output ``` { "overall_accuracy": 0.8271653543307087, "aspect_accuracies": { "food_sentiment": 0.8661417322834646, "service_sentiment": 0.9251968503937008, "ambience_sentiment": 0.7047244094488189, "price_sentiment": 0.8149606299212598, "overall_sentiment": 0.8248031496062992 } } ``` Above, we can see that the overall accuracy is 82.7% and the aspect accuracies are not very good. Thus we will use the curated dataset to finetune a 8B parameter model. Below is the dataset if you wish to analyze further: {% embed url="" %} #### Formatting the dataset for finetuning ````python def _format_response(data_point): return f""" ```json {{ "food_sentiment": "{data_point['food_sentiment_gt']}", "service_sentiment": "{data_point['service_sentiment_gt']}", "ambience_sentiment": "{data_point['ambience_sentiment_gt']}", "price_sentiment": "{data_point['price_sentiment_gt']}", "overall_sentiment": "{data_point['overall_sentiment_gt']}" }} ``` """ finetuning_dataset = [] for data_point in train_dataset: finetuning_dataset.append({ "messages": [ {"role": "system", "content": PROMPT}, {"role": "user", "content": f"The review is: {data_point['text']}"}, {"role": "assistant", "content": _format_response(data_point)} ], }) # upload the dataset to together # create a temporary file and upload it to together with open("finetuning_dataset.jsonl", "w") as f: for data_point in finetuning_dataset: f.write(json.dumps(data_point) + "\n") # upload the file to together client = Together() file = client.files.upload("finetuning_dataset.jsonl") ```` ```python client = Together() fine_tune_response =client.fine_tuning.create( training_file = file.id, model = 'meta-llama/Meta-Llama-3.1-8B-Instruct-Reference', n_epochs = 3, suffix = '-aspect-based-sentiment-analysis-lora', lora = True, lora_r = 64, wandb_api_key = os.environ.get("WANDB_API_KEY", None) ) ``` ```python # Wait until job is completed !together fine-tuning list-events ft-xyz # paste your job ID here ``` ``` | | Message | Type | Created At | Hash | +====+===================================================+==============================================+============================+========+ | 0 | Fine tune request created | FinetuneEventType.JOB_PENDING | 2025-04-02 04:48:37.014000 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 1 | Job started at Wed Apr 2 04:49:08 UTC 2025 | FinetuneEventType.JOB_START | 2025-04-02 04:49:08 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 2 | Model data downloaded for togethercomputer/Meta- | FinetuneEventType.MODEL_DOWNLOAD_COMPLETE | 2025-04-02 04:49:10 | | | | Llama-3.1-8B-Instruct-Reference__TOG__FT at Wed | | | | | | Apr 2 04:49:10 UTC 2025 | | | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 3 | Data downloaded for togethercomputer/Meta- | FinetuneEventType.TRAINING_DATA_DOWNLOADING | 2025-04-02 04:50:21 | | | | Llama-3.1-8B-Instruct-Reference__TOG__FT at | | | | | | $2025-04-02T04:50:21.782349 | | | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 4 | Training started for model togethercomputer/Meta- | FinetuneEventType.TRAINING_START | 2025-04-02 04:52:05 | | | | Llama-3.1-8B-Instruct-Reference__TOG__FT | | | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 5 | Epoch completed, at step 31 | FinetuneEventType.EPOCH_COMPLETE | 2025-04-02 04:54:24 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 6 | Epoch completed, at step 62 | FinetuneEventType.EPOCH_COMPLETE | 2025-04-02 04:56:42 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 7 | Epoch completed, at step 93 | FinetuneEventType.EPOCH_COMPLETE | 2025-04-02 04:59:16 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 8 | Training completed for togethercomputer/Meta- | FinetuneEventType.TRAINING_COMPLETE | 2025-04-02 04:59:37 | | | | Llama-3.1-8B-Instruct-Reference__TOG__FT at Wed | | | | | | Apr 2 04:59:36 UTC 2025 | | | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 9 | Uploading output model | FinetuneEventType.MODEL_UPLOADING | 2025-04-02 05:00:20 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 10 | Compressing output model | FinetuneEventType.MODEL_COMPRESSING | 2025-04-02 05:00:39 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 11 | Model compression complete | FinetuneEventType.MODEL_COMPRESSION_COMPLETE | 2025-04-02 05:00:58 | | +----+---------------------------------------------------+----------------------------------------------+----------------------------+--------+ | 12 | Model upload complete | FinetuneEventType.MODEL_UPLOAD_COMPLETE | 2025-04-02 05:03:00 | ``` ```python # Run the finetuned model on the test dataset ft_output = annotated_dataset = AspectBasedSentimentCurator( # Replace with the model id of the fine-tuned model # You will get the model ID from here: https://api.together.xyz/models "together_ai/mahesh_bespoke/Meta-Llama-3.1-8B-Instruct-Reference--aspect-based-sentiment-analysis-lora-xyz", generation_params = { "temperature": 0.0, }, backend_params = { "max_tokens_per_minute": 100000000, } )(test_dataset).dataset ft_eval = evaluate_sentiment(ft_output) print(json.dumps(ft_eval, indent=4)) ``` Output ``` { "overall_accuracy": 0.917716535433071, "aspect_accuracies": { "food_sentiment": 0.9035433070866141, "service_sentiment": 0.9409448818897638, "ambience_sentiment": 0.8858267716535433, "price_sentiment": 0.8937007874015748, "overall_sentiment": 0.9645669291338582 } } ``` ### Comparing Results ```python # Compare the results import pandas as pd from IPython.display import display base_model_results = base_eval fine_tuned_results = ft_eval # Create a comparison table comparison_data = { "Metric": ["Overall Accuracy"] + [f"{k.replace('_', ' ').title()}" for k in base_model_results["aspect_accuracies"].keys()], "Base Model": [base_model_results["overall_accuracy"]] + list(base_model_results["aspect_accuracies"].values()), "Fine-tuned Model": [fine_tuned_results["overall_accuracy"]] + list(fine_tuned_results["aspect_accuracies"].values()), } # Create and display the DataFrame comparison_df = pd.DataFrame(comparison_data) pct_improvement = (comparison_df["Fine-tuned Model"] - comparison_df["Base Model"]) / comparison_df["Base Model"] * 100 comparison_df["Percentage improvement"] = pct_improvement.apply(lambda x: f"{x:.2f}%") display(comparison_df.style.format({ "Base Model": "{:.3f}", "Fine-tuned Model": "{:.3f}", }).set_caption("Model Performance Comparison")) ```
MetricBase ModelFine-tuned Model% Improvement
0Overall Accuracy0.8270.91810.95 %
1Food Sentiment0.8660.9044.32 %
2Service Sentiment0.9250.9411.70 %
3Ambience Sentiment0.7050.88625.70 %
4Price Sentiment0.8150.8749.66 %
5Overall Sentiment0.8250.96516.95 %
### Conclusion We can see that the fine-tuned model has higher overall accuracy and also better aspect accuracies. Also, it is 13.8x cheaper than the teacher model ($0.18 for the 8B model on together.ai vs. $2.5 for GPT-4o, per million tokens)! As next steps, we can rerun with a larger dataset and better hyperparameter settings, to match the performance of GPT-4o. # Finetuning a model to identify features of a product **Note:** This example requires a GPU for finetuning. If you don't have a machine with GPUs handy, you can use the Colab version below with free T4 GPUs. [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1YoA23-cBcWpaSErULzBI2bo2LPGo37GQ) We will go through a small example here to create data with Ollama using Curator, finetune with Unsloth, and then evaluate it again using Curator. Imagine you are a product wizard at a fictional product company called Azanom Inc., and want to highlight product features in the description of each product.
Code for displaying product ```python from IPython.display import HTML, display import re def display_product( product_name, description, features, image_url, ): """Displays a product give its product_name, features, and description""" # Product description and features def highlight_features(text, features): # Sort features by length in descending order to handle overlapping matches sorted_features = sorted(features, key=len, reverse=True) # Create a copy of the text for highlighting highlighted_text = text # Replace each feature with its highlighted version for feature in sorted_features: pattern = re.compile(re.escape(feature), re.IGNORECASE) highlighted_text = pattern.sub( f'{feature}', highlighted_text ) return highlighted_text # Create HTML content with CSS styling html_content = f"""

{product_name}

""" if image_url: html_content += f'{product_name}' html_content += f"""

{highlight_features(description, features)}

""" display(HTML(html_content)) display_product( product_name="Apple Airpods Pro", description="The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", features=[ "lightweight in-ear earbuds", "contoured design", "sits at an angle for comfort", "better direct audio to your ear", "stem is 33% shorter than the second generation AirPods", "force sensor to easily control music and calls", "Spatial Audio with dynamic head tracking", "immersive, three-dimensional listening experience"], image_url="https://store.storeimages.cdn-apple.com/4982/as-images.apple.com/is/airpods-pro-2-hero-select-202409_FMT_WHH?wid=750&hei=556&fmt=jpeg&qlt=90&.v=1724041668836") ```
Given a product and its description, your first instinct is to use GPT-4o, to get the features given a product description. But you quickly realize that you don't need a jackhammer to nail this one and want to find a much cheaper and scalable alternative. So let's try to train a 1B model by generating data from a 8B model. This data generation should cost $0. We can always use bigger models to generate higher-quality data. Note that we have simplified this example for demonstration purposes. ## Installation ```bash # Install Python packages !pip install bespokelabs-curator==0.1.15.post1 !pip install fuzzywuzzy datasets pydantic !pip install unsloth !pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git !pip install bitsandbytes triton unsloth_zoo # Install ollama !curl https://www.ollama.com/install.sh | OLLAMA_VERSION="0.5.4" sh # We need llama3.1:8b for data generation and llama3.2:1b model for finetuning !ollama pull llama3.1:8b !ollama pull llama3.2:1b ``` Import the required library 
# Make sure the following imports fine before proceeding, since this is needed for finetuning.
from unsloth import FastLanguageModel
from bespokelabs import curator

import os
import re
import json
import torch
import random
import numpy as np
from typing import List
from fuzzywuzzy import fuzz
from pydantic import BaseModel, Field
from datasets import Dataset, load_dataset
## Generate training data using Llama-3.1-8B Our goal is to extract features from the product descriptions. We will use `Curator` to easily generate a dataset of products and their features. We seed the dataset with personas from PersonaHub and create products for each persona for diverse products. To make the data generation process easy for LLMs, we include the and tag in the output description. This way, we get high quality descriptions for given features. ```python # load the personas dataset personas = load_dataset("proj-persona/PersonaHub", 'persona') personas = personas['train'].take(100) personas[0] ``` We can then create a `ProductCurator` object and curate products using personas ````python class ProductCurator(curator.LLM): # input prompt to the curator def prompt(self, row): return f"""Generate a product for the following persona: {row['persona']} The product should be a product that is relevant to the persona. Give a name, description and features for the product. Give upto 10 features for the product. Features should be relevant, extremely detailed and useful for the persona. Then, generate a description for the product. Note that each feature should exactly be mentioned in the description. Do not add any other features or miss any features. An example output is: {{ "name": "Apple AirPods Pro", "features": [ "lightweight in-ear earbuds", "contoured for a comfortable fit", "sits at an angle for comfort", "better direct audio to your ear", "stem is 33% shorter than the second generation AirPods", "force sensor to easily control music and calls", "Spatial Audio with dynamic head tracking", "immersive, three-dimensional listening experience" ] "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and each airpod sits at an angle for comfort. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", }} Ensure each feature in the paragraph matches exactly as written in the description, including the and tags. Make sure your output is a JSON and is in the following format. DO NOT OUTPUT ANYTHING ELSE. INCLUDE THE ```json tag in your response. ```json {{ "name": "name of the product", "features": [ "feature 1", "feature 2", "feature 3", ... ], "description": "description of the product" }}``` """ def parse(self, row, response): """Parse the LLM response to extract the product name, features and description.""" default_response = { "name": "Apple AirPods Pro", "features": [ "lightweight in-ear earbuds", "contoured for a comfortable fit", "sits at an angle for comfort", "better direct audio to your ear", "stem is 33% shorter than the second generation AirPods", "force sensor to easily control music and calls", "Spatial Audio with dynamic head tracking", "immersive, three-dimensional listening experience" ], "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and each airpod sits at an angle for comfort. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", } if type(response) == type(''): pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: response = json.loads(json_string) except: response = default_response else: response = default_response else: response = response.dict() try: row['product'] = response['name'] # note that because the LLM isn't perfect, the features in the response may not be fully accurate # row['original_features'] = response.features # that's why, we parse the features from the output pattern = r"(.*?)" matches = re.findall(pattern, response['description']) if matches: row['features'] = matches else: # backup row['features'] = response['features'] row['description'] = response['description'].replace('','').replace('','') except: return [] return row product_curator = ProductCurator( model_name="ollama/llama3.1:8b", # Ollama model identifier backend_params={ "base_url": "http://localhost:11434", "max_tokens_per_minute": 3000000, "max_requests_per_minute": 10, }, ) ```` Next, let's create some products for the personas with `ProductCurator`! This can take a while. You can use Together.ai or Deepinfra through Curator and [LiteLLM](/bespoke-curator/how-to-guides/using-litellm-with-curator) to speed up this up. ```python # Generate products for the personas. This will take a while. # You can use , for example, to speed this up. products = product_curator(personas).dataset ``` Here's an example of a generated product:
Example generated product **PERSONA:** A Political Analyst specialized in El Salvador's political landscape. **PRODUCT:** Salvadoria: El Salvador's Political Landscape Analyzer **DESCRIPTION:** Salvadoria is a cutting-edge tool designed specifically for Political Analysts specializing in El Salvador's political landscape. It offers Advanced natural language processing for news articles and social media posts, allowing users to quickly analyze the tone, sentiment, and key themes of online discussions. The customizable keyword alert system enables analysts to track specific topics and hashtags in real-time, ensuring they stay up-to-date on the latest developments. Salvadoria also features an interactive map of El Salvador with election results, demographic data, and key infrastructure information, providing a comprehensive view of the country's political landscape. With access to a comprehensive database of past elections, including voter turnout, candidate performance, and electoral district boundaries, analysts can gain valuable insights into historical trends and patterns. The tool also includes an in-depth analysis of government spending, revenue, and budget allocation by department and agency, allowing users to identify areas of inefficiency or potential corruption. Salvadoria's real-time tracking of public opinion polls, surveys, and focus groups on various political issues keeps analysts informed about shifting public sentiment and policy preferences. Users can customize their dashboard with a range of visualizations and metrics using the customizable dashboard, while also exporting data in CSV format for further analysis or integration with other tools via the ability to export data in CSV format. Regular updates include new data, including special reports on election forecasts, economic indicators, and policy changes, which are integrated seamlessly through the integration with popular spreadsheet software. **FEATURES:** * advanced natural language processing for news articles and social media posts * keyword alert system * interactive map of El Salvador with election results, demographic data, and key infrastructure information * comprehensive database of past elections * in-depth analysis of government spending, revenue, and budget allocation by department and agency * real-time tracking of public opinion polls, surveys, and focus groups * customizable dashboard * ability to export data in CSV format * integration with popular spreadsheet software
## Evaluate the baseline performance on eval data from gpt-4o-mini ### **Set up the EvaluationLLM object using curator** We can create an `EvaluationLLM` object to evaluate the performance of our models ````python FEATURE_PROMPT = """ You are given a product's name, description and features. You will generate a list of features for the product. An example input is: {{ "name": "Apple AirPods Pro", "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", }} An example output is: {{ "features": [ "lightweight in-ear earbuds", "contoured for a comfortable fit", "sit at an angle for easy access to the controls", "stem is 33% shorter than the second generation AirPods", "force sensor to easily control music and calls", "Spatial Audio with dynamic head tracking", "immersive, three-dimensional listening experience" ] }} Now, generate a list of features for the product. You should output all the features that are mentioned in the description exactly as they are written. You should not miss any features, or add any features that are not mentioned in the description. Your output should be in this format. ```json{{features: ["feature 1","feature 2","feature 3",...]}}``` Product: Name: {product_name} Description: {product_description} Output: """ class EvaluationLLM(curator.LLM): # prompt for evaluation def prompt(self, row): return FEATURE_PROMPT.format(product_name=row['product'], product_description=row['description']) # function to parse the LLM responses given by curator # this function also contains the logic for evaluation of the LLM responses def parse(self, row, response): true_set = set(row['features']) pred_set = set() # Fuzzy matching threshold SIMILARITY_THRESHOLD = 0.85 if type(response) != type(""): predicted_features = response.features else: # string pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: out_dict = json.loads(json_string) predicted_features = out_dict.get("features", []) except: print("Incorrect output format..") predicted_features = [] else: predicted_features = [] for pred_feature in predicted_features: # Check if any true feature matches this predicted feature best_match_score = 0 for true_feature in true_set: similarity = fuzz.ratio(pred_feature.lower(), true_feature.lower()) / 100.0 best_match_score = max(best_match_score, similarity) if best_match_score >= SIMILARITY_THRESHOLD: pred_set.add(pred_feature) # Calculate metrics row['true_positives'] = len(pred_set) # Features that matched above threshold row['false_positives'] = len(predicted_features) - row['true_positives'] # Predicted features that didn't match row['false_negatives'] = len(true_set) - row['true_positives'] # True features that weren't matched return row ```` We also set up some utilities to run the evaluation, calculate precision, recall, and F1 metrics, and tabulate them in a nice format. ```python # @title Utilities to calculate precision, recall, f1, run evaluations and tabulate results def calculate_metrics(evaluation): tp = sum(evaluation['true_positives']) fp = sum(evaluation['false_positives']) fn = sum(evaluation['false_negatives']) micro_precision = tp / (tp + fp) micro_recall = tp / (tp + fn) micro_f1 = (2 * micro_precision * micro_recall) / (micro_precision + micro_recall) return {'precision': micro_precision,'recall': micro_recall, 'f1':micro_f1} # Common function to run evaluation on different models def run_evaluation(model_name, dataset): evaluator = EvaluationLLM( model_name=model_name, backend_params={ "max_requests_per_minute":10000, "max_tokens_per_minute":30000000 } ) evaluation = evaluator(dataset).dataset metrics = calculate_metrics(evaluation) return evaluation, metrics # Tabulate eval results from tabulate import tabulate def tabulate_eval_results(model_and_metrics): metrics_names = ['Precision', 'Recall', 'F1'] # Create table data table_data = [] for model, metrics in model_and_metrics.items(): table_data.append([ model, f"{metrics['precision']:.3f}", f"{metrics['recall']:.3f}", f"{metrics['f1']:.3f}" ]) # Print table print(tabulate(table_data, headers=['Model', 'Precision', 'Recall', 'F1'], tablefmt='grid')) ``` ### **Create an eval set with gpt-4o-mini** In order to prevent bias from using the same model to generate train and eval data, we are not going to create a train and test split using the newly created data from Llama-3.1-8B. Instead, we will use all of it for training but generate eval data with a completely different LLM, gpt-4o-mini. ```python import getpass os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") eval_product_curator = ProductCurator( model_name="gpt-4o-mini" ) train_dataset = products test_dataset = eval_product_curator(personas.take(40)).dataset ``` ### **Run the evaluation and get results** ```python evaluation_results_1b, metrics_1b = run_evaluation('ollama/llama3.2:1b', test_dataset) evaluation_results_8b, metrics_8b = run_evaluation('ollama/llama3.1:8b', test_dataset) tabulate_eval_results(model_and_metrics={"llama-3.2-1b": metrics_1b, "llama-3.1-8b": metrics_8b}) ``` ``` +--------------+-------------+----------+-------+ | Model | Precision | Recall | F1 | +==============+=============+==========+=======+ | llama-3.2-1b | 0.668 | 0.394 | 0.496 | +--------------+-------------+----------+-------+ | llama-3.1-8b | 0.726 | 0.753 | 0.739 | +--------------+-------------+----------+-------+ ``` We can see that Llama-3.1-8B is not able to extract the features as well as 8B (as expected). So, we will finetune it on the training set. ## Finetune Llama3.2-1B using Unsloth ### Prepare data for finetuning ````python from unsloth import FastLanguageModel from unsloth.chat_templates import get_chat_template max_seq_length = 1024 load_in_4bit = True dtype = None # for auto model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-1B-Instruct-bnb-4bit", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) peft_model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ ) # doing this so ollama creates a modelfile tokenizer = get_chat_template( tokenizer, chat_template = "llama-3.1", ) # Prepare a dataset for finetuning def formatting_prompts_func(row): texts = [] features = {'features': row['features']} messages = [ {"role": "user", "content": FEATURE_PROMPT.format(product_name=row['product'], product_description=row['description'])}, {"role": "assistant", "content": f"```json{json.dumps(features)}```"}, ] text = tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = False) return {'text': text} ft_dataset = train_dataset.map(formatting_prompts_func, batched = False,) ft_dataset[0] ```` 
{'persona': "A Political Analyst specialized in El Salvador's political landscape.",
 'product': 'SalvadorAlert',
 'features': ["real-time updates on El Salvador's legislative calendar",
  'in-depth analysis of proposed laws and bills',
  'topics that matter most to them',
  'data visualization of voting patterns and trends',
  'comparative analysis of past and present legislative data',
  'upcoming hearings and committee meetings',
  "detailed information on El Salvador's presidential and congressional elections",
  'analysis of public opinion polls and surveys',
  'news from local and international sources',
  'monitors social media activity of key politicians and influencers'],
 'description': "The SalvadorAlert is a cutting-edge tool for political analysts specializing in El Salvador's political landscape. It provides real-time updates on El Salvador's legislative calendar, including key dates and events. The platform offers in-depth analysis of proposed laws and bills, allowing users to stay on top of the latest developments. Users can also customize their alert system to receive priority notifications on topics that matter most to them. The SalvadorAlert features data visualization of voting patterns and trends, providing a clear picture of the current political climate. Additionally, users can access comparative analysis of past and present legislative data to inform their research. The platform also includes alerts on upcoming hearings and committee meetings, ensuring users are always informed. SalvadorAlert provides detailed information on El Salvador's presidential and congressional elections, including key statistics and analysis. Users can also access analysis of public opinion polls and surveys to gauge public sentiment. Furthermore, the SalvadorAlert aggregates news from local and international sources related to El Salvador's politics, providing a comprehensive view of the news cycle. Finally, the platform monitors social media activity of key politicians and influencers, allowing users to stay ahead of the curve.",
 'text': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 July 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n\n    You are given a product\'s name, description and features. You will generate a list of features for the product.\n\n    An example input is:\n    {\n        "name": "Apple AirPods Pro",\n        "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.",\n    }\n\n    An example output is:\n    {\n        "features": [\n            "lightweight in-ear earbuds",\n            "contoured for a comfortable fit",\n            "sit at an angle for easy access to the controls",\n            "stem is 33% shorter than the second generation AirPods",\n            "force sensor to easily control music and calls",\n            "Spatial Audio with dynamic head tracking",\n            "immersive, three-dimensional listening experience"\n        ]\n\n    }\n\n    Now, generate a list of features for the product. You should output all the features that are mentioned in the description exactly as they are written. You should not miss any features, or add any features that are not mentioned in the description.\n\n    Your output should be in this format.\n    ```json{features: ["feature 1","feature 2","feature 3",...]}```\n\n    Product:\n      Name: SalvadorAlert\n      Description: The SalvadorAlert is a cutting-edge tool for political analysts specializing in El Salvador\'s political landscape. It provides real-time updates on El Salvador\'s legislative calendar, including key dates and events. The platform offers in-depth analysis of proposed laws and bills, allowing users to stay on top of the latest developments. Users can also customize their alert system to receive priority notifications on topics that matter most to them. The SalvadorAlert features data visualization of voting patterns and trends, providing a clear picture of the current political climate. Additionally, users can access comparative analysis of past and present legislative data to inform their research. The platform also includes alerts on upcoming hearings and committee meetings, ensuring users are always informed. SalvadorAlert provides detailed information on El Salvador\'s presidential and congressional elections, including key statistics and analysis. Users can also access analysis of public opinion polls and surveys to gauge public sentiment. Furthermore, the SalvadorAlert aggregates news from local and international sources related to El Salvador\'s politics, providing a comprehensive view of the news cycle. Finally, the platform monitors social media activity of key politicians and influencers, allowing users to stay ahead of the curve.\n    Output:\n<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n```json{"features": ["real-time updates on El Salvador\'s legislative calendar", "in-depth analysis of proposed laws and bills", "topics that matter most to them", "data visualization of voting patterns and trends", "comparative analysis of past and present legislative data", "upcoming hearings and committee meetings", "detailed information on El Salvador\'s presidential and congressional elections", "analysis of public opinion polls and surveys", "news from local and international sources", "monitors social media activity of key politicians and influencers"]}```<|eot_id|>'}
### Run SFT finetuning with Unsloth ```python from trl import SFTTrainer from transformers import TrainingArguments, DataCollatorForSeq2Seq from unsloth import is_bfloat16_supported from unsloth.chat_templates import train_on_responses_only trainer = SFTTrainer( model = peft_model, tokenizer = tokenizer, train_dataset = ft_dataset, dataset_text_field = "text", max_seq_length = max_seq_length, data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer), dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, num_train_epochs = 1, # Set this for 1 full training run. learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # Use this for WandB etc ), ) trainer = train_on_responses_only( trainer, instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", ) trainer_stats = trainer.train() ``` ### Save the finetuned model and serving it using Ollama ```python # save quantized model peft_model.save_pretrained_gguf("llama_finetune", tokenizer) ``` 
# Unsloth automatically creates a Modelfile!
cat /content/llama_finetune/Modelfile
# Create an ollama model using the saved Modelfile
ollama create llama_finetune -f ./llama_finetune/Modelfile
# Verify that the finetuned model exists
ollama ls
## **Final results** Running the evaluation on the new finetuned model, we found that F1 for Llama-3.2-1B model jumped from 0.496 to 0.688, a significant improvement! ```python # evaluate the finetuned model finetuned_llama_evaluation_results, finetuned_llama_metrics = run_evaluation('ollama/llama_finetune', test_dataset) print(finetuned_llama_evaluation_results[0]) print(finetuned_llama_metrics) tabulate_eval_results(model_and_metrics={"llama-3.2-1b": metrics_1b, "llama-3.1-8b": metrics_8b, "finetuned-llama-3.2-1b": finetuned_llama_metrics}) ``` ``` +------------------------+-------------+----------+-------+ | Model | Precision | Recall | F1 | +========================+=============+==========+=======+ | llama-3.2-1b | 0.668 | 0.394 | 0.496 | +------------------------+-------------+----------+-------+ | llama-3.1-8b | 0.726 | 0.753 | 0.739 | +------------------------+-------------+----------+-------+ | finetuned-llama-3.2-1b | 0.796 | 0.606 | 0.688 | +------------------------+-------------+----------+-------+ ``` Just for fun, let try running our new finetuned model on a new example: ````python product_name = "Ryobi Circular Saw" product_description = """ Expand your RYOBI 18V ONE+ System with the RYOBI 18V ONE+ Cordless Circular Saw. Make over 215 fast, clean cuts per charge on the ONE+ Cordless 5 1/2 in. Circular Saw with 4,700 RPM and the included 18T Carbide Tipped Blade. This saw is ideal for cross cuts in 2-by material with 1-11/16 in. maximum depth of cut. Bevel up to 50 degrees to complete a wide variety of cuts and with 1-3/16 in. depth of cut at 45 Degrees of bevel. Purchase the accessory vacuum dust adaptor (sold separately) to connect this saw to your wet/dry vac for quick and easy clean up. Best of all, it is part of the RYOBI ONE+ System of over 300 Cordless Products that all work on the same battery platform. This 18V ONE+ Cordless 5-1/2 in. Circular Saw is backed by the RYOBI 3-Year Manufacturer's Warranty. Battery and charger sold separately.""" class Extractor(curator.LLM): def prompt(self, input): return FEATURE_PROMPT.format(product_name=input['product'], product_description=input['description']) extractor = Extractor(model_name='gpt-4o') result = extractor(Dataset.from_list([{'product': product_name, 'description': product_description}])).dataset['response'][0] print(result) def get_parsed_response(response): if type(response) == type(''): pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: response = json.loads(json_string) except: raise ValueError("Failed to parse") else: raise ValueError("Failed to parse") else: response = response.dict() return response response = get_parsed_response(result) display_product( product_name, product_description, response['features'], image_url='' ) ````
This is not bad for a quick start. In some cases, you will see that the LLM doesn't output exact text (which happens for even GPT-4o)! Great next steps: 1. Increase the number of training examples. 2. Systematically evaluate the error types. 3. Run this in local machine and run `curator-viewer` to visualize your data. 4. Create complex strategies for data curation (involving multiple curator.LLM stages). 5. Star ! ``` ``` # API Reference ## curator.LLM The `LLM` class serves as the primary interface for prompting Large Language Models in Curator. It provides a flexible and extensible way to generate synthetic data using various LLM providers. Returns `CuratorResponse` which holds dataset, statistics (performance, token usage, cost etc) attributes. ### Class Definition ```python class LLM: def __init__( self, model_name: str, response_format: Type[BaseModel] | None = None, batch: bool = False, backend: Optional[str] = None, generation_params: dict | None = None, backend_params: BackendParamsType | None = None, ) ``` ### Constructor Parameters
ParameterTypeDefaultDescription
model_namestrRequiredName of the LLM to use
response_formatType[BaseModel] | NoneNonePydantic model specifying the expected response format
batchboolFalseEnable batch processing mode
backendOptional[str]NoneLLM backend to use ("openai", "litellm", or "vllm"). Auto-determined if None
generation_paramsdict | NoneNoneAdditional parameters for the generation API
backend_paramsBackendParamsType | NoneNoneConfiguration parameters for request processor
### Backend Parameters Configuration The `backend_params` dictionary supports various configuration options based on the execution mode. Here's a comprehensive breakdown: #### Common Parameters These parameters are available across all backends: | Parameter | Type | Default | Description | | ------------------------ | ---------------- | ------------------------------ | ------------------------------------------------------- | | `max_retries` | `int` | `3` | Maximum number of retry attempts for failed requests | | `require_all_responses` | `bool` | `False` | Whether to require successful responses for all prompts | | `base_url` | `Optional[str]` | `None` | Optional base URL for API endpoint | | `request_timeout` | `int` | `600` | Timeout in seconds for each request | | `api_key` | `Optional[str]` | `None` | Api key for the selected model. | | `in_mtok_cost` | `Optional[int]` | `None` | Optional cost per million input tokens. | | out`_mtok_cost` | `Optional[int]` | `None` | Optional cost per million output tokens. | | `invalid_finish_reasons` | `Optional[list]` | `['content_filter', 'length'`] | List of api finish reasons which are considered failed. | ```python # Example: Common parameters configuration backend_params = { "max_retries": 3, "require_all_responses": True, "base_url": "https://custom-endpoint.com/v1", "request_timeout": 300 } ``` #### Online Mode Parameters Parameters for online processor mode: | Parameter | Type | Description | | -------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------- | | `max_requests_per_minute` | `int` | Maximum number of API requests per minute | | `max_tokens_per_minute` | `int` | Maximum number of tokens per minute | | `seconds_to_pause_on_rate_limit` | `float` | Duration to pause when rate limited | | `max_concurrent_requests` | `int` | Maximum number of concurrent requests. | | `max_input_tokens_per_minute` | `int` | Maximum number of input tokens allowed per minute. Note: Only valid with seperate token strategy i.e Anthropic. | | `max_output_tokens_per_minute` | `int` | Maximum number of output tokens allowed per minute. Note: Only valid with seperate token strategy i.e Anthropic. | ```python # Example: Online mode configuration backend_params = { "max_requests_per_minute": 2000, "max_tokens_per_minute": 4_000_000, "seconds_to_pause_on_rate_limit": 15.0 } ``` #### Batch Processing Parameters Parameters available when `batch=True`: | Parameter | Type | Description | | ------------------------------- | ------- | -------------------------------------------------------------------------------------------- | | `batch_size` | `int` | Number of prompts to process in each batch | | `batch_check_interval` | `float` | Time interval between batch completion checks | | `delete_successful_batch_files` | `bool` | Whether to delete successful batch files | | `delete_failed_batch_files` | `bool` | Whether to delete failed batch files | | `completion_window` | `str` |

Time window to wait for batch completion.

Note: only valid for some providers.

| ```python # Example: Batch processing configuration backend_params = { "batch_size": 100, "batch_check_interval": 1.0, "delete_successful_batch_files": True, "delete_failed_batch_files": False, } ``` #### Offline Mode Parameters (VLLM) Parameters for local model deployment with VLLM: | Parameter | Type | Description | | ------------------------ | ------- | ------------------------------------------ | | `tensor_parallel_size` | `int` | Number of GPUs for tensor parallelism | | `enforce_eager` | `bool` | Whether to enforce eager execution | | `max_model_length` | `int` | Maximum sequence length for the model | | `max_tokens` | `int` | Maximum tokens for generation | | `min_tokens` | `int` | Minimum tokens for generation | | `gpu_memory_utilization` | `float` | Target GPU memory utilization (0.0 to 1.0) | | `batch_size` | `int` | Batch size for VLLM processing | ```python # Example: VLLM configuration backend_params = { "tensor_parallel_size": 2, "max_model_length": 4096, "max_tokens": 2048, "min_tokens": 1, "gpu_memory_utilization": 0.85, "batch_size": 32 } ``` ### Methods ### prompt() ```python def prompt(self, input: _DictOrBaseModel) -> _DictOrBaseModel ``` Generates a prompt for the LLM based on the input data. **Parameters** * `input`: Input row used to construct the prompt **Returns** A prompt that can be either: 1. A string for a single user prompt 2. A list of dictionaries for multiple messages **Example** ```python def prompt(self, input: dict) -> str: return f"Generate a {input['type']} about {input['topic']}" ``` ### parse() ```python def parse(self, input: _DictOrBaseModel, response: _DictOrBaseModel) -> _DictOrBaseModel ``` Processes the LLM's response and optionally can be used to combine it with the input data. **Parameters** * `input`: Original input row used for the prompt * `response`: Raw response from the LLM **Returns** A parsed output combining the input and response data **Example** ```python def parse(self, input: dict, response: str) -> dict: return { "prompt_topic": input["topic"], "generated_text": response, "timestamp": datetime.now().isoformat() } ``` ### Returns A `CuratorResponse` object which consists statistics about token usage, performance and cost along with the dataset and viewer link. #### `CuratorResponse` #### Attributes #### **Core Data** * dataset (Dataset): The curated dataset * cache\_dir (Optional\[str]): Directory for caching results * failed\_requests\_path (Optional\[Path]): Path to file containing failed requests * viewer\_url (Optional\[str]): URL for Curator Viewer * batch\_mode (bool): Whether the processing was done in batch mode #### Model Information * model\_name (str): Name of the LLM model used * max\_requests\_per\_minute (int | None): Rate limit for requests per minute * max\_tokens\_per\_minute (int | None): Rate limit for tokens per minute #### Statistics * token\_usage (TokenUsage): Statistics about token usage * cost\_info (CostInfo): Information about processing costs * request\_stats (RequestStats): Statistics about request processing * performance\_stats (PerformanceStats): Performance metrics * metadata (Dict\[str, Any]): Additional metadata ### Response Format (Optional) The `response_format` class attribute can be set to a Pydantic model to enforce structured output: ```python from pydantic import BaseModel class RecipeResponse(BaseModel): title: str ingredients: List[str] instructions: List[str] class RecipeGenerator(LLM): response_format = RecipeResponse ``` ### Usage Examples #### Basic Usage ```python class Cuisines(BaseModel): """A list of cuisines.""" cuisines_list: List[str] = Field(description="A list of cuisines.") class CuisineGenerator(curator.LLM): """A cuisine generator that generates diverse cuisines.""" response_format = Cuisines def prompt(self, input: dict) -> str: """Generate a prompt for the cuisine generator.""" return "Generate 10 diverse cuisines." def parse(self, input: dict, response: Cuisines) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return [{"cuisine": t} for t in response.cuisines_list] ``` # Bespoke MiniCheck A state-of-the-art grounded factuality model > *“Hallucination-free answers demand verifiable grounding. Bespoke-MiniCheck makes that easy.”* #### What is grounded factuality? Grounded factuality (a.k.a. textual entailment) measures whether a *claim* is supported, refuted, or not verifiable given an explicit *context* document. The metric is critical for Retrieval-Augmented Generation (RAG): if a claim is not grounded in the retrieved context, the model has hallucinated. #### Why Bespoke-MiniCheck? * Best-in-class accuracy – Tops the public **LLM-AggreFact** leaderboard at 77.4 %, surpassing models that are many times larger. * Fast – \~200 ms end-to-end latency on a single modern GPU; < 100 ms with optional optimisations. * Lightweight – Runs comfortably on consumer laptops (MacBook-class hardware). * Easy to integrate – Drop-in HuggingFace model with a single probability output: *support score*. # Self-Hosting You can access the model here: Feel free to use this [colab](https://colab.research.google.com/drive/1s-5TYnGV3kGFMLp798r5N-FXPD8lt2dm?usp=sharing) which uses the MiniCheck library that supports automated chunking of long documents. Or, you can host the model directly on vLLM with docker as follows: ````sh ```shellscript sudo docker run \ --runtime=nvidia \ --gpus=all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=hf_xyz" \ --ipc=host \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model bespokelabs/Bespoke-MiniCheck-7B --trust_remote_code --api-key your_api_key --disable-log-requests \ --dtype bfloat16 \ --max-model-len 32768 \ --tensor-parallel-size 1 & ``` ```` Please contact us for commercial licensing. # Integrations ### Guardrails Bespoke MiniCheck is available as a Guardrails validator here: Example usage: ```python # Import Guard and Validator from guardrails.hub import BespokeMiniCheck from guardrails import Guard # Setup Guard guard = Guard().use( BespokeMiniCheck, split_sentences=True, threshold=0.5, on_fail="fix" ) # Validator passes guard.validate("Alex likes cats.", metadata={"context": "Alex likes cats and dogs"}) # Validator fails guard.validate("Alex likes cats.", metadata={"context": "Alex likes dogs, but not cats."}) ``` ### Ollama Bespoke-MiniCheck-7B is available from Ollama [here](https://ollama.com/library/bespoke-minicheck). More information can be found from their [blog post](https://ollama.com/blog/reduce-hallucinations-with-bespoke-minicheck). Once you have Ollama, it is pretty straightforward to use the model. Note that Ollama doesn't yet support getting logits from the model, therefore we just output "yes" or "no". As part of Ollama, there are two examples available: 1. [Fact checking](https://github.com/ollama/ollama/tree/main/examples/python-grounded-factuality-simple-check) 2. [RAG use case](https://github.com/ollama/ollama/tree/main/examples/python-grounded-factuality-rag-check) # API Service Using the api service is quite easy. #### Step 1: API Key Setup First, get your API key at the [Bespoke Console](https://console.bespokelabs.ai). ``` export BESPOKE_API_KEY=besoke-... ``` #### Step 2: Install Dependencies Install the package: ``` pip install bespokelabs ``` #### Step 3: Run ```python import os from bespokelabs import BespokeLabs bl = BespokeLabs( # This is the default and can be omitted auth_token=os.environ.get("BESPOKE_API_KEY"), ) response = bl.minicheck.factcheck.create( claim="claim", context="context", ) print(response.support_prob) ``` Lot more information about the library is available at the [bespokelabs pypi page](https://pypi.org/project/bespokelabs/). # Bespoke MiniChart Playground: Model on HF: (also has inference code). # OpenThinker Please find more information about this at .