Bespoke Labs
  • Welcome
  • BESPOKE CURATOR
    • Getting Started
      • Quick Tour
      • Key Concepts
      • Visualize your dataset with the Bespoke Curator Viewer
      • Automatic recovery and caching
      • Structured Output
    • Save $$$ on LLM inference
      • Using OpenAI for batch inference
      • Using Anthropic for batch inference
      • Using Gemini for batch inference
      • Using Mistral for batch inference
      • Using kluster.ai for batch inference
    • How-to Guides
      • Using vLLM with Curator
      • Using Ollama with Curator
      • Using LiteLLM with curator
      • Handling Multimodal Data in Curator
      • Executing LLM-generated code
      • Using HuggingFace inference providers with Curator
    • Data Curation Recipes
      • Generating a diverse QA dataset
      • Using SimpleStrat block for generating diverse data
      • Curate Reasoning data with Claude-3.7 Sonnet
      • Synthetic Data for function calling
    • Finetuning Examples
      • Aspect based sentiment analysis
      • Finetuning a model to identify features of a product
    • API Reference
  • Models
    • Bespoke MiniCheck
      • Self-Hosting
      • Integrations
      • API Service
    • Bespoke MiniChart
    • OpenThinker
Powered by GitBook
On this page
  • Prerequisites
  • Steps
  • Example Output
  • Batch Configuration
  1. BESPOKE CURATOR
  2. Save $$$ on LLM inference

Using Mistral for batch inference

You can use Mistral for batch inference in Curator to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task.

Prerequisites

  • Python 3.10+

  • Curator: Install via pip install bespokelabs-curator

  • Mistral: Mistral API key

Steps

1. Setup environment vars

export MISTRAL_API_KEY=<your_api_key>

2. Create a curator.LLM subclass

Create a class that inherits from curator.LLM. Implement two key methods:

  • prompt(): Generates the prompt for the LLM.

  • parse(): Processes the LLM's response into your desired format.

Here’s the implementation:

"""Example of reannotating the WildChat dataset using curator."""

import logging
from bespokelabs import curator

# To see more detail about how batches are being processed
logger = logging.getLogger("bespokelabs.curator")
logger.setLevel(logging.INFO)


class WildChatReannotator(curator.LLM):
    """A reannotator for the WildChat dataset."""

    def prompt(self, input: dict) -> str:
        """Extract the first message from a conversation to use as the prompt."""
        return input["conversation"][0]["content"]

    def parse(self, input: dict, response: str) -> dict:
        """Parse the model response along with the input to the model into the desired output format.."""
        instruction = input["conversation"][0]["content"]
        return {"instruction": instruction, "new_response": response}

3. Configure the Anthropic model

distiller = WildChatReannotator(model_name="mistral-tiny", 
                                batch=True)

4. Generate Data

Generate the structured data and output the results as a pandas DataFrame:

from datasets import load_dataset
dataset = load_dataset("allenai/WildChat", split="train")
dataset = dataset.select(range(100))

distilled_dataset = distiller(dataset)
print(distilled_dataset)
print(distilled_dataset[0])

Example Output

Using the above example, the output might look like this:

instruction
new_response

Write a very long, elaborate, descriptive and ...

Scene: Omelette Apocalypse\n\n**INT. DINER...

what are you?

I am a large language model, trained by Mistral

Batch Configuration

PreviousUsing Gemini for batch inferenceNextUsing kluster.ai for batch inference

Last updated 1 month ago

Check out complete

batch configuration