| Parameter | Type | Default | Description |
|---|---|---|---|
model_name | str | Required | Name of the LLM to use |
response_format | Type[BaseModel] | None | None | Pydantic model specifying the expected response format |
batch | bool | False | Enable batch processing mode |
backend | Optional[str] | None | LLM backend to use ("openai", "litellm", or "vllm"). Auto-determined if None |
generation_params | dict | None | None | Additional parameters for the generation API |
backend_params | BackendParamsType | None | None | Configuration parameters for request processor |
Time window to wait for batch completion.
Note: only valid for some providers.
| ```python # Example: Batch processing configuration backend_params = { "batch_size": 100, "batch_check_interval": 1.0, "delete_successful_batch_files": True, "delete_failed_batch_files": False, } ``` #### Offline Mode Parameters (VLLM) Parameters for local model deployment with VLLM: | Parameter | Type | Description | | ------------------------ | ------- | ------------------------------------------ | | `tensor_parallel_size` | `int` | Number of GPUs for tensor parallelism | | `enforce_eager` | `bool` | Whether to enforce eager execution | | `max_model_length` | `int` | Maximum sequence length for the model | | `max_tokens` | `int` | Maximum tokens for generation | | `min_tokens` | `int` | Minimum tokens for generation | | `gpu_memory_utilization` | `float` | Target GPU memory utilization (0.0 to 1.0) | | `batch_size` | `int` | Batch size for VLLM processing | ```python # Example: VLLM configuration backend_params = { "tensor_parallel_size": 2, "max_model_length": 4096, "max_tokens": 2048, "min_tokens": 1, "gpu_memory_utilization": 0.85, "batch_size": 32 } ``` ### Methods ### prompt() ```python def prompt(self, input: _DictOrBaseModel) -> _DictOrBaseModel ``` Generates a prompt for the LLM based on the input data. **Parameters** * `input`: Input row used to construct the prompt **Returns** A prompt that can be either: 1. A string for a single user prompt 2. A list of dictionaries for multiple messages **Example** ```python def prompt(self, input: dict) -> str: return f"Generate a {input['type']} about {input['topic']}" ``` ### parse() ```python def parse(self, input: _DictOrBaseModel, response: _DictOrBaseModel) -> _DictOrBaseModel ``` Processes the LLM's response and optionally can be used to combine it with the input data. **Parameters** * `input`: Original input row used for the prompt * `response`: Raw response from the LLM **Returns** A parsed output combining the input and response data **Example** ```python def parse(self, input: dict, response: str) -> dict: return { "prompt_topic": input["topic"], "generated_text": response, "timestamp": datetime.now().isoformat() } ``` ### Returns A `CuratorResponse` object which consists statistics about token usage, performance and cost along with the dataset and viewer link. #### `CuratorResponse` #### Attributes #### **Core Data** * dataset (Dataset): The curated dataset * cache\_dir (Optional\[str]): Directory for caching results * failed\_requests\_path (Optional\[Path]): Path to file containing failed requests * viewer\_url (Optional\[str]): URL for Curator Viewer * batch\_mode (bool): Whether the processing was done in batch mode #### Model Information * model\_name (str): Name of the LLM model used * max\_requests\_per\_minute (int | None): Rate limit for requests per minute * max\_tokens\_per\_minute (int | None): Rate limit for tokens per minute #### Statistics * token\_usage (TokenUsage): Statistics about token usage * cost\_info (CostInfo): Information about processing costs * request\_stats (RequestStats): Statistics about request processing * performance\_stats (PerformanceStats): Performance metrics * metadata (Dict\[str, Any]): Additional metadata ### Response Format (Optional) The `response_format` class attribute can be set to a Pydantic model to enforce structured output: ```python from pydantic import BaseModel class RecipeResponse(BaseModel): title: str ingredients: List[str] instructions: List[str] class RecipeGenerator(LLM): response_format = RecipeResponse ``` ### Usage Examples #### Basic Usage ```python class Cuisines(BaseModel): """A list of cuisines.""" cuisines_list: List[str] = Field(description="A list of cuisines.") class CuisineGenerator(curator.LLM): """A cuisine generator that generates diverse cuisines.""" response_format = Cuisines def prompt(self, input: dict) -> str: """Generate a prompt for the cuisine generator.""" return "Generate 10 diverse cuisines." def parse(self, input: dict, response: Cuisines) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return [{"cuisine": t} for t in response.cuisines_list] ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.bespokelabs.ai/bespoke-curator/api-reference.md?ask=