vLLM

You can use vLLM with Curator in two ways.

Offline mode. vLLM loads the model on your machine, inside your Python process.
Online mode. vLLM runs as a separate server, and Curator sends requests to it.

This guide generates recipes with structured output in both modes.

Prerequisites

Python 3.10 or later.
Curator, installed with pip install bespokelabs-curator.
vLLM, installed with pip install vllm, and a GPU.

Define the model and the generator

Both modes use this code.

from typing import List

from pydantic import BaseModel, Field

from bespokelabs import curator


class Recipe(BaseModel):
    title: str = Field(description="Title of the recipe")
    ingredients: List[str] = Field(description="List of ingredients needed")
    instructions: List[str] = Field(description="Step by step cooking instructions")
    prep_time: int = Field(description="Preparation time in minutes")
    cook_time: int = Field(description="Cooking time in minutes")
    servings: int = Field(description="Number of servings")


class RecipeGenerator(curator.LLM):
    response_format = Recipe

    def prompt(self, input: dict) -> str:
        return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic."

    def parse(self, input: dict, response: Recipe) -> dict:
        return {
            "title": response.title,
            "ingredients": response.ingredients,
            "instructions": response.instructions,
            "prep_time": response.prep_time,
            "cook_time": response.cook_time,
            "servings": response.servings,
        }


cuisines = [{"cuisine": c} for c in ["Italian", "Chinese", "Mexican"]]

Offline mode

Set backend="vllm" and pass the vLLM settings in backend_params.

generator = RecipeGenerator(
    model_name="Qwen/Qwen2.5-3B-Instruct",
    backend="vllm",
    backend_params={
        "tensor_parallel_size": 1,  # Set this to the number of GPUs.
        "gpu_memory_utilization": 0.7,
    },
)

recipes = generator(cuisines)
print(recipes.dataset.to_pandas())

Curator uses guided decoding in vLLM for structured output. If the model does not support it, Curator logs a warning.

Offline settings

Parameter	Default	Description
`tensor_parallel_size`	`1`	The number of GPUs for tensor parallelism.
`gpu_memory_utilization`	`0.95`	The share of GPU memory vLLM can use, from 0 to 1.
`max_model_length`	`4096`	The longest sequence the model can handle.
`max_tokens`	`4096`	The most tokens to generate.
`min_tokens`	`1`	The fewest tokens to generate.
`enforce_eager`	`False`	Whether to force eager execution.
`batch_size`	`256`	The number of prompts vLLM processes together.
`dtype`	`"auto"`	The data type of the model weights.

Online mode

Start the vLLM server

vllm serve Qwen/Qwen2.5-3B-Instruct \
    --host localhost \
    --port 8787 \
    --api-key token-abc123

Connect Curator to the server

Use the LiteLLM backend with the hosted_vllm/ prefix on the model name.

import os

os.environ["HOSTED_VLLM_API_KEY"] = "token-abc123"

generator = RecipeGenerator(
    model_name="hosted_vllm/Qwen/Qwen2.5-3B-Instruct",
    backend="litellm",
    backend_params={
        "base_url": "http://localhost:8787/v1",
        "request_timeout": 30,
    },
)

recipes = generator(cuisines)
print(recipes.dataset.to_pandas())

Example output

Each row holds one recipe, like this one.

{
    "title": "Spicy Szechuan Noodles",
    "ingredients": [
        "400g wheat noodles",
        "2 tbsp Szechuan peppercorns",
        "3 cloves garlic, minced",
        "2 tbsp soy sauce"
    ],
    "instructions": [
        "Boil noodles according to package instructions",
        "Heat oil in a wok over medium-high heat",
        "Add peppercorns and garlic, stir-fry until fragrant",
        "Add noodles and soy sauce, toss to combine"
    ],
    "prep_time": 15,
    "cook_time": 20,
    "servings": 4
}

Get started

Core features

Guides

Recipes

Fine-tuning

Reference

Prerequisites

Define the model and the generator

Offline mode

Offline settings

Online mode

Example output

​Prerequisites

​Define the model and the generator

​Offline mode

​Offline settings

​Online mode

​Example output

Prerequisites

Define the model and the generator

Offline mode

Offline settings

Online mode

Example output