Online Processing

This guide demonstrates how to use Curator for online processing by generating diverse topics for poems and composing poems based on these topics. We'll focus on key parameters to manage rate limits effectively, ensuring smooth operation with LLMs.

Prerequisites

• Python 3.10+

• Curator: Install via pip install bespokelabs-curator

• Access to an LLM provider (e.g., OpenAI or an equivalent API)

Steps

1. Define the Response Formats

Code Example:

from typing import List
from pydantic import BaseModel, Field

class Topics(BaseModel):
    """A list of topics."""
    topics_list: List[str] = Field(description="A list of topics.")

class Poems(BaseModel):
    """A list of poems."""
    poems_list: List[str] = Field(description="A list of poems.")

---

2. Create a Muse

Define a subclass of curator.LLM to generate a list of topics for our poems.

Code Example:

from bespokelabs import curator

class Muse(curator.LLM):
    response_format = Topics

    def prompt(self, input: dict) -> str:
        return "Generate 10 diverse topics that are suitable for writing poems about."

    def parse(self, input: dict, response: Topics) -> dict:
        return [{"topic": t} for t in response.topics_list]

# Instantiate the Topic Generator
topic_generator = TopicGenerator(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100})

# Generate topics
from datasets import Dataset
topics: Dataset = topic_generator()
print(topics["topic"])  # View the generated topics

---

3. Create a Poet

Define another subclass of curator.LLM to create poems based on the generated topics.

Code Example:

class Poet(curator.LLM):
    """A poet that generates poems about given topics."""
    response_format = Poems

    def prompt(self, input: dict) -> str:
        return f"Write two poems about {input['topic']}."

    def parse(self, input: dict, response: Poems) -> dict:
        return [{"topic": input["topic"], "poem": p} for p in response.poems_list]

# Instantiate the Poet
poet = Poet(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100})

# Generate poems based on topics
poems = poet(topics)
print(poems.to_pandas())  # View the poems in a tabular format

---

Example Output

                                           topic                                               poem
0                            Dreams vs. reality  In the realm where dreams take flight,\nWhere ...
1                            Dreams vs. reality  Reality stands with open eyes,\nA weighty thro...
2           Urban loneliness in a bustling city  In the city's heart where shadows blend,\nAmon...
3           Urban loneliness in a bustling city  Among the crowds, I walk alone,\nA sea of face...

---

Online Processing Configuration

Curator provides several configuration parameters to better control your API usage policies and improve efficiency.

1. max_requests_per_minute

Definition: The maximum number of API requests allowed per minute.

• Use Case: Controls the frequency of requests to prevent exceeding API quotas.

• Example: Setting this to 60 ensures only 60 requests are sent per minute.

Example:

backend_params={
    "max_requests_per_minute": 60  # Sets maximum requests in a minute.
}

Note: Retries will also count under the limit set.

2. max_tokens_per_minute

Definition: The maximum number of tokens allowed to be processed per minute.

• Use Case: Defines contraint on maximum usable tokens in a minute.

• Example: Setting this to 30,000 ensures token limits are respected for responses.

Example:

backend_params={
    "max_tokens_per_minute": 1_000_000  # Sets maximum tokens in a minute.
}

Note: Token limit will be counted including input and output tokens.

3. seconds_to_pause_on_rate_limit

Definition: The duration (in seconds) to pause when a rate limit is hit.

• Use Case: Automatically waits before retrying, ensuring compliance with API limits.

• Example: Setting this to 60 pauses the processing for one minute if a rate limit error occurs.

Example:

backend_params={
    "seconds_to_pause_on_rate_limit": 60  # Sets seconds to wait before new request on rate limit.
}