> For the complete documentation index, see [llms.txt](https://docs.bespokelabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bespokelabs.ai/bespoke-curator/how-to-guides/online-processing.md).

# Online Processing

This guide demonstrates how to use **Curator** for **online processing** by generating diverse topics for poems and composing poems based on these topics. We'll focus on key parameters to manage rate limits effectively, ensuring smooth operation with LLMs.

## Prerequisites

* Python 3.10+
* Curator: Install via `pip install bespokelabs-curator`
* Access to an LLM provider (e.g., OpenAI or an equivalent API)

## Steps

### 1. Define the Response Formats

**Code Example:**

```python
from typing import List
from pydantic import BaseModel, Field

class Topics(BaseModel):
    """A list of topics."""
    topics_list: List[str] = Field(description="A list of topics.")

class Poems(BaseModel):
    """A list of poems."""
    poems_list: List[str] = Field(description="A list of poems.")
```

***

### 2. Create a Muse

Define a subclass of `curator.LLM` to generate a list of topics for our poems.

**Code Example:**

```python
from bespokelabs import curator

class Muse(curator.LLM):
    response_format = Topics

    def prompt(self, input: dict) -> str:
        return "Generate 10 diverse topics that are suitable for writing poems about."

    def parse(self, input: dict, response: Topics) -> dict:
        return [{"topic": t} for t in response.topics_list]

# Instantiate the Topic Generator
topic_generator = TopicGenerator(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100})

# Generate topics
from datasets import Dataset
topics: Dataset = topic_generator()
print(topics["topic"])  # View the generated topics
```

***

### 3. Create a Poet

Define another subclass of `curator.LLM` to create poems based on the generated topics.

**Code Example:**

```python
class Poet(curator.LLM):
    """A poet that generates poems about given topics."""
    response_format = Poems

    def prompt(self, input: dict) -> str:
        return f"Write two poems about {input['topic']}."

    def parse(self, input: dict, response: Poems) -> dict:
        return [{"topic": input["topic"], "poem": p} for p in response.poems_list]

# Instantiate the Poet
poet = Poet(model_name="gpt-4o-mini", backend_params={"max_requests_per_minute": 100})

# Generate poems based on topics
poems = poet(topics)
print(poems.to_pandas())  # View the poems in a tabular format
```

***

### Example Output

```plaintext
                                           topic                                               poem
0                            Dreams vs. reality  In the realm where dreams take flight,\nWhere ...
1                            Dreams vs. reality  Reality stands with open eyes,\nA weighty thro...
2           Urban loneliness in a bustling city  In the city's heart where shadows blend,\nAmon...
3           Urban loneliness in a bustling city  Among the crowds, I walk alone,\nA sea of face...
```

***

## Online Processing Configuration

Curator provides several configuration parameters to better control your  API usage policies and improve efficiency. 

#### 1. **`max_requests_per_minute`**

**Definition**: The maximum number of API requests allowed per minute.

* **Use Case**: Controls the frequency of requests to prevent exceeding API quotas.
* **Example**: Setting this to `60` ensures only 60 requests are sent per minute.

**Example**:

```python
backend_params={
    "max_requests_per_minute": 60  # Sets maximum requests in a minute.
}
```

**Note**: Retries will also count under the limit set.

#### 2. **`max_tokens_per_minute`**

**Definition**: The maximum number of tokens allowed to be processed per minute.

* **Use Case**: Defines contraint on maximum usable tokens  in a minute.
* **Example**: Setting this to `30,000` ensures token limits are respected for responses.

**Example**:

```python
backend_params={
    "max_tokens_per_minute": 1_000_000  # Sets maximum tokens in a minute.
}
```

**Note**: Token limit will be counted including input and output tokens.

#### 3. **`seconds_to_pause_on_rate_limit`**

**Definition**: The duration (in seconds) to pause when a rate limit is hit.

* **Use Case**: Automatically waits before retrying, ensuring compliance with API limits.
* **Example**: Setting this to `60` pauses the processing for one minute if a rate limit error occurs.

**Example**:

```python
backend_params={
    "seconds_to_pause_on_rate_limit": 60  # Sets seconds to wait before new request on rate limit.
}
```