> For the complete documentation index, see [llms.txt](https://docs.bespokelabs.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bespokelabs.ai/bespoke-curator/data-curation-recipes/using-simplestrat-block-for-generating-diverse-data.md).

# Using SimpleStrat block for generating diverse data

## StratifiedGenerator: Generate Balanced Question-Answer Pairs

### Overview

The `StratifiedGenerator` is a powerful tool for creating high-quality question-answer (QA) pairs with balanced and diverse coverage across your input questions. It ensures your generated dataset avoids biases and provides comprehensive representation of your input space.

> 📝 **Research Background**: For a comprehensive understanding of the methodology and theoretical foundation, see the paper: [Stratified Generation for Artificial Data in Question Answering](https://arxiv.org/pdf/2410.09038)

### Installation

```bash
pip install bespokelabs-curator
```

### Quick Start Example

```python
from datasets import Dataset
from bespokelabs.curator.blocks.simplestrat import StratifiedGenerator

# Create a simple dataset of questions
questions = Dataset.from_dict({"question": [f"{i}. Name a periodic element" for i in range(20)]})

# Initialize the generator with your preferred model
generator = StratifiedGenerator(model_name="gpt-4o-mini")

# Generate stratified QA pairs
qa_pairs = generator(questions).dataset

# Examine the results
print(f"Generated {len(qa_pairs)} QA pairs")
print(qa_pairs[0])  # View the first QA pair
```

### How StratifiedGenerator Works

1. **Input**: A `Dataset` containing questions you want to generate answers for
2. **Stratification Process**: The algorithm:
   * Clusters similar questions together
   * Ensures balanced coverage across different question types
   * Prevents overrepresentation of common patterns
3. **Model Integration**: Uses your specified LLM to generate high-quality answers
4. **Output**: Returns a new dataset of QA pairs with well-distributed coverage

### Advanced Usage

#### Customizing the Generator

```python
# With custom parameters
generator = StratifiedGenerator(
    model_name="gpt-4o",  # Use a more powerful model
    generation_params={
        "temperature":0.7,      # Adjust creativity
    }
)
```

#### Saving and Loading Results

```python
# Save your generated QA pairs
qa_pairs.push_to_hub('hf_org/dataset_name')
```

### Performance Considerations

* **Model Selection**: Larger models (e.g., GPT-4) produce higher quality answers but cost more

### Common Applications

* Creating balanced training datasets for QA systems
* Generating diverse test sets for robustness evaluation
* Augmenting existing datasets with additional QA pairs
* Creating instruction-tuning datasets with varied coverage

### Troubleshooting

**Q: My generated answers seem too similar across different questions.**\
A: Try increasing the temperature parameter or the number of clusters.

**Q: I'm getting API errors during generation.**\
A: Try changing the model or backend params. You can check the API reference [here](broken://pages/VAxLcu9BxAB30omwvyzp).

### Additional Resources

* [BespokeLabs Documentation](https://docs.bespokelabs.io/)

Happy data generation!<br>