Diverse QA dataset

This recipe builds a set of question and answer pairs that covers many subjects. It follows the approach of the CAMEL dataset. The pipeline first generates subjects, then subsubjects for each subject, and then questions and answers for each subsubject. The pairs are “ungrounded”, which means the LLM writes them instead of taking them from existing text. The full code is in the ungrounded QA example on GitHub.

Set up the environment

# pip install bespokelabs-curator
import os
from typing import List

from pydantic import BaseModel, Field

from bespokelabs import curator

# Remove this line if you do not want to use the Curator Viewer.
os.environ["CURATOR_VIEWER"] = "1"

Define the data models

These Pydantic models describe the structured output of each step.

class Subject(BaseModel):
    """A single subject."""

    subject: str = Field(description="A subject")

class Subjects(BaseModel):
    """A list of subjects."""

    subjects: List[Subject] = Field(description="A list of subjects")

class QA(BaseModel):
    """A question and answer pair."""

    question: str = Field(description="A question")
    answer: str = Field(description="An answer")

class QAs(BaseModel):
    """A list of question and answer pairs."""

    qas: List[QA] = Field(description="A list of QAs")

Generate subjects

This step generates broad subjects, e.g., “Math”.

class SubjectGenerator(curator.LLM):
    """Generate diverse subjects."""

    response_format = Subjects

    def prompt(self, input: dict) -> str:
        return "Generate a diverse list of 3 subjects. Keep it high-level (e.g. Math, Science)."

    def parse(self, input: dict, response: Subjects) -> list:
        return [{"subject": s.subject} for s in response.subjects]

Generate subsubjects

This step takes each subject and generates narrower topics. For “Math”, it might return “Calculus”, “Algebra”, and “Statistics”.

class SubsubjectGenerator(curator.LLM):
    """Generate diverse subsubjects for a given subject."""

    response_format = Subjects

    def prompt(self, input: dict) -> str:
        return f"For the given subject {input['subject']}. Generate 3 diverse subsubjects. No explanation."

    def parse(self, input: dict, response: Subjects) -> list:
        return [{"subject": input["subject"], "subsubject": s.subject} for s in response.subjects]

Generate questions and answers

This step writes question and answer pairs for each subsubject. Each row keeps its subject and subsubject.

class QAGenerator(curator.LLM):
    """Generate diverse questions and answers for a given subsubject."""

    response_format = QAs

    def prompt(self, input: dict) -> str:
        return f"For the given subsubject {input['subsubject']}. Generate 3 diverse questions and answers. No explanation."

    def parse(self, input: dict, response: QAs) -> list:
        return [
            {
                "subject": input["subject"],
                "subsubject": input["subsubject"],
                "question": qa.question,
                "answer": qa.answer,
            }
            for qa in response.qas
        ]

Run the pipeline

subject_generator = SubjectGenerator(model_name="gpt-4o-mini")
subjects = subject_generator()

subsubject_generator = SubsubjectGenerator(model_name="gpt-4o-mini")
subsubjects = subsubject_generator(subjects)

qa_generator = QAGenerator(model_name="gpt-4o-mini")
qas = qa_generator(subsubjects)

# Strip whitespace from the answers.
qa_dataset = qas.dataset.map(lambda row: {"answer": row["answer"].strip()}, num_proc=2)

print(qa_dataset.to_pandas())

Example output

With 3 subjects, 3 subsubjects each, and 3 pairs each, the pipeline returns 27 rows. The first rows look like this.

subject	subsubject	question	answer
Mathematics	Calculus	What is the derivative of the function f(x) = e^x?	The derivative of f(x) = e^x is also e^x.
Mathematics	Calculus	What is the fundamental theorem of calculus?	It states that differentiation and integration are inverse processes.
Mathematics	Linear Algebra	What are eigenvalues and eigenvectors?	For a matrix A, if Av = λv, then λ is an eigenvalue and v is an eigenvector.

Customize the pipeline

To change a step, subclass it and override prompt.

class CustomSubjectGenerator(SubjectGenerator):
    def prompt(self, input: dict) -> str:
        return "Generate a diverse list of 5 subjects spanning sciences, arts, and humanities."


class ComplexQAGenerator(QAGenerator):
    def prompt(self, input: dict) -> str:
        return f"""For the given subsubject {input['subsubject']}, generate 3 diverse questions and answers.
Include at least one factual question, one conceptual question, and one application question.
Make the questions challenging but clear."""

You can use this pipeline to make training data for question answering models or to test how much a model knows across subjects. You can extend it with question types, difficulty levels, or new subject areas.

Get started

Core features

Guides

Recipes

Fine-tuning

Reference

Example output

Customize the pipeline

​Example output

​Customize the pipeline

Example output

Customize the pipeline