Pythia: An automated prompt optimization engine

Our AI-driven evaluation loop iteratively improves your language model prompts by analyzing failures, synthesizing improvements, and determining when it's optimized.

Pythia provides clinical, privacy-preserving and locally hosted sensitivity and specificity aware optimization that runs on open source models, meaning optimization stays transparent and within the institution. Optimization can be performed on any scale of model, from one billion to one trillion parameters, allowing optimization to scale with the user's needs.

Get Started

Platform

Our prompt engineering workflow

Run your prompt against a held-out evaluation set and measure performance with configurable metrics. Every result is tracked so the optimizer can compare across iterations.
A specialist LLM examines failures in aggregate, identifies root causes, and generates targeted hypotheses for why the current prompt underperforms.
An improver agent rewrites the prompt to address identified weaknesses. Multiple candidate rewrites are generated and ranked before the best is selected for evaluation.
The controller node decides whether to continue iterating, backtrack to a stronger prior version, or halt when the prompt has converged. So no manual babysitting is required.
Every prompt version, score, and controller decision is logged. Compare runs, export results, and roll back to any prior checkpoint at any time.
Optimization run In progress
Iteration 4 / 10
Current score 0.847
Best score 0.863
Status Synthesizing...
Score history
Iter 1 0.612 baseline
Iter 2 0.741  ↑
Iter 3 0.863  ↑ best
Iter 4 0.847  ↓ backtracking
Error pattern: model missing temporal context in multi-step reasoning tasks
When a new iteration scores lower than the best seen so far, Pythia backtracks to that best version rather than continuing from a degraded prompt. This prevents Pythia drifting away from a strong result and ensures the final output is always the highest-scoring prompt across all iterations instead of picking the final prompt blindly.

Live Demo

Watch the optimization loop run

About

We are building a framework that turns prompt engineering from art into science.

Pythia implements a fully autonomous optimization loop: evaluate, analyze, improve, repeat. Pythia is powered by LangGraph's stateful agentic workflows, so you can stop guessing at prompts and start iterating systematically.

Read the docs 
# Run the optimization loop
from pythia import Pythia

optimized_result = Pythia(
    LLMbackend = backend,
    dev_data_path= dev_eval_dataset,
    val_data_path= val_eval_dataset,
    output_dir = "outputs",
    SOP = "",
    initial_prompt = "Do the patients show signs of a common cold?",
    max_iterations=10,
    sens_threshold = 0.7,
    spec_threshold = 0.7,
)

print(optimized_result.selected_prompt)

Research

Grounded in agentic AI research

Pythia draws on work in automatic prompt optimization, chain-of-thought prompting, and multi-agent LLM systems. The evaluation-synthesis loop is inspired by other prompt optimizing systems, and uses a stateful LangGraph workflow with backtracking and intelligent halting logic to provide systematic senstivity and specificity based clinical prompt optimization.

View on GitHub Our Previous Work

Team

Meet the Pythia Team

Grant Acknowledgements

National Institute on Aging

This project was supported by the National Institute on Aging (R01AG074372 RF1AG074372)

National Institute of Allergy and Infectious Diseases

This project was supported by the National Institute of Allergy and Infectious Diseases (R01AI165535)