AdvancedИнженерияclaude

Prompt Evaluations: Anthropic's Official Course

How to measure and improve prompt quality: code-graded evals, model-graded assessments, workbench tests, PromptFoo. Anthropic's methodology for production systems.

4modules

8lessons

240 mintotal time

AI engineers building reliable LLM pipelinesaudience

Module 1

Evaluation Fundamentals

What evals are, why they matter, and how they are structured.

Evaluations 101: Why Measure Prompts

The difference between benchmarks and customer evals. The four eval components: input, golden answer, output, score. Three grading approaches: human-based, code-based, model-based.

30 min

Workbench Evals: Rapid Prototyping

Anthropic Workbench for manual prompt testing. Running evals across multiple test cases. Comparing prompt versions (v1 vs v2). Human scoring on a 1-5 scale.

30 min

Module 2

Code-Graded Evals

Automated programmatic evaluation — fast, scalable, objective.

Code-Graded Eval: From Zero to Baseline

Building a code-graded eval from scratch. Animal leg-counting test. Prompt v1 vs v2 vs v3 (chain-of-thought). Extracting answers from <answer> tags.

30 min

Code-Graded Classification Evals

Evaluating a multi-category classification prompt. Set-based grading for multiple correct categories. Improved prompt with few-shot examples: from 85% to 100% accuracy.

30 min

Module 3

PromptFoo: Scalable Evals

The PromptFoo framework for automated evals with a dashboard, CSV tests, and custom graders.

PromptFoo: Introduction and Basic Configuration

Installing PromptFoo. YAML configuration: providers, prompts, tests. CSV dataset with __expected. Comparing simple_prompt vs better_prompt via the visual dashboard.

30 min

PromptFoo: Custom Code Graders

Writing a Python get_assert() function for PromptFoo. Counting exact word occurrences in text. Comparing Haiku vs Sonnet on the same eval. The GradingResult dict: pass, score, reason.

30 min

Module 4

Model-Graded Evals

LLM-as-judge: when code-graded falls short and subjective criteria are needed.

Model-Graded Evals: LLM as Judge

When model-graded evals are needed. The built-in llm-rubric in PromptFoo. Evaluating a school assistant prompt: does it reject irrelevant questions? Grading for apologetic tone. Comparing three prompt versions.

30 min

Custom LLM Judge: Multi-Metric Scoring

Writing a custom llm_eval() function scoring on multiple metrics (conciseness, accuracy, tone 1-5). get_assert() for PromptFoo. Prefill <json> for reliable JSON output. Comparing basic vs better vs best summarization prompt.

30 min