Prompt Evaluations: Anthropic's Official Course

PromptFoo: Introduction and Basic Configuration

Installing PromptFoo. YAML configuration: providers, prompts, tests. CSV dataset with __expected. Comparing simple_prompt vs better_prompt via the visual dashboard.

PromptFoo: Custom Code Graders

Writing a Python get_assert() function for PromptFoo. Counting exact word occurrences in text. Comparing Haiku vs Sonnet on the same eval. The GradingResult dict: pass, score, reason.

Model-Graded Evals: LLM as Judge

When model-graded evals are needed. The built-in llm-rubric in PromptFoo. Evaluating a school assistant prompt: does it reject irrelevant questions? Grading for apologetic tone. Comparing three prompt versions.

Custom LLM Judge: Multi-Metric Scoring

Writing a custom llm_eval() function scoring on multiple metrics (conciseness, accuracy, tone 1-5). get_assert() for PromptFoo. Prefill <json> for reliable JSON output. Comparing basic vs better vs best summarization prompt.