Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar

Lenny's Podcast

7 chapters7 takeaways14 key terms5 questions

Overview

This video explains the critical role of "evals" (evaluations) in building successful AI products. Evals are a systematic process for measuring and improving AI application performance, moving beyond simple "vibe checks" or basic unit tests. The process involves analyzing real user interaction data, identifying errors through "open coding," categorizing these errors into "axial codes," and then using this analysis to build automated tests or "LLM judges." This structured approach helps product builders gain confidence, iterate effectively, and ensure their AI applications meet user needs and business goals. The speakers emphasize that while AI can assist in the process, human oversight and domain expertise are crucial for effective evals.

How was this?

Save this permanently with flashcards, quizzes, and AI chat

Chapters

Evals are a systematic way to measure and improve AI applications, offering a high return on investment for product builders.
The goal of evals is not perfection, but actionable product improvement.
Common misconceptions include believing AI can fully automate the evaluation process or that evals are overly complex and time-consuming.
Effective evals provide a feedback signal for confident iteration and experimentation.

Understanding evals is crucial because they are becoming a fundamental skill for anyone building AI products, enabling them to create more reliable and effective applications.

A real estate AI assistant that fails to correctly identify available apartments with specific features, highlighting the need for systematic measurement beyond initial prompt fixes.

Evals are a broad spectrum of methods for measuring AI application quality, not just simple unit tests.
They can include checking for non-negotiable functionalities, assessing performance on ambiguous tasks, and tracking user satisfaction metrics over time.
Evals help identify new user cohorts or unexpected data distributions that might require product adjustments.
The process should be manageable and not prohibitively expensive to implement.

Recognizing the full scope of evals helps product builders avoid limiting their evaluation methods to basic tests and instead embrace a more comprehensive approach to understanding AI performance.

For a real estate AI agent, evals could range from checking if it correctly books appointments (like a unit test) to assessing how well it handles vague user requests or adapts to new user types.

The first step in building evals is data analysis, specifically looking at what's going wrong in real user interactions.
Observability tools can log detailed interaction traces, showing system prompts, tool calls, and AI responses.
Product managers should be involved in error analysis because they understand the user experience and desired product outcomes.
Initial error analysis involves manually reviewing traces and writing notes about observed issues (open coding).

Grounding your evaluation process in actual user data and identifying real-world errors is essential for building effective and relevant tests, rather than relying on assumptions.

Reviewing logs for an AI property manager and noting that it failed to offer a virtual tour when asked, or that a text message interaction became garbled and unmanageable.

Open coding is the process of writing free-form notes on errors observed during data analysis.
The goal is to capture the first significant error encountered in a trace and then move on, rather than trying to document every single issue.
A "benevolent dictator" is a single person, ideally with domain expertise, appointed to lead the open coding process to avoid committee paralysis and maintain efficiency.
This approach makes the process tractable and prevents it from becoming too expensive.

This structured yet efficient approach to error identification ensures that valuable insights are captured without getting bogged down in excessive debate or analysis paralysis.

A product manager reviewing AI interactions and noting 'conversation flow is janky because of text message' or 'did not confirm call transfer with user' to document specific issues.

After open coding, errors are synthesized into broader categories called axial codes.
LLMs can be used to help categorize the free-form notes (open codes) into these axial codes, identifying common failure modes.
Axial codes provide a structured way to understand the prevalence of different types of problems.
The goal is to move from a mess of individual issues to actionable clusters of failure modes.

Categorizing errors transforms raw observations into meaningful insights, allowing product teams to prioritize and address the most significant issues impacting their AI application.

Using an LLM to categorize open codes into axial codes such as 'capability limitations,' 'misrepresentation,' 'human handoff issues,' or 'communication quality.'

Once failure modes are identified and quantified, automated evaluators can be built to check for their prevalence at scale.
These can be code-based (like traditional tests) for simpler checks or LLM-based "judges" for more complex, nuanced evaluations.
LLM judges are trained on specific failure modes and output binary pass/fail results, making them tightly scoped and reliable.
It's crucial to align LLM judges with human judgment by measuring their agreement and iterating on the judge's prompt.

Automated evaluators, especially LLM judges, allow for continuous monitoring of AI performance in production, providing confidence in deployments and enabling rapid detection of regressions.

Creating an LLM judge prompt that determines if an AI assistant correctly handled a human handoff by checking for specific conditions like explicit human request ignored, policy-mandated transfer, or sensitive resident issues.

It's critical to validate LLM judges by comparing their outputs to human judgments, using metrics beyond simple agreement percentage.
A confusion matrix helps identify specific types of misalignments between the human reviewer and the LLM judge.
Evals, especially LLM judges derived from data analysis, function as dynamic, data-driven product requirements documents.
The process of building and refining evals is iterative and should be grounded in real user data, not just theoretical expectations.

Proper validation ensures that the automated evals are trustworthy and accurately reflect desired product behavior, preventing the deployment of flawed AI systems.

Using a confusion matrix to analyze where a human reviewer and an LLM judge disagree on whether a human handoff error occurred, and then refining the LLM judge prompt based on these discrepancies.

Key takeaways

1Evals are a systematic process for measuring and improving AI applications, essential for product success.
2Start evals by analyzing real user data to identify actual errors, rather than creating tests in a vacuum.
3Open coding (manual annotation of errors) is a crucial first step, followed by synthesis into axial codes (failure categories).
4Leverage LLMs for tasks like categorizing errors (axial coding) and acting as judges, but always maintain human oversight.
5Appoint a "benevolent dictator" to streamline the error analysis process and avoid committee delays.
6Automated evaluators (like LLM judges) should be rigorously validated against human judgment to ensure reliability.
7Evals derived from data analysis serve as living, data-driven product requirements, guiding development and ensuring quality.

Key terms

EvalsAI ApplicationVibe ChecksError AnalysisObservability ToolsTraceSystem PromptOpen CodingAxial CodesBenevolent DictatorAutomated EvaluatorLLM JudgeTheoretical SaturationConfusion Matrix

Test your understanding

1What is the primary goal of conducting evals for an AI product?
2How does the process of error analysis using open coding differ from traditional software unit testing?
3Why is the concept of a "benevolent dictator" important in the context of evals?
4What is the role of LLMs in the eval process, and what are the limitations of relying solely on AI for evaluations?
5How can product builders ensure that their LLM judges are reliable and accurately reflect desired AI behavior?