Project case study
AI Email Evaluation
At The Home Depot, I am building a test set and evaluation workflow for AI-written business emails. The first question is what a good email needs to contain. A domain expert’s rubric gives us a standard to test against before trying to improve the model.
My role: Evaluation workflow and test-pack development with AI coding agents, using a senior analyst’s rubric and domain feedback.
Watch the interactive walkthroughHow it works
Set the rubric
The team defines what a useful email should contain.
Expert criteria are represented as structured checks. A candidate checklist needs domain review before it becomes the evaluation target.
Compare two emails
Two fictional drafts expose missing explanation.
The example applies the same illustrative criteria to both drafts. The walkthrough does not send internal emails to a public model.
Check the grader
Automatic judgments are compared with reviewed examples.
Calibration compares grader outputs with expert judgments rather than treating model-generated scores as truth.
Keep the release gate closed
A weak evaluator cannot justify optimization.
When agreement falls short, the workflow retains the in-progress status. A higher demo score is not proof that a change is ready to ship.
Test the judge before trusting the judgment.
In my Home Depot work, I have been turning analyst expectations into an email evaluation process. The workflow brings together example emails, a senior analyst’s rubric and a checklist for useful insights. A model-based grader is only helpful if its judgments hold up against reviewed examples.
- Focus
- Example sets, expert rubric, structured grading and calibration
- Status
- In progress; grader calibration incomplete
How it works
Example emails and domain criteria form the evaluation set. Structured grades are compared with expert judgments, so changes to prompts or scoring can be examined against the same target rather than judged from one convincing sample.
A decision that mattered
The initial grader did not meet its own quality threshold. That is a reason to investigate the evaluator before using its score to optimize prompts or approve changes. A polished grading interface should not hide disagreement with expert judgment.
Status
The evaluator is still being calibrated. The failed first check is part of the work, and I am not presenting it as a successful release.