AGENTS.md for an AI email agent
Drafts replies to customer emails and sends the routine ones itself. Paste this section into AGENTS.md or CLAUDE.md, and Claude Code, Cursor or Codex will build the AI parts so you can see which results users had to redo, and fix those first.
## AI features: AI email agent Drafts replies to customer emails and sends the routine ones itself. ### Steps 1. **find**: pull the thread, the customer's account and the relevant policies 2. **draft**: write the reply 3. **send check**: decide whether it can go out without a person 4. **send**: send it, or file it as a draft for a person ### Every AI call - Call models only from the server. API keys never reach the browser. - Give every AI result a row in `ai_results`: id, user_id, feature, step, model, prompt_version, settings, input reference, output reference, cost_usd, latency_ms, created_at. - Never overwrite a result. A retry or regeneration is a new row with `retry_of` set to the earlier result's id. - Prompts live in versioned files (`prompts/<feature>/<step>/v<N>.md`). Don't edit a version that has shipped; add a new one. ### What counts as "worked first time" in this app Record these as rows in `ai_events` (result_id, user_id, event, created_at), written by the server, not by a browser analytics script. Worked: - the draft was sent without edits - the customer didn't reply with a complaint or correction Needed a redo: - the user rewrote more than a sentence before sending (store the edit size) - they discarded the draft - they undid an automatic send - the customer replied that it was wrong Where users give no signal: Each week, read 20 emails the agent sent on its own and record whether a person would have sent them as they were. ### Before changing a prompt or a model 1. Build a test set from the last 150 real threads, with the reply a person finally sent. 2. Replay it on the change. Compare how often results work first time and the cost per result that works, step by step. 3. Ship to 10% of users behind a flag. Keep it only if users redo less and the cost per result that works doesn't rise. 4. Write down what changed and why in `prompts/CHANGELOG.md`. ### Cost guardrails - A spending cap per user per day, checked before each AI call. - Rate limits on every AI endpoint. - Retries capped at 2, each with a timeout. - automatic sends only to existing customers, never about money or legal matters - at most 50 automatic sends per account per day - every automatic send can be undone for 30 seconds
What each part is for
- Every AI call
- One row per result, never overwritten, tied to its prompt version and model. Without it, nobody can say which change broke what.
- What counts as worked first time
- The signals an email agent gives when a result worked or needed a redo, recorded by the server. This is the number to improve.
- Before changing a prompt or a model
- Replay real requests first, then ship to a slice of users. Your agent stops changing prompts by feel.
- Cost guardrails
- Caps and limits so one heavy user or a retry loop can't run up the bill.
Run the AI launch checklistWork out your real cost per result
Questions
Where do I put this?
In AGENTS.md at the root of your repo; Codex, Cursor and most coding agents read it. Claude Code reads CLAUDE.md, so paste it there too, or have CLAUDE.md point to AGENTS.md.
Why does it say what counts as a redo for an email agent?
Because that's how you find out whether the AI worked without asking users to rate it. Each kind of app shows a failed result differently: asking again, editing, regenerating, overriding. Recording those as events, tied to the result's prompt version and model, is what lets your agent fix the cases that fail and prove the fix worked.
Do I need evals for this?
Not to start. Most of it runs on what users already do. Each week, read 20 emails the agent sent on its own and record whether a person would have sent them as they were.