AGENTS.md for an AI support bot
Answers customer questions from your help docs and hands over to a person when it can't. Paste this section into AGENTS.md or CLAUDE.md, and Claude Code, Cursor or Codex will build the AI parts so you can see which results users had to redo, and fix those first.
## AI features: AI support bot Answers customer questions from your help docs and hands over to a person when it can't. ### Steps 1. **find**: pick the 3–5 help-doc passages that match the question 2. **answer**: write the reply from those passages only, with links 3. **hand-off**: decide whether a person should take over ### Every AI call - Call models only from the server. API keys never reach the browser. - Give every AI result a row in `ai_results`: id, user_id, feature, step, model, prompt_version, settings, input reference, output reference, cost_usd, latency_ms, created_at. - Never overwrite a result. A retry or regeneration is a new row with `retry_of` set to the earlier result's id. - Prompts live in versioned files (`prompts/<feature>/<step>/v<N>.md`). Don't edit a version that has shipped; add a new one. ### What counts as "worked first time" in this app Record these as rows in `ai_events` (result_id, user_id, event, created_at), written by the server, not by a browser analytics script. Worked: - the customer said thanks, or left without asking again within 10 minutes - they clicked a linked article - they marked it helpful Needed a redo: - they asked the same question again in other words - they corrected the answer - they asked for a person - they reopened a ticket within 3 days Where users give no signal: Each week, read 20 hand-off decisions by hand and record whether a person was really needed. ### Before changing a prompt or a model 1. Build a test set from the last 200 real questions, including every one that ended in a redo. 2. Replay it on the change. Compare how often results work first time and the cost per result that works, step by step. 3. Ship to 10% of users behind a flag. Keep it only if users redo less and the cost per result that works doesn't rise. 4. Write down what changed and why in `prompts/CHANGELOG.md`. ### Cost guardrails - A spending cap per user per day, checked before each AI call. - Rate limits on every AI endpoint. - Retries capped at 2, each with a timeout. - at most 30 questions per user per hour - answers capped at 400 tokens - retrieval limited to published help articles
What each part is for
- Every AI call
- One row per result, never overwritten, tied to its prompt version and model. Without it, nobody can say which change broke what.
- What counts as worked first time
- The signals an support bot gives when a result worked or needed a redo, recorded by the server. This is the number to improve.
- Before changing a prompt or a model
- Replay real requests first, then ship to a slice of users. Your agent stops changing prompts by feel.
- Cost guardrails
- Caps and limits so one heavy user or a retry loop can't run up the bill.
Run the AI launch checklistWork out your real cost per result
Questions
Where do I put this?
In AGENTS.md at the root of your repo; Codex, Cursor and most coding agents read it. Claude Code reads CLAUDE.md, so paste it there too, or have CLAUDE.md point to AGENTS.md.
Why does it say what counts as a redo for an support bot?
Because that's how you find out whether the AI worked without asking users to rate it. Each kind of app shows a failed result differently: asking again, editing, regenerating, overriding. Recording those as events, tied to the result's prompt version and model, is what lets your agent fix the cases that fail and prove the fix worked.
Do I need evals for this?
Not to start. Most of it runs on what users already do. Each week, read 20 hand-off decisions by hand and record whether a person was really needed.