AGENTS.md for an AI search feature
Finds the right items, documents or answers from a natural-language query. Paste this section into AGENTS.md or CLAUDE.md, and Claude Code, Cursor or Codex will build the AI parts so you can see which results users had to redo, and fix those first.
## AI features: AI search feature Finds the right items, documents or answers from a natural-language query. ### Steps 1. **understand**: turn the query into filters and a search phrase 2. **find**: fetch candidates and re-rank the top 10 ### Every AI call - Call models only from the server. API keys never reach the browser. - Give every AI result a row in `ai_results`: id, user_id, feature, step, model, prompt_version, settings, input reference, output reference, cost_usd, latency_ms, created_at. - Never overwrite a result. A retry or regeneration is a new row with `retry_of` set to the earlier result's id. - Prompts live in versioned files (`prompts/<feature>/<step>/v<N>.md`). Don't edit a version that has shipped; add a new one. ### What counts as "worked first time" in this app Record these as rows in `ai_events` (result_id, user_id, event, created_at), written by the server, not by a browser analytics script. Worked: - the user opened a result in the top 3 and didn't search again - they bought, saved or used the result Needed a redo: - they searched again within 2 minutes - they opened nothing - they scrolled past the first page ### Before changing a prompt or a model 1. Build a test set from the last 300 real queries, with the result each user finally used. 2. Replay it on the change. Compare how often results work first time and the cost per result that works, step by step. 3. Ship to 10% of users behind a flag. Keep it only if users redo less and the cost per result that works doesn't rise. 4. Write down what changed and why in `prompts/CHANGELOG.md`. ### Cost guardrails - A spending cap per user per day, checked before each AI call. - Rate limits on every AI endpoint. - Retries capped at 2, each with a timeout. - at most 60 searches per user per minute - re-ranking only the top 20 candidates
What each part is for
- Every AI call
- One row per result, never overwritten, tied to its prompt version and model. Without it, nobody can say which change broke what.
- What counts as worked first time
- The signals an AI search feature gives when a result worked or needed a redo, recorded by the server. This is the number to improve.
- Before changing a prompt or a model
- Replay real requests first, then ship to a slice of users. Your agent stops changing prompts by feel.
- Cost guardrails
- Caps and limits so one heavy user or a retry loop can't run up the bill.
Run the AI launch checklistWork out your real cost per result
Questions
Where do I put this?
In AGENTS.md at the root of your repo; Codex, Cursor and most coding agents read it. Claude Code reads CLAUDE.md, so paste it there too, or have CLAUDE.md point to AGENTS.md.
Why does it say what counts as a redo for an AI search feature?
Because that's how you find out whether the AI worked without asking users to rate it. Each kind of app shows a failed result differently: asking again, editing, regenerating, overriding. Recording those as events, tied to the result's prompt version and model, is what lets your agent fix the cases that fail and prove the fix worked.
Do I need evals for this?
Not to start. Most of it runs on what users already do. Where users give no signal, check a small sample by hand each week.