---
name: ai-apps-that-work-tools
description: Real cost per AI result, model prices and the cost of switching LLMs, model shutdown dates, A/B test sample sizes, LLM-judge agreement, judge prompts and an AI launch checklist, from AI Apps That Work's free API. For questions about what an AI feature costs per result that works, whether a model switch saves money, when a model is retired, how many results an A/B test needs, whether an LLM judge can be trusted, or what an AI app needs before launch.
---

# AI Apps That Work tools

The free tools at https://aiappsthatwork.com/tools?via=skill as a JSON API at https://aiappsthatwork.com. No key and no sign-up. Inputs are computed and thrown away, never stored. Each call is a POST with a JSON body; each response is JSON with the numbers, a one-line `summary` of facts where the tool has one, and a `permalink`: a page on the site that shows the same result, for a person to open.

The same tools are on the MCP server at https://aiappsthatwork.com/mcp (Streamable HTTP, no login). When it is connected, its tools work the same way.

Errors are JSON `{"error": "…"}`: 400 for bad JSON or input outside the schema, 422 for input that can't be computed (such as a model id that isn't priced, with close matches named), 429 over 120 calls a minute.

Model ids are the ones in https://aiappsthatwork.com/models.json (for example `openai/gpt-4o`, `anthropic/claude-sonnet-5.5`); the price and switch tools also accept OpenRouter ids and model names. Prices come from the providers' own pages, checked on the date each response gives.

## Tools

### Real cost per AI result
`POST /api/tools/real-cost-per-ai-result` (MCP tool `real_cost_per_result`). Prices the AI steps behind one request from current list prices (LLM tokens, images, video seconds or a fixed amount) and returns the cost per request, the real cost per result that worked first time (cost per request divided by the worked-first-time share), the monthly spend wasted on results that didn't work, and credit-pricing numbers: margin per result and the credits per request needed to break even or reach a target margin.

- `steps` (value[], required): The AI steps behind one request, priced from /models.json. Their costs add up to the cost per request.
- `workedFirstTimePct` (number, required): Share of results that worked first time: no try again, undo, fix, re-ask or overrule. 1 to 100.
- `requestsPerMonth` (number, required): Requests per month, try-agains included
- `pricing` (object, default {}): Credit pricing. Defaults: $10 for 1,000 credits, 1 credit per request, a 2.9% + $0.30 fee and a 70% target margin.

```sh
curl -s https://aiappsthatwork.com/api/tools/real-cost-per-ai-result?via=skill -H 'content-type: application/json' -d '{"steps":[{"kind":"llm","model":"google/gemini-3.8-flash","inputTokens":4000,"outputTokens":800,"note":"Reads the invoice into fields"},{"kind":"llm","model":"anthropic/claude-haiku-5.5","inputTokens":1500,"outputTokens":100,"note":"Checks the totals"}],"workedFirstTimePct":75,"requestsPerMonth":20000}'
```

### Model switch cost
`POST /api/tools/model-switch-cost` (MCP tool `model_switch_cost`). Compares two priced LLMs on six common task shapes (support answer, invoice extraction, lead scoring, email draft, translation, search re-ranking). Returns list prices, cost per 1,000 requests on each model, the cost ratio, the break-even (how often the new model's results must work first time, relative to the current one, to cost no more per result that works) and the current model's announced shutdown.

- `from` (string, required): The current model: an LLM id from /models.json (openai/gpt-4o), an OpenRouter id or the model's name
- `to` (string, required): The new model: an LLM id from /models.json (openai/gpt-6-luna), an OpenRouter id or the model's name

```sh
curl -s https://aiappsthatwork.com/api/tools/model-switch-cost?via=skill -H 'content-type: application/json' -d '{"from":"openai/gpt-4o","to":"openai/gpt-6-luna"}'
```

### AI model prices
`POST /api/tools/model-prices` (MCP tool `model_prices`). Looks up list prices for LLMs (USD per million input, output and cached input tokens, and context size), image models (USD per image by quality and size) and video models (USD per second by resolution), with each price's source page, the date prices were checked and any announced API shutdown.

- `query` (string): A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the id, name or provider, such as 'claude sonnet'. Omitted: every model.
- `kind` ("llm" | "image" | "video" | "all", default "all"): llm for text models priced per token, image per image, video per second
- `limit` (integer, default 25): Most models to return

```sh
curl -s https://aiappsthatwork.com/api/tools/model-prices?via=skill -H 'content-type: application/json' -d '{"query":"claude","kind":"llm"}'
```

### AI model shutdown dates
`POST /api/tools/model-shutdown-calendar` (MCP tool `model_shutdowns`). Lists announced API shutdown dates for AI models from OpenAI, Anthropic, Google and other providers, with days left, the suggested replacement, the source page, and links to a cost comparison and a side-by-side test of the replacement where this site has them.

- `query` (string): A model id such as 'gpt-4o' (only exact id matches, if any), or words that must all appear in the model ids or provider, such as 'anthropic'. Omitted: every model.
- `includePast` (boolean, default false): Include shutdowns whose date has passed
- `today` (string): Date to count days from, YYYY-MM-DD. Omitted: today in UTC.

```sh
curl -s https://aiappsthatwork.com/api/tools/model-shutdown-calendar?via=skill -H 'content-type: application/json' -d '{"query":"gpt-4o"}'
```

### Prompt or model A/B test sample size
`POST /api/tools/prompt-ab-test-sample-size` (MCP tool `ab_test_sample_size`). Returns how many results each version of a prompt or model needs before a difference in how often results work first time is statistically real (two-proportion z-test, two-sided), the total across both versions, and how many days that takes at a given daily volume.

- `baselinePct` (number, required): Share of version A's (the current prompt or model) results that work first time, percent (above 0, below 100)
- `targetPct` (number, required): Share of version B's results that work first time, percent (above 0, below 100)
- `confidencePct` (90 | 95 | 99, default 95): Confidence level, percent. Two-sided.
- `powerPct` (80 | 90, default 80): Chance of detecting a real difference of this size, percent
- `resultsPerDay` (number): Results per day across both versions, split evenly. Gives the number of days.

```sh
curl -s https://aiappsthatwork.com/api/tools/prompt-ab-test-sample-size?via=skill -H 'content-type: application/json' -d '{"baselinePct":60,"targetPct":70,"resultsPerDay":100}'
```

### LLM judge agreement
`POST /api/tools/llm-judge-agreement` (MCP tool `judge_agreement`). Compares a person's verdicts with an AI judge's verdicts on the same results. Returns the agreement rate, Cohen's kappa with confidence intervals, precision and recall per verdict, the agreement of a judge that always gives the most common verdict, a trust verdict (kappa 0.6 or more with at least 80% recall on every verdict; 0.4 to 0.6 is borderline; under 30 results is too few) and the rows where the two disagree.

- `rows` (object[]): One row per result. Common words are mapped onto two verdicts: worked (worked, pass, yes, true, 1, good, kept, accepted, ok) and didn't work (didn't work, fail, no, false, 0, bad, redo, retry, rejected, wrong). Other labels are compared as written.
- `csv` (string): The same data as CSV or TSV text instead of rows: columns named human and judge (and optionally note), or the first two columns in that order.

```sh
curl -s https://aiappsthatwork.com/api/tools/llm-judge-agreement?via=skill -H 'content-type: application/json' -d '{"rows":[{"human":"worked","judge":"pass"},{"human":"worked","judge":"fail","note":"Correct two-line answer the judge called too short"},{"human":"didn'\''t work","judge":"pass","note":"Quoted a 30-day refund window; the policy is 14 days"},{"human":"didn'\''t work","judge":"fail"}]}'
```

### Judge prompt builder
`POST /api/tools/judge-prompt-builder` (MCP tool `build_judge_prompt`). Builds a pass/fail LLM-as-a-judge prompt for one kind of AI step (make, answer, transform, extract, judge, find, predict, plan, act or learn). Returns the prompt with {{input}} and {{result}} placeholders, numbered pass conditions (the block's starter conditions unless you send your own), up to 3 pass and 3 fail examples interleaved, and a JSON or XML answer format.

- `block` ("make" | "answer" | "transform" | "extract" | "judge" | "find" | "predict" | "plan" | "act" | "learn", required): What the AI step does. make: creates new content from a request (an image, a post, slides, advice); answer: replies to a question that has a right answer (a support reply, a policy question, a how-to); transform: reshapes material faithfully (a rewrite, a translation, text to speech); extract: pulls out what's in the input (fields from a PDF, speech to text, text from a photo); judge: gives a label, score or choice that can be checked now (moderation, routing, a guardrail); find: picks the few relevant items out of many (search, recommendations); predict: makes an estimate that can only be checked later (will this lead buy, is this fraud, when will it arrive); plan: turns a goal into steps (a research plan, a multi-step task); act: takes one action with side effects (send an email, take a payment, call an API); learn: updates a model or prompt from examples (a voice clone, prompt tuning).
- `featureDescription` (string, default ""): Who uses the feature, what they ask for and what a good result gives them
- `criteria` (string[]): Pass conditions, one per item. Omitted: the starter conditions for the block.
- `workedExamples` (object[], default []): Real results that worked first time; labelled pass in the prompt
- `failedExamples` (object[], default []): Real results that didn't work; labelled fail in the prompt
- `outputFormat` ("json" | "xml-tags", default "json"): How the judge answers
- `includeReasoning` (boolean, default true): Whether the judge writes a short reason before its verdict

```sh
curl -s https://aiappsthatwork.com/api/tools/judge-prompt-builder?via=skill -H 'content-type: application/json' -d '{"block":"extract","featureDescription":"An invoice reader for a bookkeeping app. It fills in supplier, invoice number, date, line items, VAT and total."}'
```

### AI launch checklist
`POST /api/tools/ai-launch-checklist` (MCP tool `launch_checklist`). The 20 AI-specific launch checks for an AI app (spending caps, rate limits, server-side keys, credit charging, version tags, records of which results worked first time, real cost per result, switching models safely). Returns the checks not yet done, each with why it matters and a request for a coding agent, plus one combined request.

- `done` (string[], default []): Ids of checks already done. Ids: spend-cap, rate-limits, server-keys, capped-retries, provider-budget, every-call-charged, deduct-server-side, refund-failures, one-charge-per-request, output-metadata, versioned-regenerations, outcome-events, events-by-version, step-costs, wasted-spend, who-it-fails-for, prompts-in-git, switch-flags, replay-set, shutdown-plan.
- `group` ("cost" | "charging" | "traceability" | "signals" | "change"): Only checks in one group: cost (cost and abuse), charging (charging for ai use), traceability (a record of every ai result), signals (what worked, for whom, at what cost), change (switching safely)

```sh
curl -s https://aiappsthatwork.com/api/tools/ai-launch-checklist?via=skill -H 'content-type: application/json' -d '{"done":["spend-cap","server-keys"]}'
```

## Reference
- Full input and output schemas: https://aiappsthatwork.com/openapi.json
- Model prices and shutdowns as one JSON file: https://aiappsthatwork.com/models.json
- Any page on the site as Markdown: add .md to its URL (https://aiappsthatwork.com/index.md for the home page)