Every time an AI coding tool reads your API docs, it costs tokens. But does the format of those docs matter? We ran 12,000 integration tests across 5 AI models to find out.
Developers still read your API docs. But increasingly, AI coding tools read them first. Every token your documentation consumes is a token that can't be used for reasoning, code generation, or context about the user's project.
Every popular API is in LLM training data. Testing with Stripe's docs means you're measuring memorization, not comprehension. We built a custom API from scratch to eliminate contamination.
Identical information documented in five different formats. Same endpoints, same parameters, same constraints. The only variable is how it's structured.
From 7B parameter models to 200B+. Format preferences change with model capability, and the patterns aren't what you'd predict.
Token efficiency, code generation pass rates, error handling quality, and overall accuracy. Not just "does it work" but "how well, at what cost."
The problem nobody's measuring. Why your doc format creates a hidden tax on every AI interaction with your API.
12,000 integration tests. 5 formats. 5 models. The complete results, with data on token efficiency, code generation quality, and error handling.
The decision framework. How to choose the right format for your team, test your own docs, and optimize without rewriting everything.
Single source, dual output. Author once, generate docs for both humans and AI. Complete implementation with build pipeline.
Where documentation is heading. Plus the complete testing toolkit, the BookClub API reference in all formats, and everything you need to reproduce the research yourself.
Paste your API documentation and see the token cost in real time. The first step to understanding the problem.
Early access — in progress, free updates as chapters ship
Available on Leanpub. All formats: PDF, EPUB, and web. Buy now, get every future update free as new chapters publish.
New findings, chapter releases, and documentation strategy insights. No spam, no fluff.