Preparing Our Docs for AI Agents: How and Why We Implemented llms.txt in Astro

Question

Html2Pdf is officially live in production:

• Live Web App & Interactive Demo: html2pdf.akcraft.app

• Official TypeScript / ESM SDK: @akcraft/html2pdf-sdk

• Live Web API: api.html2pdf.akcraft.app

If you have used AI coding assistants like Cursor, Claude Code, GitHub Copilot, or Antigravity recently, you have likely noticed a recurring frustration: when an agent tries to read a documentation website, it often struggles.

It might crawl an entire webpage, choke on thousands of lines of client-side React hydration boilerplate, get confused by cookie consent banners and responsive sidebar markup, and then proceed to hallucinate function names, parameters, or types that don't exist in your SDK.

In our project—html2pdf, a developer API and TypeScript SDK for converting HTML and CSS to pixel-perfect PDFs with native Indian language support—we decided to fix this.

We recently adopted the emerging llms.txt standard across our Astro documentation site.

Here is why web documentation is broken for AI models, how the llms.txt specification solves it, and how we implemented both llms.txt and llms-full.txt in our Astro SSR pipeline to give AI agents instant, zero-hallucination context.

---

1. The Problem: The "Documentation Tax" on AI Agents

Documentation websites are built for humans, not Large Language Models.

Consider what happens when a developer asks Cursor or Claude:

"Write a TypeScript script that uses the @akcraft/html2pdf-sdk to generate a landscape A4 invoice with 15mm margins."

If the AI tries to fetch https://html2pdf.akcraft.app to learn how our SDK works, the server responds with a full HTML document:

<!DOCTYPE html>
<html lang="en">
  <head>
    <script type="module" src="/_astro/hoisted.B_x9Z1.js"></script>
    <style>/* 25 KB of minified utility CSS */</style>
  </head>
  <body>
    <header class="sticky top-0 z-50 backdrop-blur border-b border-slate-200">
      <nav aria-label="Main Navigation">
        <!-- 150 lines of desktop and mobile navigation links -->
      </nav>
    </header>
    <main>
      <!-- Interactive React Islands, SVG icons, marketing copy -->
    </main>
    <footer>
      <!-- 200 lines of social links, legal disclaimers, and copyright -->
    </footer>
  </body>
</html>

Why this degrades AI output:

1. Context Window Waste: Modern LLMs have generous context windows, but processing 15,000 tokens of raw HTML, SVG paths, and CSS class names is an inefficient waste of attention and cost.
2. Attention Dilution: Crucial SDK parameter types (format: PaperFormat.A4, orientation: PdfOrientation.LANDSCAPE) get buried under navigation links, cookie disclosures, and marketing slogans.
3. Hallucination Risk: When models can't parse messy DOM trees reliably, they guess. You end up with generated code importing non-existent packages like html2pdf-client or passing invalid options like { pageSize: 'A4' }.

We need a dedicated, machine-friendly format: documentation stripped down to pure, structured markdown.

---

2. The Standard: What is llms.txt?

Proposed by Jeremy Howard (Answer.ai) and gaining rapid adoption across projects like FastHTML, Anthropic docs, and Mintlify, llmstxt.org proposes a simple convention:

Place an /llms.txt file at the root of your domain to serve as a curated, machine-readable index of your documentation for LLMs.

Similar to how robots.txt tells web scrapers what to index, llms.txt tells AI agents where your authoritative technical knowledge lives.

Anatomy of an llms.txt File

The specification requires clean, strict Markdown:

1. Single H1 Title: The name of the project or API.
2. Blockquote Summary: A 1–2 sentence high-level summary of capabilities.
3. Preamble: Short contextual guidance.
4. H2 Section Headers: Curated Markdown lists containing URLs and concise descriptions.
5. Optional Section (## Optional): Secondary or non-critical resources (e.g., login portals, dashboards, license terms) that agents can skip when context window space is constrained.

Alongside llms.txt, the standard encourages a companion file: llms-full.txt. While llms.txt is an index of links, llms-full.txt is a single, complete, copy-pasteable document containing all core API schemas, parameters, and examples.

---

3. How We Designed Our Files in html2pdf

We created both llms.txt and llms-full.txt tailored to our developer workflow.

1. website/public/llms.txt (The Index)

Here is our live llms.txt:

# Html2Pdf
> High-performance developer API and TypeScript SDK for converting HTML and CSS to pixel-perfect PDF documents with native Indian language typography.

Html2Pdf provides a headless Chromium-based rendering engine built on Fastify, with bundled Noto font shapers for 21 Indian languages across 12 scripts, per-user rate limiting, S3-compatible cloud storage, and signed URL generation.

## Documentation
- [TypeScript SDK Reference](https://www.npmjs.com/package/@akcraft/html2pdf-sdk): Official @akcraft/html2pdf-sdk TypeScript client, methods, and types.
- [API OpenAPI / Swagger](https://html2pdf.akcraft.app/swagger): Fastify OpenAPI specification and interactive Swagger UI.
- [Full Documentation](https://html2pdf.akcraft.app/llms-full.txt): Comprehensive single-file reference for LLMs, including schemas, code samples, and Indic language matrix.

## Core Features
- [Interactive Studio Demo](https://html2pdf.akcraft.app/#studio): Browser studio for testing HTML payloads, presets (invoice, report, Indic text), and print options.
- [Indic Language Typography](https://html2pdf.akcraft.app/#languages): Zero-configuration rendering for Devanagari, Bengali, Tamil, Telugu, Gujarati, Gurmukhi, Kannada, Malayalam, Odia, Urdu, Ol Chiki, and Meetei Mayek.
- [Architecture Pipeline](https://html2pdf.akcraft.app/#architecture): 4-step rendering pipeline: Key Auth -> Sandboxed Chromium -> S3 Storage -> Expiring Presigned URL.

## Optional
- [Dashboard](https://html2pdf.akcraft.app/dashboard): Manage API keys, monitor monthly conversion quotas, and download generated PDFs.
- [Sign In](https://html2pdf.akcraft.app/login): Authenticate via email/password or Google OAuth.
- [npm Package](https://www.npmjs.com/package/@akcraft/html2pdf-sdk): Client package on npm.

2. website/public/llms-full.txt (The Master Knowledge Base)

When an agent needs complete implementation details, it points to llms-full.txt. We packed our entire technical interface into a compact 300-line Markdown file:

• Full SDK Usage: Real TypeScript snippets showing imports, initialization with apiKey, and calling sendHtmlRequest().

• Exact Enums & Types:

  export enum PaperFormat {
    Letter = 'Letter',
    Legal = 'Legal',
    Tabloid = 'Tabloid',
    Ledger = 'Ledger',
    A0 = 'A0',
    // ... A1 through A6
  }
  
  export enum PdfOrientation {
    PORTRAIT = 'portrait',
    LANDSCAPE = 'landscape',
  }
  

• Fastify TypeBox Schema: Complete request payload shapes (userId, content, options, margin, printBackground, headerTemplate, footerTemplate).
• cURL HTTP Examples: Direct REST calls sending the x-api-key header and parsing pdfUrl.
• Status & Error Handling: The Html2PdfError class structure (statusCode, retryAfter, apiError) and HTTP 429 rate limit behaviors.
• The Indic Font Matrix: Supported font families (Noto Sans Devanagari, Noto Serif Bengali, etc.) and CSS snippet patterns.

---

4. Implementation in Astro SSR

Our website is built with Astro 7 SSR running on the Netlify adapter (@astrojs/netlify).

When adding static machine-readable files to an Astro SSR application, you want to avoid unnecessary server-side rendering overhead.

Static Placement via public/

By placing both files inside the website/public/ directory:

website/
├── public/
│   ├── llms.txt            <-- Served at /llms.txt
│   ├── llms-full.txt       <-- Served at /llms-full.txt
│   ├── robots.txt
│   └── favicon.svg
├── src/
│   ├── pages/
│   └── layouts/
└── astro.config.mjs

During npm run build, Astro automatically copies files in public/ directly to dist/client/.

Key Advantages of This Approach:

1. Zero Lambda / Serverless Invocations: Netlify's edge CDN serves /llms.txt and /llms-full.txt directly from the static edge cache without waking up a Node.js SSR function.
2. Correct Content-Type: Served cleanly with Content-Type: text/plain; charset=utf-8 or text/markdown, preventing browsers or scrapers from expecting HTML tags.
3. Hermetic & Git-Tracked: The documentation is version-controlled alongside our code. Whenever an engineer updates an API schema or SDK option, llms-full.txt is updated in the exact same Git commit.

We verified our build and test suite:

cd website
npm run build
# Astro outputs:
# dist/client/llms.txt
# dist/client/llms-full.txt
# All 52 website unit tests green.

---

5. The Real-World Impact: Testing with AI Agents

To see the difference, we tested an AI coding assistant (Cursor / Claude 3.5 Sonnet) on a prompt:

"Write a Node.js script using @akcraft/html2pdf-sdk that sets up custom 20mm margins, prints backgrounds, and handles rate limiting."

Before llms.txt (Crawling the raw homepage):

• The model imported Html2Pdf instead of Html2PdfClient.
• It passed { margins: { top: 20 } } (numbers instead of strings with units like '20mm').
• It completely fabricated a non-existent method client.convert().

After llms.txt (Passing @https://html2pdf.akcraft.app/llms.txt):

• 100% Type-Safe: Imported Html2PdfClient, PaperFormat, and PdfOrientation.
• Accurate Margins:

  options: {
    format: PaperFormat.A4,
    orientation: PdfOrientation.PORTRAIT,
    printBackground: true,
    margin: {
      top: '20mm',
      bottom: '20mm',
      left: '20mm',
      right: '20mm',
    },
  }
  

• Error Handling: Correctly caught Html2PdfError and checked err.statusCode === 429 with err.retryAfter.

Zero guesswork. Zero hallucinations.

---

6. Checklist: How to Add llms.txt to Your Project

If you maintain an open-source library, a SaaS API, or a developer documentation site, adding llms.txt takes less than an hour:

1. [ ] Create public/llms.txt:
   • Add your H1 and blockquote description.
   • Link directly to your primary developer guides and OpenAPI specs.
   • Include an ## Optional section for secondary pages.
2. [ ] Create public/llms-full.txt:
   • Provide a complete, self-contained Markdown reference of all types, endpoints, schemas, and common recipes.
3. [ ] Keep It in Sync:
   • Treat your llms.txt as a first-class citizen in CI/CD. When updating your TypeScript types or endpoints, update your LLM documentation in the same PR.
4. [ ] Link it in Your Footer:
   • Add a subtle /llms.txt link in your website footer so both human developers and agents can discover it.

---

Summary

The web is evolving. While human developers appreciate beautiful landing pages, interactive sandboxes, and modern typography, AI coding agents need concise, unambiguous, and token-efficient truth.

By implementing the llms.txt specification, we made html2pdf AI-native. Any developer using an AI assistant can now integrate our PDF microservice in seconds without fighting outdated training data or hallucinated APIs.

You can inspect our live files anytime:

• Index: https://html2pdf.akcraft.app/llms.txt
• Full Reference: https://html2pdf.akcraft.app/llms-full.txt