Automating llms.txt Generation with mkdocs-llmstxt Plugin¶

Today we integrated the mkdocs-llmstxt plugin into Instructor's documentation pipeline. This powerful plugin automatically generates llms.txt files from our MkDocs documentation, making our comprehensive guides instantly accessible to AI language models.

About the mkdocs-llmstxt Plugin¶

The mkdocs-llmstxt plugin by Timothée Mazzucotelli is a brilliant solution to a common problem: how do you keep an llms.txt file synchronized with your evolving documentation?

Key Features¶

Automatic Generation: The plugin generates llms.txt files directly from your MkDocs source files during the build process. No manual maintenance required.

Flexible Section Control: You can specify exactly which parts of your documentation to include:

plugins:
  - llmstxt:
      sections:
        Getting Started:
          - index.md: Introduction to structured outputs
          - installation.md: Setup instructions
        Core Concepts:
          - concepts/*.md

Clean Markdown Conversion: The plugin converts your documentation to clean, LLM-friendly markdown format, removing HTML artifacts and navigation elements.

Customizable Descriptions: You can provide both short and long descriptions of your project, giving AI models the context they need.

Our Implementation¶

Here's how we configured the plugin for Instructor:

plugins:
  - llmstxt:
      markdown_description: >
        Instructor is a Python library that makes it easy to work with structured outputs 
        from large language models (LLMs). Built on top of Pydantic, it provides a simple, 
        type-safe way to extract structured data from LLM responses across multiple providers 
        including OpenAI, Anthropic, Google, and many others.
      sections:
        Getting Started:
          - index.md: Introduction to structured outputs with LLMs
          - getting-started.md: Quick start guide
          - installation.md: Installation instructions
        Core Concepts:
          - concepts/*.md
        Integrations:
          - integrations/*.md

Why These Sections?¶

We carefully selected these sections because they provide AI models with the essential information needed to understand and use Instructor:

• Getting Started: Core concepts and installation
• Core Concepts: Deep dive into features like validation, streaming, and patterns
• Integrations: Provider-specific guidance for OpenAI, Anthropic, Google, and others

Technical Benefits¶

Build Integration¶

The plugin seamlessly integrates into our existing MkDocs build pipeline. Every time we deploy documentation updates, the llms.txt file is automatically regenerated with the latest content.

Content Freshness¶

Unlike manually maintained llms.txt files, our generated version is always up-to-date. When we add new integration guides or update existing concepts, the changes are automatically reflected.

Glob Pattern Support¶

The plugin supports glob patterns like concepts/*.md, making it easy to include entire directories without manually listing each file.

Plugin Architecture¶

The mkdocs-llmstxt plugin works by:

1. Parsing Configuration: Reading your sections configuration during the MkDocs build
2. File Processing: Converting specified markdown files to clean, LLM-friendly format
3. Content Assembly: Combining sections with metadata into the standard llms.txt format
4. Output Generation: Writing the final llms.txt file to your site root

Installation and Setup¶

Adding the plugin to your own MkDocs project is straightforward:

pip install mkdocs-llmstxt

Then add it to your mkdocs.yml:

site_url: https://your-site.com/  # Required for the plugin

plugins:
  - llmstxt:
      markdown_description: Description of your project
      sections:
        Documentation:
          - docs/*.md

Resources¶

• mkdocs-llmstxt Plugin
• llms.txt Specification
• Instructor Documentation

Special thanks to Timothée Mazzucotelli for creating this excellent plugin!