Documentation

Getting Started with Profound

Profound automatically generates and manages llms.txt files for your websites, making your content AI-ready and discoverable.

What is llms.txt?

The /llms.txt file is a proposed standard that helps AI models better understand and interact with your website content. Similar to robots.txt, it provides structured information specifically designed for Large Language Models.

It contains curated links to your most important pages organized into sections like Documentation, API Reference, Support, and more.

Learn more about the llms.txt standard →

Quick Start

1. Generate llms.txt (No account required)

  1. Visit the homepage
  2. Enter your website URL (e.g., https://example.com)
  3. Click "Generate"
  4. Wait for the crawler to analyze your site (typically 10-30 seconds)
  5. Download or copy the generated llms.txt file

2. Add to Your Website

Place the generated file in your website's root directory:

your-website.com/llms.txt

Dashboard Features

Sign up for free to access automated monitoring and updates.

Add a Site

  1. Navigate to Dashboard
  2. Click "Add Site"
  3. Configure your settings:
    • Base URL: Your website's main URL
    • Max Pages: How many pages to crawl (default: 50)
    • Include/Exclude Paths: Filter specific sections
    • Schedule: How often to check for updates

Automatic Updates

Profound automatically monitors your website and regenerates the llms.txt file when changes are detected.

Schedule Options:

  • Off: Manual crawls only
  • Daily: Check once per day
  • Every 6 hours: Check 4 times daily
  • Hourly: Check every hour

Manual Crawls

Trigger an immediate crawl anytime:

  1. Open your site's detail page
  2. Click "Run Crawl"
  3. Monitor progress in the Runs tab
  4. Download the updated llms.txt file

How It Works

1. Crawling

Profound's crawler:

  • Starts from your homepage and sitemap.xml
  • Follows internal links to discover pages
  • Respects robots.txt rules
  • Extracts titles, descriptions, and metadata
  • Limits to your configured max pages

2. Intelligent Categorization

Pages are automatically organized into sections:

  • Docs: /docs, /guide, /tutorial paths
  • API: /api, /reference, /sdk paths
  • Support: /faq, /help, /contact paths
  • Company: /about, /team, /careers paths
  • Optional: /blog, /changelog, /pricing paths

3. Smart Prioritization

Pages are scored based on importance:

  • Homepage and top-level pages rank highest
  • Pages with complete metadata score better
  • Documentation and API pages are prioritized
  • Pagination and tag pages are deprioritized

4. Change Detection

Profound tracks changes to regenerate when needed:

  • Content hashing detects page modifications
  • New pages are automatically discovered
  • Removed pages are tracked
  • Only regenerates when changes are found

Example Output

# Your Website Name

> A brief summary of what your website offers

## Docs

- [Getting Started](/docs/getting-started): Quick start guide
- [Installation](/docs/installation): Setup instructions
- [Configuration](/docs/configuration): Configure your project

## API Reference

- [API Overview](/api): Complete API documentation
- [Authentication](/api/auth): Authentication guide

## Support

- [FAQ](/faq): Frequently asked questions
- [Contact](/contact): Get in touch with our team

## Optional

- [Blog](/blog): Latest updates and articles
- [Changelog](/changelog): Version history

Best Practices

✅ Do

  • Use clear, descriptive page titles and meta descriptions
  • Organize your site with logical URL structure
  • Keep your sitemap.xml updated
  • Set appropriate crawl schedules (daily is usually sufficient)
  • Use path filters to focus on important sections

❌ Don't

  • Set max pages too low (you'll miss important content)
  • Crawl too frequently (hourly is rarely necessary)
  • Include admin or private pages
  • Forget to update your llms.txt file on your server

Frequently Asked Questions

How many pages should I crawl?

Start with 50-100 pages. Focus on quality over quantity. Use path filters to include only your most important content like documentation, guides, and key pages.

Will this slow down my website?

No. Profound crawls your site from external servers with rate limiting (max 2 requests/second) and respects your robots.txt rules. It's no different than a search engine crawler.

Can I exclude certain pages?

Yes! Use the "Exclude Paths" setting to filter out specific sections. For example, exclude /admin or /private.

What happens if my site changes?

Profound automatically detects changes during scheduled crawls and regenerates your llms.txt file. You'll see the changes tracked in your run history.

Is llms.txt required for AI models?

Not yet, but it's gaining adoption. Having one makes your content more discoverable and helps AI models understand your site structure better. Think of it as future-proofing your web presence.

Ready to Get Started?

Generate your first llms.txt file in seconds