Robots Txt Python Packages

protego

A pure-Python robots.txt parser with support for modern conventions.

ultimate-sitemap-parser

Ultimate Website Sitemap Parser

advertools

advertools - online marketing productivity and analysis tools

scovant-core

Open-source, evidence-first scanner for passive AI agent readiness signals on websites.

sluicer

Turn any web page, or a whole site, into structured data, with no model in the loop: every record a page declares with where each value came from, fields learnt from one example and checked on every page, crawls, feeds and archives.

django-bots

User-agent parsing, crawler detection and AI bot blocking for Django.

django-icv-sitemaps

Scalable sitemap generation for Django: background XML sitemaps, robots.txt, llms.txt, security.txt, redirects, and 404 logging

datasette-block-robots

Datasette plugin that blocks robots and crawlers using robots.txt

crawlableseo

Make a client-rendered SPA crawlable from its own HTML shell: per-URL meta tags, JSON-LD, robots.txt, sitemap.xml, llms.txt and IndexNow from one declaration. No headless browser, no SSR.

rastrolog

See which AI crawlers read your site and which AI chat products send you visitors. CLI, middleware, JS snippet.

jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

texting-robots-py

Python binding for the Texting Robots robots.txt parser

atomno-mcp-seo-audit

MCP server for technical SEO audits powered by the detail.web engine: health score, 78 checks across 8 categories, robots/sitemap/JSON-LD/meta tools, GEO sub-score.

django-cs-robots

Editable robots txt

harvester-kit

Polite, reproducible web harvesting: fetch once, parse forever. Resumable crawls, offline re-parsing, RFC 9309 robots.txt, provenance on every record. Built on Scrapling.

yirabot

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

astrocrawl

Async web crawler powered by Playwright headless Chromium — JavaScript rendering, AI-powered extraction (OpenAI / Anthropic / Google), CSS rule engine with visual preview, and a full PySide6 desktop GUI. SQLite WAL crash recovery, proxy circuit breaker, RFC 9309 robots.txt, ReDoS-immune regex, 3,822 tests. pip install astrocrawl[full,gui,fast].

fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

wbactl

Local-first toolkit for Web Bot Auth (RFC 9421): agent-side doctor, verifier conformance vectors, passive AI-access audit.

geo-check

Check whether a website is reachable and readable by AI crawlers, and score it. Two scores, never averaged. Blocking model training costs no points.

sitemap-harvester

Crawl sitemap of a given website and export metadata of its pages recursively into CSV format.

django-seo-suite

Lightweight, extensible SEO metadata layer for Django: titles, meta tags, canonical URLs, hreflang, JSON-LD and Open Graph for models and views — plus a versioned robots.txt.

indexnow-ping

Submit sitemap URLs through IndexNow and optionally Google Search Console, with a robots.txt guard for pre-launch sites.

greenflare

Open-Source Python Based SEO Web Crawler