Robots Txt Python Packages

protego

A pure-Python robots.txt parser with support for modern conventions.

ultimate-sitemap-parser

Ultimate Website Sitemap Parser

advertools

advertools - online marketing productivity and analysis tools

datasette-block-robots

Datasette plugin that blocks robots and crawlers using robots.txt

texting-robots-py

Python binding for the Texting Robots robots.txt parser

django-seo-suite

Lightweight, extensible SEO metadata layer for Django: titles, meta tags, canonical URLs, hreflang, JSON-LD and Open Graph for models and views — plus a versioned robots.txt.

jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

django-cs-robots

Editable robots txt

atomno-mcp-seo-audit

MCP server for technical SEO audits powered by the detail.web engine: health score, 78 checks across 8 categories, robots/sitemap/JSON-LD/meta tools, GEO sub-score.

yirabot

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

context-linter

LLM readiness linter for websites. Audits robots.txt, llms.txt, Schema.org, and content density on a 0-100 scale. Includes MCP server. Published on PyPI: pip install context-cli.

sitemap-harvester

Crawl sitemap of a given website and export metadata of its pages recursively into CSV format.

greenflare

Open-Source Python Based SEO Web Crawler

agent-readiness-cli

Score any URL for AI-agent readiness — llms.txt, JSON-LD, AI-bot robots.txt, canonical, MCP, meta, sitemap.

weboptout

Checks the Copyright information of online works and their conditions of use.

rbdt

rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

pyrobotstxt

pyrobotstxt: Python Package for robots.txt Files

sitemap-grabber

Grab and recursively parse website sitemaps, robots.txt, and other related files.

astel

An asyncronous web crawling library for Python.

aeo-cli

LLM readiness linter for websites. Audits robots.txt, llms.txt, Schema.org, and content density on a 0-100 scale. Includes MCP server. Published on PyPI: pip install context-cli.