Robots Txt Python Packages

protego

A pure-Python robots.txt parser with support for modern conventions.

ultimate-sitemap-parser

Ultimate Website Sitemap Parser

advertools

advertools - online marketing productivity and analysis tools

datasette-block-robots

Datasette plugin that blocks robots and crawlers using robots.txt

fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

yirabot

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

texting-robots-py

Python binding for Texting Robots

jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

django-cs-robots

A django app to change robots.txt from the admin panel without using the database.

agent-readiness-cli

Score any URL for AI-agent readiness — llms.txt, JSON-LD, AI-bot robots.txt, canonical, MCP, meta, sitemap

sitemap-harvester

Crawl sitemap of a given website and export metadata of its pages recursively into CSV format.

greenflare

SEO Web Crawler and Analysis Tool

context-linter

LLM readiness linter for websites. Audits robots.txt, llms.txt, Schema.org, and content density on a 0-100 scale. Includes MCP server. Published on PyPI: pip install context-cli.

rbdt

rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

pyrobotstxt

pyrobotstxt: Python Package for robots.txt Files

weboptout

Opt-Out tool to check Copyright reservations in a way that even machines can understand.

sitemap-grabber

A python library to recursively crawl every sitemap.xml for a website. Also handles robots.txt and other well-knowns.

astel

An asyncronous web crawling library for Python.

aeo-cli

Agentic Engine Optimization CLI — audit URLs for AI crawler readiness