Robotstxt Python Packages

robotspy

Alternative robots parser module for Python

fast-robotstxt

Fast and modern robots.txt parser and matcher (C++20). Fork of Google's library with zero-copy parsing (~30% faster) and RFC 9309 compliance fixes.

jwm-robotstxt

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

rbdt

rbdt is a python library (written in rust) for parsing robots.txt files for large scale batch processing.

site-seo-scanner

The site seo scanner.

forgejo-bot-guard

This package helps administrators of Forgejo (or Git hosting platforms) generate clear, actionable anti-crawling measures by analyzing user-provided text inputs. When given a description of suspicious