robots-parser
A pure-Python robots.txt parser with support for modern conventions.
Alternative robots parser module for Python
Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.
