One Place for Python Ecosystem Intelligence
PyPI shows download counts. GitHub shows stars and forks. Data from both is combined to surface new insights: packages with millions of downloads but few stars, projects gaining momentum before they trend, and popular libraries that may need new maintainers.
Data Sources
Download statistics come from Google BigQuery's public PyPI dataset, maintained by the Python Software Foundation. Daily, weekly, and monthly download counts are queried.
Historical download data is powered by ClickHouse, which PyPI uses to store and serve download statistics.
Repository metadata (stars, forks, issues, topics, last commit) comes from the GitHub GraphQL API. GitHub URLs are extracted from PyPI package metadata.
Package details (description, version, release dates) come from the PyPI JSON API, and new packages and releases are picked up from PyPI's RSS feeds within minutes.
Publish provenance comes from PyPI's Integrity API (PEP 740 attestations), which cryptographically links a release to the exact GitHub repository it was built from. GitHub rankings default to packages whose linked repository matches their attestation.
Malicious package and vulnerability advisories come from OSV.dev, Google's open-source vulnerability database. Refreshed hourly.
Health scores, momentum indicators, and insight categories are calculated from this raw data.
Collection Schedule
Every 10 minutes: PyPI's RSS feeds are polled, so new packages and releases appear within minutes.
Hourly: Security advisories are refreshed from OSV.dev, and the leaderboard projections are rebuilt so fresh data becomes visible.
Nightly: Download statistics are fetched from ClickHouse, and GitHub metadata (stars, forks, issues, topics) is updated in batches - the full catalogue cycles every few nights.
Weekly: A full BigQuery snapshot refreshes download counts and package metadata for all of PyPI.
Architecture
The API is built with FastAPI and asyncpg for non-blocking database queries. Data lives in PostgreSQL with hourly-rebuilt projection tables for fast leaderboard and search queries.
Smart cache warming is used: when a collection run completes, the cache is cleared and popular pages are pre-populated. Cache TTL is set until midnight UTC, giving ~19 hours of cache hits before the next potential update.
Pages are server-rendered by FastAPI with Jinja templates - plain HTML with a little JavaScript and no build step. Cloudflare provides edge caching.
The API has read-only database access, separate from the collector, and is rate-limited.
Privacy
Cloudflare Analytics provides cookieless traffic insights. No personal data is collected and no tracking cookie is set. All links to PyPI and GitHub go directly to those sites.
Limitations
- Download stats update nightly; GitHub metadata updates throughout the week
- Not all packages have GitHub URLs, so some lack repository metrics
- Packages deleted from PyPI may persist with stale data
- Packages sharing a GitHub repo (monorepos) show identical GitHub stats
- GitHub URLs in package metadata are self-reported and not always verified - any package can link to any repository. Where a package publishes PEP 740 attestations, the claimed repository is checked against the attested one: GitHub rankings default to these verified packages, verified detail pages show a badge, and mismatches get a warning banner
- Star counts reflect the linked GitHub repository, not the package itself - packages linking to popular repos will appear to have high stars regardless of the package's own popularity
Security
Packages are cross-referenced against the OSV database to identify malicious, vulnerable, or typosquatting packages. Malicious and typosquatting packages are excluded from leaderboards and lists, but remain searchable so users can find information about them. Popular packages whose latest version has a known high-severity vulnerability stay listed and gain a warning banner instead. Known malware advisories are browsable on the advisories page.