Sources and methodology
Where each PyRank metric originates, how it is transformed, when it is collected, and what it cannot prove.
Live data freshness
See the insights page's server-rendered data freshness panel for the latest successful source and publication runs.
Metric provenance
“Original source” identifies the upstream authority. “Serving layer” shows how PyRank obtains and publishes it. On smaller screens, scroll the table horizontally for every column.
| Metric | Original source | Serving layer | Calculation / window | Expected cadence | Limitations | Licence |
|---|---|---|---|---|---|---|
| Downloads and distributions | PyPI download events, streamed by Linehaul. The same events underpin ClickPy. | ClickPy on ClickHouse → PyRank downloads_daily / downloads_current → read projection. Breakdowns: BigQuery → weekly snapshots → read projection. | Rolling 1, 7 and 30 calendar-day sums ending on the source date frozen at the start of each sweep. OS, Python-version and country breakdowns cover the 24 hours before each weekly import. | One sweep after each successful weekly import, in hourly batches at :07 UTC until complete; breakdowns with the import, Sundays at 00:30 UTC; projection hourly at :50. | Counts are requests, not people or installations. CI, caches and mirrors can dominate. The source date stays fixed during a sweep; uncollected dates leave gaps in history. Breakdowns are a one-day sample of downloads that report those details. | CC BY 4.0 for PyPI's public dataset. |
| Package metadata and releases | PyPI JSON API, Simple index, RSS feeds and the BigQuery distribution-metadata table. | PyPI collectors → packages, releases and metadata observations in PostgreSQL → read projection. | Latest non-prerelease version plus published release history. Feeds add new projects/releases; the weekly import collects metadata for every project in PyPI's current Simple index. | Feeds every 10 minutes; full catalogue Sundays at 00:30 UTC; projection hourly at :50. | Metadata is supplied by publishers and may be incomplete or inaccurate. BigQuery metadata is immutable, so deleted projects or releases can remain. | BigQuery tables: CC BY 4.0. API metadata remains subject to PyPI terms and project-specific licences. |
| GitHub activity | GitHub GraphQL API; repository claims originate in PyPI project metadata. | Resumable GitHub sweep → github_current → hourly read projection and cached pages. | Current stars, forks, issues, topics, contributors (approximated by GitHub's mentionable users) and commit timestamps at collection time; history records successive observations. | One sweep after each successful weekly import, with up to 100,000 packages daily at 01:00 UTC until complete; projection hourly at :50. | Not every package names a repository. Claims are self-reported unless provenance-verified; monorepo packages share the same counts. A catalogue sweep spans multiple daily batches. | GitHub API Terms; repository content retains its own licence. |
| Dependencies and dependents | PyPI Requires-Dist core metadata via the distribution-metadata dataset. | Weekly import parses declarations → package_dependencies → dependency-intelligence queries and projection. | Direct requirements declared by the latest stable release at the last weekly import. Reverse-dependent counts group direct references by normalized package name. | Full catalogue Sundays at 00:30 UTC; projection hourly at :50. | These are declarations, not resolved installations. Environment markers, extras, dynamic metadata and invalid requirement strings affect what applies. A new release's dependencies appear after the next weekly import. | BigQuery tables: CC BY 4.0; dependency metadata also reflects each publisher's project. |
| Security advisories | OSV's PyPI ecosystem data, aggregating OpenSSF malicious-package reports and vulnerability databases. | OSV export → malicious-advisory and vulnerability tables → search, warnings and impact analysis. | Names are normalized to PyPI conventions. Known malicious packages are excluded from rankings. Vulnerability warnings cover high, critical and unrated advisories whose ranges include a package's latest version. | OSV import hourly at :20; projection hourly at :50. | Coverage begins with reports received by OSV's sources. Publication can lag discovery; ranges, severity and withdrawal status are source-defined. Low- and medium-severity vulnerabilities are not shown. This is not a security audit. | Source-specific open-data licences listed by OSV. |
| Repository provenance | PyPI Index API and Integrity API implementing PEP 740. | Attestation publisher identity → verified GitHub owner/repository fields → badges, mismatch warnings and verified-only rankings. | The repository claimed in metadata is compared with the GitHub repository in a release's cryptographically verifiable publish attestation. | New-release checks every 10 minutes; projection hourly at :50. | Only releases with supported attestations can be verified. Missing provenance means “unknown”, not unsafe; capped and backfill checks may delay a result. | PyPI terms; attested project artefacts retain their own licences. |
| Health score, momentum and rankings | Derived by PyRank from the PyPI, ClickHouse, GitHub and provenance inputs above. | Go calculation layer over the current read projection → server-rendered insights and rankings. | Health is 0–100: maintenance 27%, popularity 23%, adoption 22%, community 18%, provenance 10%. Package momentum compares the 7-day daily average with the 30-day daily average. Rising lists and the downloads ranking's change compare each 7-day total with the previous weekly sweep's. Star and fork changes compare each repository's latest weekly check with the previous one. | Projection hourly at :50. | Heuristics indicate activity and adoption, not code quality, fitness or safety. Packages without a linked GitHub repository are not assigned a comparable health score. | Derived output; the underlying source licences and terms above continue to apply. |
Collection and publication cadence
All scheduled times are UTC. The full BigQuery catalogue import is scheduled for Sundays at 00:30. Each successful import starts one download sweep and one GitHub sweep. Downloads run in hourly batches at :07, using a fixed source date for the whole sweep. GitHub processes up to 100,000 packages daily at 01:00, after any running import finishes. Both wait for the next successful import once complete, so these schedules do not mean each package is refreshed hourly or daily.
PyPI feeds continue checking for new releases every 10 minutes. OSV imports run hourly at :20, and the read projection is rebuilt hourly at :50.
Cadence is an expectation, not proof of freshness. The insights page's data freshness panel reports the latest successful completion recorded for each job. A source import becomes visible after the next projection swap and cache refresh.
How to interpret the data
- “Unknown” and missing values are not zero.
- Package names are normalized using PyPI naming rules.
- Rankings compare the newest published observation available for each package; row ages can differ.
- Follow the original-source links when making security, legal or procurement decisions.
Architecture and privacy
Separate Python collectors write source observations to PostgreSQL. An hourly atomic projection gives the read-only Go web service a consistent view for server-rendered pages, search and rankings; Redis and Cloudflare cache published responses.
Cloudflare Web Analytics provides cookieless traffic counts. PyRank does not set a tracking cookie or collect personal profile data through this page. Questions or corrections are welcome through feedback.