No known security advisories.
The current version has no OSV.dev malware reports and no critical or high-severity vulnerabilities.
lm-eval has 1.7M monthly downloads and 13K GitHub stars. Version 0.4.12 supports Python 3.10+, first published in 2021.
It is actively maintained under a permissive license, last released in May 2026.
It is built on top of datasets, dill and evaluate. It is used by 36 tracked packages, including axolotl, wisent-extractors and heretic-llm.
GitHub topics include evaluation-framework, language-model and transformer.
Package Insights
Score breakdown - each component scores 0-100, weighted by percentage
Maintenance
80
Popularity
87
Adoption
70
Community
90
Provenance
100
How the maintenance status was determined
Last commit
12 days ago
Monthly downloads
1,690,633
•
Active
Activity within 30 days
Stable
Activity within 31-180 days
Minimal
Activity within 181-365 days, or no activity data
At Risk
No activity for > 365 days, but > 10,000 monthly downloads
Archived
Repository flagged as archived on GitHub
How the popularity category was determined
Monthly downloads
1,690,633
Stars
13,176
Stars per 1k downloads
7.79
Industry Standard
> 10,000,000 monthly downloads
Hidden Gem
< 0.5 stars per 1k downloads and > 100,000 monthly downloads
•
Community Favorite
> 5 stars per 1k downloads
Rising Star
Balanced ratio of stars to downloads
Unknown
Fewer than 1,000 monthly downloads, or no star data
How download momentum is calculated
Week daily avg
51,226
Month daily avg
56,354
Momentum
-9.1%
((week_daily_avg - month_daily_avg) / month_daily_avg) * 100Python version support tiers
requires_python
>=3.10
Cutting Edge
Minimum supported version is 3.12 or newer
•
Modern
Minimum supported version is 3.9, 3.10, or 3.11
Legacy
Supports 3.8 or older (or Python 2)
Unknown
No requires_python specified or unparseable
License categories
License
MIT
•
Permissive
MIT, Apache, BSD, ISC, Unlicense, CC0, etc.
Copyleft
GPL, LGPL, AGPL, MPL, EPL, etc.
Proprietary
Commercial, all rights reserved, custom EULAs
Unknown
No license specified, or none of the above match
How the release cadence was determined
Last release
56 days ago
Total releases
18
Releases/year
3.7
Rapid
Last release within 365 days and > 12 releases/year
•
Regular
Last release within 365 days
Slow
Last release 366-730 days ago
Stalled
Last release > 730 days ago, or no releases
Unknown
No usable release date available
How contributor risk was determined
Contributors
459
Monthly downloads
1,690,633
High Load
≤ 2 contributors and ≥ 50,000 monthly downloads
Moderate Load
< 10 contributors and ≥ 500,000 monthly downloads
Low Load
< 25 contributors and ≥ 5,000,000 monthly downloads, or sole maintainer with ≥ 10,000 monthly downloads
•
Balanced
Enough contributors for the package's popularity
Unknown
No contributor data available
Engagement
86
Issue Burden
43.4
per 1k stars
Discussions
0.2
per 1k stars
Fork Ratio
13.5%
contrib / forks
Stale Release
No
PR Velocity
Active
Weekly Downloads
GitHub Stars
Not all downloads include detailed metadata. Breakdowns reflect the subset that does.
Downloads by OS
Linux
30.5K
Windows
290
macOS
241
Python Versions
3.12
42.2%
13.0K
3.10
41.9%
12.9K
3.13
7.7%
2.4K
3.11
7.1%
2.2K
3.14
1.1%
347
Top Countries
1.
US
24.1K
US
2.
IL
1.6K
IL
3.
ID
982
ID
4.
NO
949
NO
5.
SG
836
SG
6.
FI
595
FI
7.
IE
235
IE
8.
CA
232
CA
9.
GB
176
GB
10.
LU
128
LU
Dependencies
14
- datasets >=2.16.0
- dill
- evaluate >=0.4.0
- jinja2
- more-itertools
- numpy
- pytablewriter
- rouge-score >=0.0.4
- sacrebleu >=1.5.0
- scikit-learn >=0.24.1
- sqlitedict
- tqdm >=4.59.0
- typing-extensions
- word2number
70 optional dependencies
- accelerate[hf]
- aiohttp[api]
- aiohttp[litellm]
- aiohttp[dev]
- antlr4-python3-runtime[math]
- auto-gptq[gptq]
- emoji[japanese-leaderboard]
- fugashi[japanese-leaderboard]
- fuzzywuzzy[longbench]
- gptqmodel[gptqmodel]
- ibm-watsonx-ai[ibm-watsonx-ai]
- immutabledict[ifeval]
- jieba[multilingual]
- jieba[longbench]
- jsonlines[archiver]
- kstar-planner[acpbench]
- langdetect[ifeval]
- lark[acpbench]
- librosa[audiolm-qwen]
- litellm[litellm]
- lm-eval[tasks]
- math-verify[math]
- nagisa[multilingual]
- neologdn[japanese-leaderboard]
- nltk[ifeval]
- nltk[ruler]
- numpy[wandb]
- optimum[optimum]
- optimum-habana[habana]
- optimum-intel[ipex]
- pandas[wandb]
- pandas[zeno]
- pddl[acpbench]
- peft[hf]
- pycountry[multilingual]
- pymorphy2[libra]
- pytest[dev]
- pytest-cov[dev]
- pytest-xdist[dev]
- python-dotenv[ibm-watsonx-ai]
- requests[dev]
- requests[litellm]
- requests[api]
- rouge[longbench]
- rouge-score[japanese-leaderboard]
- scipy[ruler]
- sentencepiece[sentencepiece]
- sentencepiece[dev]
- soundfile[audiolm-qwen]
- sparsify[sparsify]
- statsmodels[discrim-eval]
- sympy[math]
- tarski[acpbench]
- tenacity[api]
- tenacity[dev]
- tenacity[litellm]
- tiktoken[api]
- tiktoken[dev]
- torch[hf]
- tqdm[litellm]
- tqdm[api]
- tqdm[dev]
- trackio[trackio]
- transformers[hf]
- unitxt[unitxt]
- vllm[vllm]
- wandb[wandb]
- wonderwords[ruler]
- zeno-client[zeno]
- zstandard[archiver]
Used By
36
- axolotl
- wisent-extractors
- heretic-llm
- angelslim
- instructlab-eval
- annihilate-llm
- oumi
- annihilation-llm-tjcrims0nx
- optimus-dl
- onecomp
- efficientllm-toolkit
- llm-testkit
- green-bit-llm
- sinq
- learnware
- taker
- small-model-redteaming
- zuna
- veeksha
- aligntune
- annihilation-llm
- perception-models-perone
- schismatic
- ai-edge-eval
- adaptible
- mergenetic
- fate-llm
- torchchat
- deepcompressor-toolkit
- openseneca
- torchat
- lmquant
- angelslim-fork
- llada
- testverse
- brahmgen
GitHub Topics
Tags from the package's GitHub repository describing what it does.
Package Information
License
MIT
First Release
September 2, 2021
Last Release
1 month ago
Last Commit
1 week ago
Latest Version
0.4.12
Total Releases
18
Contributors
459
Open PRs
322
Archived
No
Python Versions
3
Wheels
py3
Tracking Since
January 6, 2026
Data Updated
July 5, 2026
Data Freshness
Fresh