Natural Language Processing Python Packages

huggingface-hub

The official CLI and Python client for the Hugging Face Hub.

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

datasets

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools

nltk

NLTK Source

sentencepiece

Unsupervised text tokenizer for Neural Network-based text generation.

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

thinc

🔮 A refreshing functional take on deep learning, compatible with your favorite libraries

spacy-loggers

📟 Logging utilities for spaCy

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

usaddress

:us: a python library for parsing unstructured United States address strings into address components

lingua-language-detector

The most accurate natural language detection library for Python, suitable for short text and mixed-language text

gensim

Topic Modelling for Humans

textblob

Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase extraction, translation, and more.

unstructured

Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models. Visit our website to learn more about our enterprise grade Platform product for production grade workflows, partitioning, enrichments, chunking and embedding.

fla-core

🚀 Efficient implementations for emerging model architectures

openvino

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

flash-linear-attention

🚀 Efficient implementations for emerging model architectures

stanza

Stanford NLP Python library for tokenization, sentence segmentation, NER, and parsing of many human languages

gliner

Generalist and Lightweight Model for Named Entity Recognition (Extract any entity types from texts)

spark-nlp

State of the Art Natural Language Processing

pythainlp

Thai natural language processing in Python

bert-score

BERT score for text generation

seqeval

A Python framework for sequence labeling evaluation(named-entity recognition, pos tagging, etc...)