Big Data Python Packages

cython

The most widely used Python to C compiler

pyspark

Apache Spark - A unified analytics engine for large-scale data processing

delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

synapseml

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

delta-sharing

An open protocol for secure data sharing

uproot

ROOT I/O in pure Python and NumPy.

starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

koalas

Koalas: pandas API on Apache Spark

feast

The Open Source Feature Store for AI/ML

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

arcticdb

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides a universal API for building parallel applications in the cloud ☁️🚀

scikit-learn-intelex

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

daal

oneAPI Data Analytics Library (oneDAL)

daal4py

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

redislite

Redis in a python module.

nipype

Workflows and interfaces for neuroimaging packages

hazelcast-python-client

Hazelcast Python Client

catboost-dev

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.