Big Data Python Packages

cython

The most widely used Python to C compiler

pyspark

Apache Spark - A unified analytics engine for large-scale data processing

delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

synapseml

Simple and Distributed Machine Learning

pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

delta-sharing

An open protocol for secure data sharing

koalas

Koalas: pandas API on Apache Spark

uproot

ROOT I/O in pure Python and NumPy.

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

nipype

Workflows and interfaces for neuroimaging packages

starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

feast

The Open Source Feature Store for AI/ML

arcticdb

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

awkward0

Manipulate arrays of complex data structures as easily as Numpy.

uproot3

ROOT I/O in pure Python and NumPy.

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

scikit-learn-intelex

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

daal

oneAPI Data Analytics Library (oneDAL)

daal4py

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

hazelcast-python-client

Hazelcast Python Client