Big Data Python Packages

cython

The most widely used Python to C compiler

pyspark

Apache Spark - A unified analytics engine for large-scale data processing

delta-spark

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs

catboost

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

graphframes

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

pyspark-client

Apache Spark - A unified analytics engine for large-scale data processing

synapseml

Simple and Distributed Machine Learning

delta-sharing

An open protocol for secure data sharing

koalas

Koalas: pandas API on Apache Spark

daft

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

graphframes-py

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

uproot

ROOT I/O in pure Python and NumPy.

starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.

feast

The Open Source Feature Store for AI/ML

nipype

Workflows and interfaces for neuroimaging packages

arcticdb

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀

awkward0

Manipulate arrays of complex data structures as easily as Numpy.

uproot3

ROOT I/O in pure Python and NumPy.

scikit-learn-intelex

Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application

daal

oneAPI Data Analytics Library (oneDAL)

hazelcast-python-client

Hazelcast Python Client

video2dataset

Easily create large video dataset from video urls