Hadoop Python Packages

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

yarn-api-client

Python client for Hadoop® YARN API

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

skein

A tool and library for easily deploying applications on Apache YARN

hadoop-fs-wrapper

Python Wrappers for Hadoop FileSystem

dvc-hdfs

HDFS/WebHDFS plugin for dvc

pyhdfs

Python HDFS client

spooq

Spooq is a PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

hadoop-yarn-rest-api

This is Python Library for YARN REST api

h2o-client

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

hadeploy

An Hadoop Application Deployment tool

streamsx-hdfs

This toolkit provides operators and functions for interacting with Hadoop File System.

yarn-dev-tools

Various scripts to automate and ease Apache Hadoop YARN development.

splitlog

Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy

knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead

cobra-policytool

Manage Apache Atlas and Ranger configuration for your Hadoop environment.

clusterdock

clusterdock is a framework for creating Docker-based container clusters

madoop

A light weight MapReduce framework for education

jumpy

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

tf-yarn

Train TensorFlow models on YARN in just a few lines of code!

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

h2o-mlflow-flavor

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

hadopy

Easy parallel map-reduce command line tool

python3-lzo-indexer

Python library for indexing block offsets within LZO compressed files