Hadoop Python Packages

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

yarn-api-client

Python client for Hadoop® YARN API

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

skein

A tool and library for easily deploying applications on Apache YARN

dvc-hdfs

HDFS/WebHDFS plugin for dvc

hadoop-fs-wrapper

Python Wrappers for Hadoop FileSystem

pyhdfs

Python HDFS client

spooq

Spooq is a PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

h2o-client

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

hadoop-yarn-rest-api

This is Python Library for YARN REST api

splitlog

Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy

hadeploy

An Hadoop Application deployment tool

h2o-mlflow-flavor

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

madoop

A light weight MapReduce framework for education

yarn-dev-tools

Various scripts to automate and ease Apache Hadoop YARN development.

cobra-policytool

Manage Apache Atlas and Ranger configuration for your Hadoop environment.

streamsx-hdfs

This toolkit provides operators and functions for interacting with Hadoop File System.

clusterdock

clusterdock is a framework for creating Docker-based container clusters

tf-yarn

Train TensorFlow models on YARN in just a few lines of code!

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

jumpy

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...

aiowebhdfs

A modern and async implementation of the WebHDFS API in python

knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead

pydatavec

Suite of tools for deploying and training deep learning models using the JVM. Highlights include model import for keras, tensorflow, and onnx/pytorch, a modular and tiny c++ library for running math code and a java based math library on top of the core c++ library. Also includes samediff: a pytorch/tensorflow like library for running deep learn...