Hadoop Python Packages

luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

yarn-api-client

Python client for Hadoop® YARN API

h2o

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

skein

A tool and library for easily deploying applications on Apache YARN

dvc-hdfs

HDFS/WebHDFS plugin for dvc

hadoop-fs-wrapper

Python Wrappers for Hadoop FileSystem

pyhdfs

Python HDFS client

h2o-client

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

spooq

Spooq is a PySpark based helper library for ETL data ingestion pipeline in Data Lakes.

hadoop-yarn-rest-api

This is Python Library for YARN REST api

hadeploy

An Hadoop Application deployment tool

splitlog

Utility to split aggregated logs from Apache Hadoop Yarn applications into a folder hierarchy

madoop

A light weight MapReduce framework for education

cobra-policytool

Manage Apache Atlas and Ranger configuration for your Hadoop environment.

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

streamsx-hdfs

HDFS integration for IBM Streams

yarn-dev-tools

No description available

clusterdock

clusterdock is a framework for creating Docker-based container clusters

h2o-mlflow-flavor

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead

jumpy

Numpy and nd4j interop

tf-yarn

Distributed TensorFlow or pythorch on a YARN cluster

aiowebhdfs

A modern and asynchronous web client for WebHDFS

pydatavec

Python interface for DataVec