Hdfs Python Packages

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

tiledb

Python interface to the TileDB storage engine

skein

A tool and library for easily deploying applications on Apache YARN

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

dvc-hdfs

HDFS/WebHDFS plugin for dvc

wradlib

weather radar data processing - python package

pyhdfs

Python HDFS client

snakebite

A pure python HDFS client

pyquery-polars

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

webhdfspy

A Python wrapper library to access Hadoop WebHDFS REST API

hadeploy

An Hadoop Application deployment tool

alphareader

A reader for large files with custom delimiters and encodings

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

streamsx-hdfs

HDFS integration for IBM Streams

pyquery-core

Enterprise-grade Headless ETL Engine with Interactive UI

schemaindex

An indexing engine for different types of data sources, including HDFS, Mysql, etc.

pydistcp

pydistcp: python WebHDFS inter/intra-cluster data copy tool.

srcd-smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

kraken-pyds

Kraken - A distributed data transfer tool.

dfspy

Distributed File System written in Python

test-cephadm

Ceph is a distributed object, block, and file storage platform

sqoopit

A python package that lets you sqoop into HDFS/Hive/HBase data from RDBMS using sqoop

spark-yarn-submit

A python library to submit spark job in yarn cluster at different distributions (Currently CDH, HDP)