Hdfs Python Packages

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

tiledb

Python interface to the TileDB storage engine

skein

A tool and library for easily deploying applications on Apache YARN

dvc-hdfs

HDFS/WebHDFS plugin for dvc

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

wradlib

weather radar data processing - python package

snakebite

A pure python HDFS client

pyhdfs

Python HDFS client

cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

hadeploy

An Hadoop Application deployment tool

pyquery-polars

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

streamsx-hdfs

This toolkit provides operators and functions for interacting with Hadoop File System.

webhdfspy

A Python wrapper library to access Hadoop WebHDFS REST API

alphareader

A reader for large files with custom delimiters and encodings

pyquery-core

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

schemaindex

SchemaIndex is designed for data scientists to index and search metadata more efficiently.

kraken-pyds

Kraken - A distributed data transfer tool.

dfspy

Distributed File System written in Python

pydistcp

A python Web HDFS based tool for inter/intra-cluster data copying.

test-cephadm

cephadm

sqoopit

A simple package to let you Sqoop into HDFS/Hive/HBase with python

srcd-smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

spark-yarn-submit

library to handle spark job submit in a yarn cluster in different environment