Hdfs Python Packages

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

tiledb

Python interface to the TileDB storage engine

skein

A tool and library for easily deploying applications on Apache YARN

wradlib

weather radar data processing - python package

dvc-hdfs

HDFS/WebHDFS plugin for dvc

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

pyhdfs

Python HDFS client

snakebite

A pure python HDFS client

cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster

hadeploy

An Hadoop Application Deployment tool

pyquery-polars

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

streamsx-hdfs

This toolkit provides operators and functions for interacting with Hadoop File System.

webhdfspy

Python wrapper to access Hadoop HDFS REST API

alphareader

A custom reader for delimited files in Python. Ability to ingest big data files.

lakekeeper

A configurable PySpark package to identify fragmented external tables and perform safe in-place compaction

test-cephadm

Ceph is a distributed object, block, and file storage platform

pyquery-core

PyQuery is a local-first data operating system built on lazy execution that processes 100GB+ files while you doomscroll. No cap. 🧢

schemaindex

SchemaIndex is designed for data scientists to index and search metadata more efficiently.

pydistcp

A python Web HDFS based tool for inter/intra-cluster data copying.

dfspy

Distributed File System written in Python

kraken-pyds

Kraken - A distributed data transfer tool.

sqoopit

A python package that lets you sqoop into HDFS/Hive/HBase data from RDBMS using sqoop

srcd-smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

spark-yarn-submit

A python library to submit spark job in yarn cluster at different distributions (Currently CDH, HDP)