Out Of Core Python Packages

polars

Extremely fast Query Engine for DataFrames, written in Rust

polars-runtime-32

Extremely fast Query Engine for DataFrames, written in Rust

polars-lts-cpu

Extremely fast Query Engine for DataFrames, written in Rust

polars-runtime-compat

Extremely fast Query Engine for DataFrames, written in Rust

polars-runtime-64

Extremely fast Query Engine for DataFrames, written in Rust

polars-u64-idx

Extremely fast Query Engine for DataFrames, written in Rust

pysuricata

Single-pass exploratory data analysis on streaming algorithms. Bounded memory at any dataset size, pandas and Polars, self-contained HTML reports.

polars-nightly

Extremely fast Query Engine for DataFrames, written in Rust

metalcyte

Fast single-cell RNA-seq analysis on Apple silicon Macs, with a Rust engine, Metal GPU kernels and support for datasets larger than memory

cytome

Single-file format for single-cell multi-omics and spatial transcriptomics — matrices, metadata, embeddings, fragments and provenance in one SQLite-backed .cytome file, with SQL metadata queries and genomic range indexes.

dyna-zarr

A lightweight library for lazy operations on Zarr arrays without task graph overhead

memzero-train

Zero-memory, multi-core-CPU local AI training for ordinary laptops.

kenze

Big-file data prep that never runs out of memory - an interactive shell and CLI, powered by DuckDB.

zenreg

ZenReg is a Python-based high-throughput, memory-efficient pipeline for N-dimensional image registration, designed for large microscopy and imaging datasets.

cytearc

Fast, out-of-core single-cell analysis in Python for RNA, ATAC, protein, and multi-omic data, scaling to 100M+ cells. Mount remote datasets, work within a memory budget, and trace every result through built-in provenance.

tilewise-ccl

Lazy, memory-bounded connected-components labeling for large N-dimensional arrays (interior-boundary reconciliation; dask output)

ome-zarr-pyramid

Read, write and downscale OME-Zarr (NGFF) pyramids with a lazy, memory-bound Pyramid object

fdict

Easy out-of-core computing with nested data structures in Python as a drop-in replacement of dict

libmf

Python bindings to the libmf c++ repo for Matrix Factorization. libmf is fast, highly parallel, and can support factoring large matrices without holding them all in memory.

pyrametric

Lazy, memory-bounded connected-components labeling and per-object measurement for OME-Zarr pyramids

polars-on-premises

Extremely fast Query Engine for DataFrames, written in Rust

polars-on-prem

Extremely fast Query Engine for DataFrames, written in Rust

jagular

Out-of-core pre-processing of big-ish electrophysiology data

polars-cluster

Extremely fast Query Engine for DataFrames, written in Rust