Spark evidence, kept local

Shuffle Works

Local-first tools for reading Spark event logs, catching PySpark schema drift, and tuning bottlenecks before they burn cluster time.

Open the toolkit

Built around Spark evidence

Small tools for a job that's slow, surprising, schema-fragile, or expensive, built around the evidence Spark leaves behind.

No upload path Keep event logs and schemas in the browser or local workflow instead of sending cluster evidence to a backend.
Event log first Start from the artifacts Spark already emits: history logs, DataFrame schemas, stages, memory pressure, and shuffle reads.
Spark setting, not superstition Explain the tuning levers that actually matter: AQE, partitioning, serialization, executor memory, and skew.

A static, zero-backend browser app for analyzing Spark History Server event logs.

Use when a job regresses and you need to inspect stages, timelines, shuffle reads, and executor behavior from the log you already have.

Open SparkForensics → Source on GitHub Airflow operator

Type annotation system for PySpark DataFrame schema validation via type hints.

Use when a DataFrame contract can drift between jobs and you want schema failures before Spark spends runtime discovering them.

Validate a schema contract → PyPI →

A hands-on reference for tuning Spark and PySpark jobs — internals, memory, shuffle, AQE, and cluster config.

Use when you need a practical path from a symptom like skew, spill, or slow joins to the Spark setting that changes the outcome.

Open Spark Tuning Reference → Source on GitHub