Introduction

This guide is a hands-on reference for optimizing Spark and PySpark jobs. It covers Spark internals, memory management, joins, shuffle, data formats, PySpark-specific patterns, Adaptive Query Execution (AQE), and cluster tuning, followed by a bottleneck diagnostic reference.

Read it top to bottom for a full grounding in Spark performance, or jump straight to the Bottleneck Reference section below when you already know which symptom you're chasing.

Severity dots

Each finding is marked with a severity dot:

• Info: worth knowing, not yet a problem
• Warning: likely hurting job performance
• Critical: actively bottlenecking the job

Each bottleneck section below documents the exact thresholds behind these dots.

Tag system

Every bottleneck has a short tag used throughout this reference:

SKEW SHFL SPILLGC COLD UTILHOST FAIL STRAGRETRY TINY FAIL-RATE

See the Bottleneck Reference for what each one means, how it's detected, and how to fix it.