SparkForensics

Appearance

Practical mechanics, stable references

Spark tuning reference

Move from a job symptom to the Spark mechanics and settings that can change its outcome without treating a tuning guess as evidence.

Browse the full reference Choose a symptom
Known routes: memory, shuffle, joins, AQE, and config. Other terms open the configuration reference.

Symptoms, mapped to the right lever

Each path stays on the canonical reference document, so existing links and saved anchors continue to work.

How to use this: SparkForensics reads your Spark event log and surfaces evidence; this reference explains the mechanics and settings behind it. Diagnose the symptom, change one informed lever, then validate against the next run's evidence. Confirm behavior and defaults against your deployed Spark version and distribution.

Spill or GC pressure

Trace executor memory, object lifetime, and the trade-offs behind changing memory settings.

Open memory model

Shuffle-heavy stages

Understand write/read costs, partition size, and why moving data becomes the bottleneck.

Open shuffle guide

Plan adapts poorly

Use AQE's runtime choices as an inspectable mechanism, not a checkbox to enable blindly.

Open AQE guide

Need the exact setting

Check a setting's scope, default, and interaction before applying it to a cluster.

Open config reference

Let SparkForensics surface the evidence. Use this reference to reason about it.

The two tools stay deliberately separate: SparkForensics reads the event log; this static reference explains the mechanics and settings behind what you find.

  1. Observe
    Read the stage or executor behaviorStart with the event log, metrics, and plan details Spark already produces.
  2. Explain
    Name the mechanism at workUse the stable reference anchors to connect symptoms to memory, shuffle, joins, AQE, or configuration.
  3. Test
    Change one informed leverMake a measured change, then compare the evidence again rather than carrying a tuning superstition forward.