Detector catalog

What SparkForensics checks in a Spark run

34 checks, grouped by what they cost you: run time, money, failed work, SQL plans and setup. Each page says what the finding means, where you see it in the dashboard, an agent, CI and Airflow, and the thresholds behind it.

Speed 12

• SKEWTask skewOne slow task holds a whole stage back while the rest of the cluster waits.Judged per stageA change in the job codebudget flag
• SHAPEStage shapeA stage runs too few tasks for its cores, or its output dwarfs its input.Judged per stageA change in the job code
• SHFLShuffle I/OMoving data between executors takes more time than computing it.Judged per stageA Spark configuration change
• PARTPartition sizingShuffle partitions are too large, too uneven or too few, so tasks spill or fail.Judged per stageA Spark configuration change
• SPILLSpillTasks run out of execution memory and write intermediate data to disk.Judged per stageA change in the job codebudget flag
• GCGC pressureExecutors lose time to garbage collection instead of running tasks.Judged per stageA Spark configuration change
• HOSTSlow executor hostOne executor host runs its tasks far slower than the rest of the cluster.Judged per stageA Spark configuration change
• SLOWSlow stageA stage runs for a long stretch of the job with no single cause flagged on it.Judged per stageA change in the job code
• STRAGStraggling taskA few tasks run far longer than their peers, and the stage waits for them.Judged per stageA change in the job code
• SPECSpeculation wasteSpeculative copies of slow tasks use cores without finishing the stage sooner.Judged per stageA Spark configuration change
• TINYTiny tasksVery short tasks spend more time being scheduled than running.Judged per stageA change in the job code
• COLDCold startThe job waits for its first executors before any task can run.Judged on the whole runA change in the job code

Cost 7

• UTILExecutor utilizationExecutors sit idle for much of the run, so the cluster is larger than the work needs.Judged on the whole runA Spark configuration changereferencebudget flag
• UTILRight-sized clusterA smaller cluster would run the same stages in about the same time.Judged on the whole runA Spark configuration change
• MEMMemory utilizationExecutors reserve far more memory than tasks use, or run close to their limit.Judged on the whole runA Spark configuration changereference
• CSTORCache utilizationA cached dataset does not fit in memory or spills to disk.Judged on the whole runA change in the job codereference
• LOCALCore localityTasks often run away from the data they read.Judged on the whole runA Spark configuration changereference
• CHRNAutoscaling churnDynamic allocation adds and removes executors again and again.Judged on the whole runA Spark configuration change
• CACHECaching opportunityThe same data is read more than once and is never cached.Judged on the whole runA change in the job code

Reliability 4

• SFAILFailed stageA stage failed and took its job down with it.Judged per stageA change in the job code
• FAILFailed tasksTasks fail and run again, so the cluster pays for the same work twice.Judged per stageA change in the job codebudget flag
• RETRYRetry wasteFailed attempts are retried, and their lost time is added to the run.Judged per stageA change in the job code
• JOBSJob failure rateA large share of the jobs in the application fail.Judged on the whole runA change in the job code

SQL plans 9

• PLANDuplicate plan subtreeThe query plan computes the same subtree more than once.Judged per SQL queryA change in the job code
• PLANSmall filesA write produces many tiny files that slow every later read.Judged per SQL queryA Spark configuration change
• PLANOversized broadcast joinA broadcast join ships a table too large to broadcast.Judged per SQL queryA Spark configuration change
• PLANMissed broadcast joinA shuffle join could broadcast its small side and skip the shuffle.Judged per SQL queryA Spark configuration change
• PLANRow-at-a-time Python UDFRow-at-a-time Python functions slow a stage that vectorized code would speed up.Judged per SQL queryA change in the job code
• PLANNested loop joinA join with no usable key compares every row with every row and multiplies its output.Judged per SQL queryA change in the job code
• PLANMERGE rewriteA Delta MERGE rewrites most of the rows it did not change.Judged per SQL queryA change in the job code
• PLANUnpruned partitioned scanA filtered scan with no partition filter reads every partition of the table.Judged per SQL queryA change in the job code
• PLANCast on a join keyA cast on a join key forces another shuffle.Judged per SQL queryA change in the job code

Setup 2

• INCMPIncomplete runThe event log ends before the application did, so every number is a partial view.Judged on the whole runA change in the job code
• CFGConfig auditSpark properties that leave the cluster uncapped or the serializer slow.Judged on the Spark configurationA Spark configuration change