Appearance
Threshold tables
The threshold tables for every bottleneck detector, and where each finding's impact band comes from.
Bottleneck thresholds
Every change to packages/core/src/detectors.ts should reference this table.
Every finding's impactBand comes from one of two places. For any finding whose impactEstimate carries a wallClock estimate (the common case for most rules below), analyzer.ts calls deriveImpactBand() (packages/core/src/impact-band.ts) immediately after estimateImpact(), which sets .impactBand purely from wallClock.high as a fraction of the app's total duration (>= 2% critical, >= 0.5% warning, else info: IMPACT_FLOOR_PCT_CRIT/ IMPACT_FLOOR_PCT_WARN, which skew's floorPctWarn and straggler's floorPctWarn/floorPctCrit default to, so a tail those gates admit grades at least warning; a tuned skew or straggler floor also grades that entry's own findings, see Tuning thresholds). For those rules, the table below documents their firing gate plus their fixed fallback constant, which surfaces only when this run's finding of that type didn't get a wallClock estimate (a stage excluded from the occupancy sweep). For rules whose finding type never gets a wallClock estimate (resourceOnly/informational basis, e.g. configAudit, or a rule that keeps its own ratio-tiered classification, e.g. failures), the full threshold table below is the real, displayed classification: detectors.ts sets impactBand directly and nothing overwrites it. partitionSizing's maxPartitionTooBig rule is a third case: it does carry a wallClock estimate but is explicitly exempted in deriveImpactBand() because it's a hardcoded-critical OOM/crash-risk safety signal, not a time-recovery one, so detectors.ts's own classification stands regardless of how small that estimate is relative to the run.
analyzer.ts then calls capCriticalByTaskTime() (impact-band.ts). It caps a critical finding at warning when its stage carries under CRITICAL_MIN_TASK_TIME_SHARE = 2% of the run's task time, whoever set the band. Failures (stageFailed, jobFailureRate), maxPartitionTooBig, a finding on no single stage and anything without recorded task time keep critical.
Fixed fallback only (usually wallClock-derived instead)
These rules have no band tiers in packages/core/src/detectors.ts (deriveImpactBand sets the band whenever a wallClock estimate is available); the constant in the last column is only a floor-case fallback. Their firing gate lives in each entry's thresholds object: it decides whether the rule reports anything, so it stays documented here in full.
| Rule | Fires when | Fallback |
|---|---|---|
| Task skew | taskDurationP95 / taskDurationP50 > 3× (taskDurationP50 and the other P50 fields are the textbook median: the mean of the two middle values on an even count; P95 stays nearest-rank) (taskDurationMax / P50 for stages under minTasksForP95 = 20 tasks), and the occupancy-clipped tail recovery (tailRecoveryMs: finalizeStage's task-level replay of the stage with every task over 4× P50 capped at P50) is ≥ floorPctWarn = 0.5% of app runtime. The clip floors the claim at the longest task the fix leaves, not the current one, and at the core work the fix leaves over every core (see impact-estimation.md's occupancy section); straggler's floors use the same clip, and the tail is data (tailAttribution: run time proportional to the slow tasks' input plus shuffle-read bytes or records accounts for at least dataShareMin = 0.5 of their extra time). An unattributed tail (no data volume to compare and nothing else explains the largest share, or no tail attribution) is straggler's | warning |
| Shuffle read | shuffleReadBytes > minBytes = 50 MiB, on a stage that isn't shorter than stageFloorPct = 0.5% of the run (a zero-length stage or an unknown run duration passes; the claim is clipped to the stage, so a shorter one would grade info, the shuffle itself still there) | info |
| Partition sizing: skew | shuffleReadMax > 5× shuffleReadP50 and shuffleReadMax > 256 MiB; it is a single-key shuffle (singleKey) when shuffleReadP50 is 0 and shuffleReadMax ≥ singleKeyShare = 90% of shuffleReadBytes | warning |
| Partition sizing: low parallelism | shuffleReadBytes ≥ 1 GiB and taskCount ≤ 7 | warning |
| Partition sizing: oversized partition | shuffleReadMax ≥ 5 GiB | critical |
| GC | executorRunTime ≥ minRunTimeMs = 10 s and gcPct > 10% | warning |
| GC (low / cost) | executorRunTime ≥ 10 s and gcPct < lowInfoPct100 = 5% (checked only when the GC row above did not fire and the log has no measured executor heap peak) and the stage lasts ≥ lowInfoFloorPct = 0.5% of the run (passes when the run's duration is unknown or the stage has zero length, such as one with no completion time). Gets no wall-clock estimate (an over-provisioning signal), so this band always stands. The floor drops low-GC notes from stages that barely ran: low GC is still true on those stages, but a memory-sizing note from a stage that barely ran adds nothing | info |
| Spill | any non-zero memoryBytesSpilled, on a stage that isn't shorter than stageFloorPct = 0.5% of the run (as for shuffle read: a shorter stage would grade info). The magnitude sub-table below classifies how much, but does not gate firing | warning |
| Cold start | firstExecutorAddedAt − firstStageSubmittedAt > gapSeconds = 30 s (no finding without executor-added events): the time a runnable stage waited for its first executor. An executor added before the first stage and still alive at submission means no wait and no finding; one removed at or before submission is ignored, so the gap runs to the next executor added after it. | warning |
| Slow host: mean-duration ratio | stage has ≥ minHosts = 3 hosts (or executors) and ≥ minTasks = 15 tasks, and lasts ≥ stageFloorPct = 0.5% of the run (passes when the run's duration is unknown or the stage has zero length, which gets no estimate and keeps its fallback band; every slow-host finding on a shorter stage would grade info, the imbalance itself still true; this gate covers every slowHost row); then per host: mean task duration / overall median ≥ ratioWarn = 2.0× and host task-share ≥ minShare = 20% and host mean ≥ floorMs = 1000 ms (absolute-magnitude floor, rules out sub-second noise) | warning |
| Slow host: duration-share | same stage gate as the row above; then per host: ≥ shareWarn = 75% of the stage's total task-duration and ≥ taskShareWarn = 50% of its task count | warning |
Stage slowness: absolute fallback, suppressed when slowHost already fired | stage wall-clock duration ≥ infoMin = 15 min. The band then comes from the partitioning-headroom estimate (see impact-estimation.md), not the duration | info |
| Straggler / speculative-execution | taskCount ≥ minTasks = 10, and the stage lasts ≥ floorPctWarn = 0.5% of the run (passes when the run's duration is unknown or the stage has zero length, such as one with no completion time; a shorter stage's tail can't cost more than its own duration, so every finding there would grade info, the slow tail itself still true), and either any speculative task ran, or straggler share > shareWarn = 5%, or straggler share > shareWarnAtFloor = 2.5% with the occupancy-clipped tail recovery (tailRecoveryMs, as for skew) already ≥ floorPctWarn (0.5% of app runtime). Skew's ratioWarn = 3 and the 0.5% floor are the values the task-level tail replay was tuned against. warnPct/critPct (10%/20% speculative share) and floorPctWarn/floorPctCrit (0.5%/2% of app runtime) do not set the band; they rank the straggler-vs-speculative tiers that pick which metric the finding reports. The data cut and skew's gate (ratioWarn, minTasksForP95, dataShareMin, floorPctWarn) are not this detector's thresholds: it reads skew's, as resolved for the run (DetectorCtx.skewThresholds), so a tail that data-driven is skew's unless skew's gate does not admit it, and tuning skew alone never drops a tail from both. A tail skew's gate admits that data does not explain is reported here without the share test, on any stage size. A straggler share has no critical tier of its own | info |
| Speculation waste | speculationWastedAttempts ≥ minWasted = 5 and speculationWasteMs ≥ minWasteMs = 60 s | warning |
| Retry waste | wastedAttempts ≥ minWasted = 3 and retryWasteMs ≥ minWasteMs = 30 s (attempts superseded by a later retry of the same task) | warning |
| Tiny tasks | taskCount ≥ minTasks = 100 and taskDurationP50 ≤ maxP50 = 500 ms and taskDurationP95 ≤ maxP95 = 1000 ms and the stage lasts ≥ stageFloorPct = 0.5% of the run (passes when the run's duration is unknown or the stage has zero length; coalescing can't save more than the stage's own duration, so every finding on a shorter stage would grade info, the tasks still tiny). Checked against each stage's measured per-task overhead (task wall time minus executor run time, the estimate's own input): a stage with 100+ tasks outside the P50/P95 gate would not save 0.5% of its run by coalescing, while one inside it can, and grades above info | info |
| Duplicate plan subtree | a subtree of ≥ minSubtreeSize = 3 nodes whose shape fingerprint repeats ≥ minOccurrences = 2× in the plan, unless its linked stages together lasted less than stageFloorPct = 0.5% of the run (a repeat with no linked stage time, or an unknown run duration, is kept; the claim counts at most each stage's own task-active time, so such a repeat would grade info, the repeat still in the plan). occurrencesIdentical records whether the repeats also agree node-for-node on normalized detail, ignoring AQE query-stage numbers; when they don't (same shape over another table, filter or projection) the finding is info with confidence low and no time claim. readsDifferentVersions is true when the repeats scan the same Delta tables at different versions recorded in the scan detail (Delta[version=N], versionAsOf); the finding is then info with no cache advice. A repeat under a show, showString or collect action (rootAction) is info with no time claim; one under count or save is unchanged. stageShares gives, per stage, the repeated operators' share of the stage's operators (WholeStageCodegen wrappers and Exchange write halves not counted); with no attributed stage the finding is info | warning |
| Small files read/write | per read/write side: file count > minFiles = 100 and average file size < maxAvgFileSizeMB = 3 MiB | warning |
| Nested loop join | a BroadcastNestedLoopJoin or CartesianProduct whose executor-side number of output rows ≥ minOutputRows = 1,000,000; a BroadcastNestedLoopJoin also needs that output ≥ minExpansion = 10 × the larger of its two inputs' rows (found by walking row-preserving operators down to the nearest number of output rows; an input with none, or one behind an operator that can change the row count, means no finding). A CartesianProduct re-reads its inputs once per partition of the other side, so its input counts are not compared and the evidence carries leftRows and rightRows as null. The band comes from the wall-clock estimate over the taskActiveMs of the stages that run the join, with an info fallback | info |
| Join key cast | a SortMergeJoin or ShuffledHashJoin with a cast(<column> as <type>) in a join key, whose output reaches an Exchange hashpartitioning keyed by that bare column before any other join, window, non-partial aggregate, Expand or Union. The exchange must not be a user repartition (REPARTITION_BY_*, REBALANCE_PARTITIONS_BY_*), and its hash keys must hold every key, cast removed, of a cast side that gives the join's output partitioning (both sides for Inner and Cross, the left for LeftOuter, LeftSemi, LeftAnti and ExistenceJoin, the right for RightOuter, neither for FullOuter). When the exchange feeds a join (past sorts and stage wrappers), its hash keys must be exactly that side's keys. No thresholds. value is the exchange's shuffle data size (0 when the log has none) | info |
| MERGE rewrite | a MergeIntoCommand node, one finding per rule. rewrittenRows: number of target rows rewritten unmodified ≥ minCopiedRows = 100,000 and ≥ minCopiedToChangedRatio = 2 × the rows inserted, updated and deleted. weakSkipping: number of target files before skipping ≥ minTargetFiles = 8, files after skipping ≥ minFilesKeptShare = 90% of them, more than 1 partition scanned (number of target partitions after skipping) and the partitions changed (the larger of number of target partitions from which files were removed and number of target partitions to which files were added) ≤ maxTouchedPartitionShare = 50% of those scanned | info |
| Unpruned scan | a scan node that reports number of partitions read (a partitioned table) with an empty PartitionFilters, at least one DataFilters predicate other than a null check, number of partitions read ≥ minPartitionsRead = 2 and size of files read ≥ minBytesRead = 1 GiB. A Delta log read is skipped | info |
| Python UDF | a SQL execution whose BatchEvalPython nodes (never ArrowEvalPython) report data sent to Python workers ≥ minBytesSent = 64 MiB and whose stages together ran ≥ minStageMs = 30 s. The floors are conservative guesses: no corpus or private log runs a Python UDF, so they are not tuned against real workloads. A node with no data sent to Python workers value is skipped | info |
| Broadcast sizing: missed | a 2-child SortMergeJoin with a buildable side (the right side for Inner, Cross, LeftOuter, LeftSingle, LeftSemi, LeftAnti and ExistenceJoin; the left side for Inner, Cross and RightOuter; neither for FullOuter) whose shuffle data size is ≥ minSmallerSideBytes = 1 MiB, ≤ overBroadcastBytes and ≤ the other side's (both sides need a shuffle data size of their own), and that is either at or under the effective threshold (spark.sql.adaptive.autoBroadcastJoinThreshold when the plan is adaptive and it is set, else spark.sql.autoBroadcastJoinThreshold, with the query's own settings applied) or < 10 MiB, or < 100 MiB with the other side > 10 GiB, or < 1 GiB with the other side > 300 GiB (broadcastTiers × comparisonTiers) | info |
| Broadcast sizing: oversized | a BroadcastExchange node whose data size metric > overBroadcastBytes = 1 GiB | warning |
Spill magnitude tiers
The spill row's band is a fixed warning fallback, but computeSpillMagnitude (packages/core/src/detectors.ts) runs on every spill finding and sets its spillMagnitude field, which the Spill widget displays. Its tiers, in evaluation order (first match wins, null when nothing matches):
| Condition | Magnitude |
|---|---|
Single-task stage: spillDiskMax ≥ singleTaskDiskGiB = 1 GiB or spillMemMax ≥ singleTaskMemGiB = 4 GiB | severe |
Multi-task stage: spillDiskMax ≥ highDiskGiB = 1 GiB, or spillDiskMax / taskCount ≥ highTaskDiskMB = 512 MiB (per-task proxy), or spillMemMax ≥ highMemGiB = 4 GiB | high |
Multi-task stage: spillDiskMax ≥ medDiskMB = 256 MiB or spillMemMax ≥ medMemGiB = 1 GiB | medium |
Skew (taskCount ≥ skewMinTasks = 10): spillDiskMax / spillDiskP50 > skewRatio = 5× and spillDiskMax ≥ skewDiskFloorMB = 128 MiB | high |
Skew (taskCount ≥ 10): spillMemMax / spillMemP50 > 5× and spillMemMax ≥ skewMemFloorMB = 256 MiB | medium |
Disk spill is weighted worse than memory spill by design: the memory thresholds sit well above their disk counterparts at every tier.
The finding also reports spillToInputRatio, the memory spill over inputBytes plus shuffleReadBytes, and sets wideRows from wideRowRatio = 10. A stage with under wideRowMinInputMB = 64 MiB of input gets no ratio. Neither threshold changes the band or the magnitude.
A volume spill adds a spark.executor.cores fix when peakExecutionMemoryMax is at least peakNearShareRatio = 0.8 of the execution memory per task slot.
Full threshold table (never wallClock-derived)
| Rule | Warning | Critical |
|---|---|---|
| Stage shape: PRatio | taskCount / totalCores < 0.5 (info, under-parallelized), on a stage lasting ≥ lowParallelismFloorPct = 0.5% of the run (passes when the run's duration is unknown or the stage has zero length): parallelizing can't save more than the stage's duration, so a shorter stage would fire below the floor | none |
| Stage shape: OIRatio | outputBytes / inputBytes > 10× (info, data explosion) | none |
| Stage shape: TaskStageSkew | taskDurationMax / stageDuration > stageShareMin = 0.5 and taskDurationMax / taskDurationP50 > skewWarn = 3× (info), on a multi-task stage lasting ≥ taskStageSkewFloorPct = 0.5% of the run. A task runs inside its stage, so the share is at most 1; the median gate is needed because on a single wave of even tasks the longest one spans the whole stage too | none |
| Failed tasks | failure rate > 5% (min 10 tasks) | > 20% |
| Stage failed outright | none | any stageFailureReason present |
| Slow host: multi-dimensional | max/median ratio across taskTime/inputBytes/shuffleBytes/storageMemory ≥ 1.33× (info); each dimension's sample must also clear an absolute floor (1000 ms for taskTime, 64 MiB for the byte dimensions). A stage lasting under stageFloorPct = 0.5% of the run is skipped (see the mean-duration row): a byte dimension there has no time estimate and would otherwise keep its ratio tier (a shorter stage's byte dimension would otherwise keep a warning or critical tier). The taskTime dimension also has to pass a chance test, since the slowest of many executors always tops the median: on the log scale of task duration, (the executor's mean − the median executor's mean) ÷ (the stage's task spread ÷ √ the executor's task count) must reach √(2 ln(executors ÷ (2 × outlierAlpha))) with outlierAlpha = 0.005 (3.4 for 3 executors, 4.2 for 75), so an executor with fewer tasks or a stage with more executors needs a larger gap. The stage spread is rebuilt from each executor's logDurationSum and logDurationM2 (the sum and M2 of ln(1 + task duration) that finalizeStage accumulates); a stage with under two tasks, no spread, or an executor missing or non-finite in either field flags no executor on task time. Of the executors that pass, the one with the largest mean is reported. The taskTime dimension falls back to info and takes its band from its wall-clock estimate; the byte dimensions get no time estimate and keep the ratio tier | ≥ ratioTiers[1] = 1.78× warning, ≥ ratioTiers[3] = 10× critical (ratioTiers[2] = 3.16 is not read) |
| Utilization | busy core time / allocated core time < 60% (always warning; ranks by its idle core time) | none |
| Right-sized cluster | the smallest core count whose modeled stage time stays within tolerancePct = 10% of the observed one is at most maxCoreShare = 75% of the peak cores (min minBaselineCores = 8; silent when the model error at the peak exceeds maxModelErrorPct = 20%; always warning, low confidence) | none |
| Autoscaling churn: short-lived executors | > 30% of removed executors alive under 2 min (min 5 removed executors) | > 60% |
| Job failure rate | ≥ 30% (≥ 10% info) | ≥ 50% |
| Idle cores | idle share of allocated core time > 50% (warning) | none |
| Memory band | highest executor peak heap / allocated < 70% over-provisioned (info); no near-capacity band, since heap used counts uncollected garbage | none |
| Caching opportunity | same input relation (or join/union subtree) scanned by ≥ minExecutions = 2 SQL executions in one run | none (single tier, info) |
| Cache utilization: partial caching | numCachedPartitions / numPartitions < 0.90 (info) | < 0.50 (warning) |
| Cache utilization: disk spillover | diskSize / (memorySize + diskSize) > 0.15 (info), MEMORY_AND_DISK* only | > 0.40 (warning) |
| Cache utilization: storage unobserved | persisted RDDs, but no SparkListenerBlockUpdated for any rdd_* block and every RDD Info figure 0 (spark.eventLog.logBlockUpdates.enabled off on Spark 2.3+): a missing-evidence caveat, not a threshold | none (single tier, info) |
| Core locality | non-local task ratio ≥ warnRatio = 15% (min minTasks = 50 tasks) | ≥ critRatio = 35% |
| Config: memory overhead | spark.executor.memoryOverhead below max(minimum, factor × executor memory) (info). The minimum is spark.executor.minMemoryOverhead on Spark 4+, else floorMB = 384 MiB; the factor is spark.executor.memoryOverheadFactor on Spark 3.3+, else floorPct = 10%. An unrecorded Spark version reads both settings | none |
The RDD Info cache figures on stage events (Number of Cached Partitions, Memory Size, Disk Size) are always 0 since Spark 2.3; Spark 1.x fills them only on StageCompleted. Real cache evidence is SparkListenerBlockUpdated, written only with spark.eventLog.logBlockUpdates.enabled=true (recordBlockUpdate in event-handlers.ts); the corpus cache-memory-only/cache-memory-and-disk logs carry it. Count a block's sizes only where its storage level says it lives: a drop to disk still reports the dropped bytes as Memory Size.
Spill classification: ≥80% tasks with zero spill → skew; <20% zero → volume; else unclassified. The classification badge is always shown in both compact and expanded spill widget states. This is independent of the magnitude tiers above: classification says what kind of spill, magnitude says how much.
Evidence fields (stageFailed / retryWaste)
Beyond a scalar metric/value, both entries' detect() attach to the finding:
numTasks:stage.taskCountat detection time.memoryBytesSpilled:stage.memoryBytesSpilledat detection time.stageFailedonly:failedTaskDetails: up to 20FailedTaskSamplerecords (taskId,attemptNumber,host,executorId,reason,peakExecMem,memSpilled,shuffleWrite) for tasks still marked failed when the stage was finalized (finalizeStage,stage-quantiles.ts).retryWasteonly:retriedTaskDetails: up to 20FailedTaskSamplerecords for attempts discarded by the retry-dedup logic inaccumulateTask(event-handlers.ts): captured at the moment they'd otherwise be thrown away, since by finalize time only the winning attempt survives.
Both sample arrays are capped at 20 entries, filled in first-encountered order (finalize order for failedTaskDetails, discard order for retriedTaskDetails), not spread across distinct hosts/executors: a stage with failures clustered on one bad host could fill the cap before a more informative failure elsewhere in the stage is ever sampled.
Evidence fields (failures)
dominantReason: the most frequent end-reason tag (ExceptionFailure,ExecutorLostFailure, ...) among tasks still failed at finalize.dominantError: the error behind that tag, from its largest failure group: the exception class, or<tag>: <loss reason>; falls back to the tag. The recommendation names it and never embeds a message.failureGroups: up to 5TaskFailureGroups (reason,className,message,lossReason,stackExcerpt,count), most frequent first: one group per distinct tag, class, message and loss reason, one excerpt each.otherFailedTasks: failed tasks that no shown group covers.
packages/core/src/task-failure.ts reads the details from Task End Reason (Class Name, Description, Full Stack Trace, Loss Reason, FetchFailed Message, Kill Reason) and bounds them at ingest: a message or loss reason to its first line (for a Python traceback, its final error line), 300 characters; an excerpt to the header, 8 frames, a Python traceback's error line and the last Caused by: line, 2000 characters in all. While parsing, a stage keeps at most 50 distinct failures (StageRecord.failureDetails, freed at finalize); later ones count only toward the tag. Redaction (redact.ts) replaces every failureGroups[].message and strips message text from stackExcerpt.
Threshold sources
Where each threshold that is not read from Spark's own source comes from.
| Threshold | Source status |
|---|---|
IMPACT_FLOOR_PCT_WARN = 0.5% and IMPACT_FLOOR_PCT_CRIT = 2% (impact-band.ts), and every runtime floor that defaults to them or to 0.5% (floorPctWarn, floorPctCrit, stageFloorPct, lowInfoFloorPct, lowParallelismFloorPct, taskStageSkewFloorPct) | Our own noise floors, unvalidated |
CRITICAL_MIN_TASK_TIME_SHARE = 2% (impact-band.ts) | IMPACT_FLOOR_PCT_CRIT applied to a stage's share of the run's task time. Our own noise floor, unvalidated |
SPILL_TASK_TIME_SHARE_PER_PASS = 50% and SPILL_TASK_TIME_SHARE_MAX = 60% (impact-model.ts) | Share of a stage's task time one pass of its read bytes through disk spill costs in serialization and sort CPU, and the cap. Measured on local sort stages, where one full pass cost 34-56% of the stage's time |
skew dataShareMin = 0.5 | Our own majority cut |
gc minRunTimeMs = 10 s | Our own noise floor, so a stage that barely ran flags in neither direction |
gc warnPct100 = 10 and lowInfoPct100 = 5 | Carried over unchanged from an external GC heuristic |
memoryUtilization idleCoreWarn = 0.50 | Carried over from an external wasted-cores heuristic |
rightSizing tolerancePct, maxCoreShare, minBaselineCores and maxModelErrorPct | Our own cuts; the model they gate is idealized. In the corpus gate, 65 of the 66 public logs have an application end, 9 hold a peak of 8 or more cores (3 with dynamic allocation), 1 of those 9 has a model error at or under 20%, and no log fires the finding. Only 2 logs remove an executor. No corpus log fires or misfires at the cuts, and the corpus has too few large, dynamic-allocation or executor-churn runs to tune them |
memoryUtilization wasteBufferMultiplier = 1.5 | Unverified buffer |
unprunedScan minBytesRead, minPartitionsRead | Our own cuts. The public Delta corpus logs scan small partitioned tables, so no default finding fires on them; the rule is tested on them with minBytesRead set to 0, and on synthetic plans |
mergeRewrite minCopiedRows, minCopiedToChangedRatio, minTargetFiles, minFilesKeptShare and maxTouchedPartitionShare | Our own cuts. The public corpus MERGE logs rewrite 2.7 times the rows they change and keep every target file, on a target with 4 partitions. No public log has skipping that fails on a partitioned target, so weakSkipping is checked on synthetic plans only |
joinKeyCast has no thresholds | The pattern is the signal; it is always info because a cast between small integer types is harmless |
nestedLoopJoin minOutputRows = 1,000,000 and minExpansion = 10 | Our own noise floors; the stage time the finding is graded on decides how much it matters |
pythonUdf minBytesSent and minStageMs | Conservative guesses, not tuned against a real workload |
The assumed throughputs and overheads in impact-model.ts | Starting assumptions, unvalidated |
Spark defaults by release
sparkConfDefault() (packages/core/src/spark-conf.ts) answers a property's default for the run's Spark version from this table. A property introduced by a release has no default before it.
| Property | Default | From |
|---|---|---|
spark.speculation.multiplier | 1.5, then 3 | 4.0.0 |
spark.speculation.quantile | 0.75, then 0.9 | 4.0.0 |
spark.sql.adaptive.enabled | false, then true | 3.2.0 |
spark.sql.adaptive.skewJoin.enabled, spark.sql.adaptive.coalescePartitions.enabled | true | 3.0.0 |
spark.sql.adaptive.skewJoin.skewedPartitionFactor = 5, spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes = 256MB | A partition is skewed above both the factor times the median partition and the threshold | 3.0.0 |
spark.sql.adaptive.forceOptimizeSkewedJoin | false; true accepts the extra shuffle a split can cost | 3.3.0 |
spark.sql.adaptive.advisoryPartitionSizeInBytes | 64MB (falls back to spark.sql.adaptive.shuffle.targetPostShuffleInputSize) | 3.0.0 |
spark.sql.adaptive.coalescePartitions.parallelismFirst | true; AQE then sizes merged tasks from defaultParallelism too, so a larger advisory size alone does not merge more | 3.2.0 |
spark.executor.memoryOverheadFactor | 0.1 (0.4 for Kubernetes non-JVM jobs, which the log does not tell apart); overhead is max(factor x executor memory, minimum) when spark.executor.memoryOverhead is unset | 3.3.0 |
spark.executor.minMemoryOverhead | 384m | 4.0.0 |
spark.sql.execution.pythonUDF.arrow.enabled | false, then true | 3.4.0 (introduced), 4.2.0 (on) |
The remaining keys in the table (spark.speculation, spark.executor.memory, spark.serializer, spark.shuffle.service.enabled, spark.locality.wait, the spark.eventLog switches, the dynamic-allocation bounds, spark.sql.shuffle.partitions, spark.sql.autoBroadcastJoinThreshold, spark.sql.files.maxPartitionBytes) have one default on every release.
Calibration data
Calibrated on 14 real logs. Each figure counts findings or groups the default thresholds produce on them. A 0.5% runtime floor drops findings on stages that barely ran, where the claim is clipped to the stage and would grade info; the condition is still true there.
| Detector | Calibration figure |
|---|---|
duplicatePlanSubtree | 270 of 546 duplicate groups differ beyond AQE query-stage numbers (occurrencesIdentical: false); 340 of 545 findings sit on repeats whose stages are shorter than the floor; a WholeStageCodegen wrapper or an Exchange write half is the only operator outside the subtree on 426 stages |
stageShape (lowParallelism) | 2,839 of 3,005 findings fall under the floor, 2,168 of them on sub-second stages |
shuffle | 182 of 284 findings fall under the floor |
spill | 12 of 38 findings fall under the floor, all info |
gc (low GC) | 464 of 685 low-GC notes fall under the floor (701 gc findings in all) |
slowHost | 324 of 452 findings fall under the floor, all info |
straggler | 671 of 753 findings fall under the floor, none above info. Scored against a task-level replay of every stage (recoverable = replay with each task over 4x P50 capped at P50), admitting 2.5-5% shares whose clipped tail already clears floorPctWarn finds 3 stages (10-48 s) for 1 borderline miss; admitting every 2.5% share adds 86 findings below the floor |
tinyTask | 132 of 151 findings fall under the floor |
coldStart | The driver's own startup before the first job takes 36-47 s on every log, so the gap is measured from the first stage's submission, not from application start |