SHAPE

Stage shape

A stage runs too few tasks for its cores, or its output dwarfs its input.

Outcome: Speed
Scope: Judged per stage
Usual fix: A change in the job code
CI gate: No budget flag: gate it with --fail-on-introduced against a baseline

An example finding

Where you see it

Dashboard

The SHAPE tag marks it in the action region of the dashboard, among the widgets that rank a fix. Every affected stage is flagged, not only the worst.How to read it in the findings guide.

Agent (MCP)

An assistant can read this entry with the list_detectors tool and its background with get_finding_documentation. The call and the response below come from a real run of the MCP server.

Call

{
  "tool": "list_detectors",
  "arguments": {
    "type": "stageShape"
  }
}

Response

{
  "count": 1,
  "detectors": [
    {
      "type": "stageShape",
      "name": "stage shape",
      "tag": "SHAPE",
      "tagSharedWith": [],
      "family": "speed",
      "pitch": "A stage runs too few tasks for its cores, or its output dwarfs its input.",
      "region": "action",
      "scope": "stage",
      "fixEffort": "code",
      "thresholdSummary": "under 0.5 tasks per core, output over 10× input, or one task spanning over 50% of the stage's wall-clock at over 3× the median task",
      "detectors": [
        {
          "detector": "stageShape",
          "version": 2,
          "thresholds": {
            "pRatioMax": 0.5,
            "oiRatioMax": 10,
            "skewWarn": 3,
            "stageShareMin": 0.5,
            "lowParallelismFloorPct": 0.005,
            "taskStageSkewFloorPct": 0.005
          }
        }
      ],
      "pages": {
        "catalog": "docs/detectors/stageShape.html",
        "guide": "docs/user-guide/understanding-findings.html#shape"
      },
      "tuning": {
        "kind": "section",
        "anchor": "#bottleneck-stage-shape",
        "path": "docs/tuning-reference/bottleneck-skew.html#bottleneck-stage-shape"
      },
      "gate": null
    }
  ]
}

get_finding_documentation call

{
  "tool": "get_finding_documentation",
  "arguments": {
    "type": "stageShape"
  }
}

Its response, with each Markdown body replaced by its length

{
  "type": "stageShape",
  "name": "Stage Shape",
  "detectionDoc": {
    "tag": "SHAPE",
    "title": "Stage shape",
    "content": "<386 characters of Markdown>"
  },
  "tuningDoc": {
    "anchor": "#bottleneck-stage-shape",
    "title": "Task Skew",
    "content": "<7179 characters of Markdown>"
  }
}

CI (CLI)

List this entry from the command line. The output below is a real run of the command. Budgets and exit codes are in the CI guide.

sparkforensics-analyze detectors --type stageShape --format md

# Detectors (1)

## Speed

| Tag | Type | Scope | Fix | Gate | What it finds |
| --- | --- | --- | --- | --- | --- |
| SHAPE | `stageShape` | stage | code |  | A stage runs too few tasks for its cores, or its output dwarfs its input. |

Report only this finding type for a run with --type:

sparkforensics-analyze <event-log> --type stageShape --impact critical,warning,info

It has no budget flag. Fail a build on a new one by comparing against a baseline with --fail-on-introduced.

Airflow

The operator has no budget parameter for it. Use fail_on_introduced with a baseline.

check_spark_job = SparkForensicsOperator(
    task_id="check_spark_job",
    log_source=...,
    backend=SubprocessAnalyzeHook(),
    baseline_log_source=...,
    fail_on_introduced="critical",
)

The operator works per run, and its summary counts findings per impact band. See the operator for its parameters.

How it is measured

Thresholds

Under 0.5 tasks per core, output over 10× input, or one task spanning over 50% of the stage's wall-clock at over 3× the median task.

stageShape version 2

Threshold	Default
`pRatioMax`	0.5
`oiRatioMax`	10
`skewWarn`	3
`stageShareMin`	0.5
`lowParallelismFloorPct`	0.005
`taskStageSkewFloorPct`	0.005

Each rule's impact bands and fixed fallbacks are in the threshold tables.

Override a default per run with --thresholds; see Tuning detector thresholds.

Docs

This finding is covered in a section of a tuning page: Spark tuning reference.
Findings guide: SHAPE in Understanding findings.