UTIL

Executor utilization

Executors sit idle for much of the run, so the cluster is larger than the work needs.

Outcome: Cost
Scope: Judged on the whole run
Usual fix: A Spark configuration change
Dashboard: Shown in the reference region, below the action widgets. It informs; it does not rank a fix.
CI gate: Budget flag --min-efficiency

An example finding

Where you see it

Dashboard

The UTIL tag marks it in the reference region of the dashboard, below the action widgets. How to read it in the findings guide.

Agent (MCP)

An assistant can read this entry with the list_detectors tool and its background with get_finding_documentation. The call and the response below come from a real run of the MCP server.

Call

{
  "tool": "list_detectors",
  "arguments": {
    "type": "utilization"
  }
}

Response

{
  "count": 1,
  "detectors": [
    {
      "type": "utilization",
      "name": "executor utilization",
      "tag": "UTIL",
      "tagSharedWith": [
        "rightSizing"
      ],
      "family": "cost",
      "pitch": "Executors sit idle for much of the run, so the cluster is larger than the work needs.",
      "region": "reference",
      "scope": "app",
      "fixEffort": "config",
      "thresholdSummary": "average executor utilization below 60%",
      "detectors": [
        {
          "detector": "utilization",
          "version": 1,
          "thresholds": {
            "minUtil": 0.6
          }
        }
      ],
      "pages": {
        "catalog": "docs/detectors/utilization.html",
        "guide": "docs/user-guide/understanding-findings.html#util"
      },
      "tuning": {
        "kind": "shared",
        "anchor": "#bottleneck-utilization",
        "path": "docs/tuning-reference/bottleneck-utilization.html#bottleneck-utilization"
      },
      "gate": {
        "cliFlag": "--min-efficiency",
        "airflowParam": "min_efficiency_pct"
      }
    }
  ]
}

get_finding_documentation call

{
  "tool": "get_finding_documentation",
  "arguments": {
    "type": "utilization"
  }
}

Its response, with each Markdown body replaced by its length

{
  "type": "utilization",
  "name": "Executor Utilization",
  "detectionDoc": {
    "tag": "UTIL",
    "title": "Low utilization",
    "content": "<1568 characters of Markdown>"
  },
  "tuningDoc": {
    "anchor": "#bottleneck-utilization",
    "title": "Executor Utilization",
    "content": "<11455 characters of Markdown>"
  }
}

CI (CLI)

List this entry from the command line. The output below is a real run of the command. Budgets and exit codes are in the CI guide.

sparkforensics-analyze detectors --type utilization --format md

# Detectors (1)

## Cost

| Tag | Type | Scope | Fix | Gate | What it finds |
| --- | --- | --- | --- | --- | --- |
| UTIL | `utilization` | app | config | `--min-efficiency` | Executors sit idle for much of the run, so the cluster is larger than the work needs. |

Report only this finding type for a run with --type:

sparkforensics-analyze <event-log> --type utilization --impact critical,warning,info

Fail the build when the run breaches a budget for it:

sparkforensics-analyze <event-log> --min-efficiency <n>

Airflow

The operator takes the budget as the min_efficiency_pct parameter.

check_spark_job = SparkForensicsOperator(
    task_id="check_spark_job",
    log_source=...,
    backend=SubprocessAnalyzeHook(),
    min_efficiency_pct=...,  # your budget
)

The operator works per run, and its summary counts findings per impact band. See the operator for its parameters.

How it is measured

Thresholds

Average executor utilization below 60%.

utilization version 1

Threshold	Default
`minUtil`	0.6

Each rule's impact bands and fixed fallbacks are in the threshold tables.

Override a default per run with --thresholds; see Tuning detector thresholds.

Docs

This finding shares a tuning page with related findings: Spark tuning reference.
Findings guide: UTIL in Understanding findings.
Shares the UTIL tag with: right-sized cluster.