MEM

Memory utilization

Executors reserve far more memory than tasks use, or run close to their limit.

Outcome: Cost
Scope: Judged on the whole run
Usual fix: A Spark configuration change
Dashboard: Shown in the reference region, below the action widgets. It informs; it does not rank a fix.
CI gate: No budget flag: gate it with --fail-on-introduced against a baseline

An example finding

Where you see it

Dashboard

The MEM tag marks it in the reference region of the dashboard, below the action widgets. How to read it in the findings guide.

Agent (MCP)

An assistant can read this entry with the list_detectors tool and its background with get_finding_documentation. The call and the response below come from a real run of the MCP server.

Call

{
  "tool": "list_detectors",
  "arguments": {
    "type": "memoryUtilization"
  }
}

Response

{
  "count": 1,
  "detectors": [
    {
      "type": "memoryUtilization",
      "name": "memory utilization",
      "tag": "MEM",
      "tagSharedWith": [],
      "family": "cost",
      "pitch": "Executors reserve far more memory than tasks use, or run close to their limit.",
      "region": "reference",
      "scope": "app",
      "fixEffort": "config",
      "thresholdSummary": "executor heap usage outside the configured band",
      "detectors": [
        {
          "detector": "memoryUtilization",
          "version": 1,
          "thresholds": {
            "idleCoreWarn": 0.5,
            "bandTooHigh": 0.7,
            "wasteBufferMultiplier": 1.5
          }
        }
      ],
      "pages": {
        "catalog": "docs/detectors/memoryUtilization.html",
        "guide": "docs/user-guide/understanding-findings.html#mem"
      },
      "tuning": {
        "kind": "page",
        "anchor": "#bottleneck-memory-utilization",
        "path": "docs/tuning-reference/bottleneck-memory-utilization.html#bottleneck-memory-utilization"
      },
      "gate": null
    }
  ]
}

get_finding_documentation call

{
  "tool": "get_finding_documentation",
  "arguments": {
    "type": "memoryUtilization"
  }
}

Its response, with each Markdown body replaced by its length

{
  "type": "memoryUtilization",
  "name": "Memory Utilization",
  "detectionDoc": {
    "tag": "MEM",
    "title": "Memory utilization",
    "content": "<1580 characters of Markdown>"
  },
  "tuningDoc": {
    "anchor": "#bottleneck-memory-utilization",
    "title": "Memory Utilization",
    "content": "<11519 characters of Markdown>"
  }
}

CI (CLI)

List this entry from the command line. The output below is a real run of the command. Budgets and exit codes are in the CI guide.

sparkforensics-analyze detectors --type memoryUtilization --format md

# Detectors (1)

## Cost

| Tag | Type | Scope | Fix | Gate | What it finds |
| --- | --- | --- | --- | --- | --- |
| MEM | `memoryUtilization` | app | config |  | Executors reserve far more memory than tasks use, or run close to their limit. |

Report only this finding type for a run with --type:

sparkforensics-analyze <event-log> --type memoryUtilization --impact critical,warning,info

It has no budget flag. Fail a build on a new one by comparing against a baseline with --fail-on-introduced.

Airflow

The operator has no budget parameter for it. Use fail_on_introduced with a baseline.

check_spark_job = SparkForensicsOperator(
    task_id="check_spark_job",
    log_source=...,
    backend=SubprocessAnalyzeHook(),
    baseline_log_source=...,
    fail_on_introduced="critical",
)

The operator works per run, and its summary counts findings per impact band. See the operator for its parameters.

How it is measured

Thresholds

Executor heap usage outside the configured band.

memoryUtilization version 1

Threshold	Default
`idleCoreWarn`	0.5
`bandTooHigh`	0.7
`wasteBufferMultiplier`	1.5

Each rule's impact bands and fixed fallbacks are in the threshold tables.

Override a default per run with --thresholds; see Tuning detector thresholds.

Docs

This finding has its own tuning page: Spark tuning reference.
Findings guide: MEM in Understanding findings.