PLAN

MERGE rewrite

A Delta MERGE rewrites most of the rows it did not change.

Outcome: SQL plans
Scope: Judged per SQL query
Usual fix: A change in the job code
CI gate: No budget flag: gate it with --fail-on-introduced against a baseline

An example finding

Where you see it

Dashboard

The PLAN tag marks it in the action region of the dashboard, among the widgets that rank a fix. How to read it in the findings guide.

Agent (MCP)

An assistant can read this entry with the list_detectors tool and its background with get_finding_documentation. The call and the response below come from a real run of the MCP server.

Call

{
  "tool": "list_detectors",
  "arguments": {
    "type": "mergeRewrite"
  }
}

Response

{
  "count": 1,
  "detectors": [
    {
      "type": "mergeRewrite",
      "name": "MERGE rewrite",
      "tag": "PLAN",
      "tagSharedWith": [
        "duplicatePlanSubtree",
        "smallFiles",
        "overBroadcast",
        "underBroadcast",
        "pythonUdf",
        "nestedLoopJoin",
        "unprunedScan",
        "joinKeyCast"
      ],
      "family": "plan",
      "pitch": "A Delta MERGE rewrites most of the rows it did not change.",
      "region": "action",
      "scope": "sql",
      "fixEffort": "code",
      "thresholdSummary": "a MERGE that rewrote at least 100,000 unmodified rows, at least 2x the rows it changed, or kept 90% of at least 8 target files after skipping while changing at most 50% of the partitions it scanned",
      "detectors": [
        {
          "detector": "mergeRewrite",
          "version": 1,
          "thresholds": {
            "minCopiedRows": 100000,
            "minCopiedToChangedRatio": 2,
            "minTargetFiles": 8,
            "minFilesKeptShare": 0.9,
            "maxTouchedPartitionShare": 0.5
          }
        }
      ],
      "pages": {
        "catalog": "docs/detectors/mergeRewrite.html",
        "guide": "docs/user-guide/understanding-findings.html#plan"
      },
      "tuning": {
        "kind": "chapter",
        "anchor": "#table-formats",
        "path": "docs/tuning-reference/table-formats.html#table-formats"
      },
      "gate": null
    }
  ]
}

get_finding_documentation call

{
  "tool": "get_finding_documentation",
  "arguments": {
    "type": "mergeRewrite"
  }
}

Its response, with each Markdown body replaced by its length

{
  "type": "mergeRewrite",
  "name": "MERGE Rewrite",
  "detectionDoc": {
    "tag": "PLAN",
    "title": "Plan advisor",
    "content": "<7876 characters of Markdown>"
  },
  "tuningDoc": {
    "anchor": "#table-formats",
    "title": "Table Formats",
    "content": "<10919 characters of Markdown>"
  }
}

CI (CLI)

List this entry from the command line. The output below is a real run of the command. Budgets and exit codes are in the CI guide.

sparkforensics-analyze detectors --type mergeRewrite --format md

# Detectors (1)

## SQL plans

| Tag | Type | Scope | Fix | Gate | What it finds |
| --- | --- | --- | --- | --- | --- |
| PLAN | `mergeRewrite` | sql | code |  | A Delta MERGE rewrites most of the rows it did not change. |

Report only this finding type for a run with --type:

sparkforensics-analyze <event-log> --type mergeRewrite --impact critical,warning,info

It has no budget flag. Fail a build on a new one by comparing against a baseline with --fail-on-introduced.

Airflow

The operator has no budget parameter for it. Use fail_on_introduced with a baseline.

check_spark_job = SparkForensicsOperator(
    task_id="check_spark_job",
    log_source=...,
    backend=SubprocessAnalyzeHook(),
    baseline_log_source=...,
    fail_on_introduced="critical",
)

The operator works per run, and its summary counts findings per impact band. See the operator for its parameters.

How it is measured

Thresholds

A MERGE that rewrote at least 100,000 unmodified rows, at least 2x the rows it changed, or kept 90% of at least 8 target files after skipping while changing at most 50% of the partitions it scanned.

mergeRewrite version 1

Threshold	Default
`minCopiedRows`	100000
`minCopiedToChangedRatio`	2
`minTargetFiles`	8
`minFilesKeptShare`	0.9
`maxTouchedPartitionShare`	0.5

Each rule's impact bands and fixed fallbacks are in the threshold tables.

Override a default per run with --thresholds; see Tuning detector thresholds.

Docs

This finding is covered in a general tuning chapter: Spark tuning reference.
Findings guide: PLAN in Understanding findings.
Shares the PLAN tag with: duplicate plan subtree, small files, oversized broadcast join, missed broadcast join, row-at-a-time Python UDF, nested loop join, unpruned partitioned scan, cast on a join key.