SparkForensics

Appearance

Detector catalog / SQL plans

PLAN

Row-at-a-time Python UDF

Row-at-a-time Python functions slow a stage that vectorized code would speed up.

Outcome
SQL plans
Scope
Judged per SQL query
Usual fix
A change in the job code
CI gate
No budget flag: gate it with --fail-on-introduced against a baseline

An example finding

Where you see it

Dashboard

The PLAN tag marks it in the action region of the dashboard, among the widgets that rank a fix. How to read it in the findings guide.

Agent (MCP)

An assistant can read this entry with the list_detectors tool and its background with get_finding_documentation. The call and the response below come from a real run of the MCP server.

Call

{
  "tool": "list_detectors",
  "arguments": {
    "type": "pythonUdf"
  }
}

Response

{
  "count": 1,
  "detectors": [
    {
      "type": "pythonUdf",
      "name": "row-at-a-time Python UDF",
      "tag": "PLAN",
      "tagSharedWith": [
        "duplicatePlanSubtree",
        "smallFiles",
        "overBroadcast",
        "underBroadcast",
        "nestedLoopJoin",
        "mergeRewrite",
        "unprunedScan",
        "joinKeyCast"
      ],
      "family": "plan",
      "pitch": "Row-at-a-time Python functions slow a stage that vectorized code would speed up.",
      "region": "action",
      "scope": "sql",
      "fixEffort": "code",
      "thresholdSummary": "over 64 MiB sent to Python workers by stages running over 30 s",
      "detectors": [
        {
          "detector": "pythonUdf",
          "version": 1,
          "thresholds": {
            "minBytesSent": 67108864,
            "minStageMs": 30000
          }
        }
      ],
      "pages": {
        "catalog": "docs/detectors/pythonUdf.html",
        "guide": "docs/user-guide/understanding-findings.html#plan"
      },
      "tuning": {
        "kind": "chapter",
        "anchor": "#pyspark",
        "path": "docs/tuning-reference/pyspark.html#pyspark"
      },
      "gate": null
    }
  ]
}

get_finding_documentation call

{
  "tool": "get_finding_documentation",
  "arguments": {
    "type": "pythonUdf"
  }
}

Its response, with each Markdown body replaced by its length

{
  "type": "pythonUdf",
  "name": "Row-At-A-Time Python UDF",
  "detectionDoc": {
    "tag": "PLAN",
    "title": "Plan advisor",
    "content": "<7876 characters of Markdown>"
  },
  "tuningDoc": {
    "anchor": "#pyspark",
    "title": "PySpark Specifics",
    "content": "<10952 characters of Markdown>"
  }
}

CI (CLI)

List this entry from the command line. The output below is a real run of the command. Budgets and exit codes are in the CI guide. 

sparkforensics-analyze detectors --type pythonUdf --format md
# Detectors (1)

## SQL plans

| Tag | Type | Scope | Fix | Gate | What it finds |
| --- | --- | --- | --- | --- | --- |
| PLAN | `pythonUdf` | sql | code |  | Row-at-a-time Python functions slow a stage that vectorized code would speed up. |

Report only this finding type for a run with --type:

sparkforensics-analyze <event-log> --type pythonUdf --impact critical,warning,info

It has no budget flag. Fail a build on a new one by comparing against a baseline with --fail-on-introduced.

Airflow

The operator has no budget parameter for it. Use fail_on_introduced with a baseline.

check_spark_job = SparkForensicsOperator(
    task_id="check_spark_job",
    log_source=...,
    backend=SubprocessAnalyzeHook(),
    baseline_log_source=...,
    fail_on_introduced="critical",
)

The operator works per run, and its summary counts findings per impact band. See the operator for its parameters.

How it is measured

Thresholds

Over 64 MiB sent to Python workers by stages running over 30 s.

pythonUdf version 1

ThresholdDefault
minBytesSent67108864
minStageMs30000

Each rule's impact bands and fixed fallbacks are in the threshold tables.

Override a default per run with --thresholds; see Tuning detector thresholds.

Docs