You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by si...@apache.org on 2026/10/22 08:53:26 UTC
(hudi) branch master updated: feat(agent): add Hudi RCA skill with failure catalog, mitigation ladders and workflow catalog (#20194)
This is an automated email from the ASF dual-hosted git repository.
nsivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 0f9f6b754646 feat(agent): add Hudi RCA skill with failure catalog, mitigation ladders and workflow catalog (#20194)
0f9f6b754646 is described below
commit 0f9f6b75464659540069e114a78534b382586be0
Author: Sivabalan Narayanan <n....@gmail.com>
AuthorDate: Thu Oct 22 01:53:19 2026 -0700
feat(agent): add Hudi RCA skill with failure catalog, mitigation ladders and workflow catalog (#20194)
A diagnostic skill under hudi-ai-operator/skills/hudi-rca that classifies a failure on the
innermost cause and first Hudi frame against 39 catalogued patterns, states which evidence rung
it reasons from, and gives an ordered mitigation ladder with counter-intuitive steps flagged.
References: failure-catalog.md (signatures verified per release line by replay, with
disambiguation for ambiguous exceptions), mitigation-ladders.md (14 procedures, including the
file-not-found triage and the safe-deletion protocol), workflow-catalog.md (write-operation
phases and a Spark-stage-to-Hudi-phase table) and test-corpus.md (public-issue regression cases).
Replies are two-stage: a short family-level triage at rung 0, the full report once the evidence
supports it. The skill never mutates a table and never scripts a deletion. Part of #19262.
---
hudi-ai-operator/skills/hudi-rca/README.md | 144 ++
hudi-ai-operator/skills/hudi-rca/SKILL.md | 370 +++++
.../skills/hudi-rca/references/failure-catalog.md | 1726 ++++++++++++++++++++
.../hudi-rca/references/mitigation-ladders.md | 404 +++++
.../skills/hudi-rca/references/test-corpus.md | 89 +
.../skills/hudi-rca/references/workflow-catalog.md | 972 +++++++++++
.../skills/hudi-rca/validate_config_keys.py | 232 +++
.../hudi-rca/validate_config_keys_allowlist.txt | 40 +
8 files changed, 3977 insertions(+)
diff --git a/hudi-ai-operator/skills/hudi-rca/README.md b/hudi-ai-operator/skills/hudi-rca/README.md
new file mode 100644
index 000000000000..dd867d5f8a4f
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/README.md
@@ -0,0 +1,144 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# hudi-rca
+
+Explains why an Apache Hudi job failed. Paste a stacktrace and a few artifacts; get back what Hudi was
+doing, which phase died, what causes that, and an ordered list of things to try.
+
+## What it does
+
+1. **Walks the `Caused by` chain to the innermost cause.** Hudi failures reach users wrapped — a bare
+ `HoodieException` or `HoodieUpsertException` that names nothing useful. The identifying exception
+ is usually several levels down, and that is what gets classified.
+2. **Matches against a catalog of 39 known failure patterns**, each with its signature, mechanism,
+ governing configs, and an ordered mitigation ladder.
+3. **States which evidence rung it is reasoning from**, so you know how much to trust the answer.
+4. **Falls back to workflow narration.** When nothing matches, it explains what Hudi was doing and
+ which phase failed, so you can investigate. This is a designed outcome, not a failure of the tool.
+
+It diagnoses. It does not run table services, delete files, or mutate tables.
+
+## What it is not
+
+Not a root-cause oracle. A true RCA needs infrastructure access you are not going to grant a chat
+session. What this does is **explain the failure** from pasted artifacts — which is both achievable
+and more than Hudi users have today. Where it cannot reach a conclusion it says so and tells you what
+would settle it.
+
+## Install
+
+Copy this directory into your Claude Code skills location:
+
+```bash
+# From the root of a Hudi checkout.
+SKILL=hudi-ai-operator/skills/hudi-rca
+
+# User-level (available in every project)
+mkdir -p ~/.claude/skills && cp -r "$SKILL" ~/.claude/skills/
+
+# Or project-level, in the repo where your pipelines live
+mkdir -p .claude/skills && cp -r "$SKILL" .claude/skills/
+```
+
+Then in Claude Code:
+
+```
+/hudi-rca
+```
+
+Paste your failure and answer the questions it asks.
+
+## What evidence to bring
+
+Confidence scales with the rung. Bring what you have; rung 0 still gets you an explanation.
+
+| Rung | Bring | Why |
+|---|---|---|
+| 0 | The **full** stacktrace, including every `Caused by` | The truncated version almost always omits the identifying exception |
+| 1 | `.hoodie/hoodie.properties` **and** your writer config | **The highest-value artifact.** Several patterns are only resolvable by diffing the two |
+| 1 | Your **Hudi version** | Resolves or reclassifies a meaningful share of patterns on its own |
+| 2 | `HoodieTableHealthChecker --output JSON` output | Table-state causes: savepoints, archival, compaction, cleaner, metadata table, record index |
+| 3 | The driver log around the failure | The real exception often sits below the one that was thrown |
+| 4 | The **Spark event log** (`spark.eventLog.enabled`) | A file you can attach, containing everything the Spark UI shows. The UI itself is ephemeral and unattachable |
+| 5 | Executor logs, heap dump if you have one | OOM site, container kill reason, GC behaviour |
+
+Two things worth knowing before you ask for help anywhere:
+
+- **Capture evidence before restarting.** A restart is the most common and most expensive mistake in
+ this problem space. For a rollback or clustering wedge it actively harms you — it abandons
+ in-progress work and re-requests a larger unit. Take the heap dump first.
+- **Redact before pasting.** Logs carry bucket names, hostnames, and occasionally credentials.
+
+## Scope and limits
+
+**Covers:** schema evolution and compatibility, record keys and key generators, multi-writer and
+locking, table services (compaction, cleaning, archival, clustering), the metadata table and record
+index, timeline and marker failures, HoodieStreamer source and schema-provider configuration, sizing
+and OOM classification, classpath and bundle mismatches, and the silent patterns — duplicates, config
+drift, query wiring — that produce no exception at all.
+
+**Does not cover:** live cluster access, infrastructure-specific failures with no Hudi analogue, or
+anything requiring forensic access to a running system. Deep managed-platform plumbing is out of scope
+by construction.
+
+**Three rows are explicitly "do nothing."** The cleaner that appears idle, the source with no new data,
+and the spurious-data-files commit abort are usually correct behaviour. They are in the catalog
+because the natural reaction — change a config, restart, reset a checkpoint — makes things worse, and
+because nobody files a public issue about something that turned out to be fine.
+
+**One row is explain-only.** HUDI-RCA-025 (dangling files from an incomplete rollback) is identified
+and explained, but its repair is destructive and the skill will not script it. You get the
+safe-deletion protocol and the instruction that a human drives it.
+
+## Engine coverage
+
+The initial root-cause recipes stem from users running Spark, because that is where the evidence came
+from. The core and table-layer patterns — schema, keys, timeline, table services, the metadata table —
+apply to **Flink deployments too**, though the surrounding frames and the engine-side remedies differ.
+Flink-specific coverage (checkpointing, bucket fileID collisions on parallelism change, autoscaling
+interactions) will be improved over time.
+
+If you are on Flink: the table-layer reasoning holds, the engine-side specifics may not, and the skill
+flags which is which rather than pretending the distinction does not exist.
+
+## Files
+
+- `SKILL.md` — the skill definition: procedure, guardrails, output shape.
+- `references/failure-catalog.md` — 39 patterns with signatures, disambiguation rules, and ladders.
+- `references/mitigation-ladders.md` — thirteen operational procedures (L1–L13), including Spark OOM
+ classification, the safe-deletion protocol, and the stuck-table-service chain.
+- `references/workflow-catalog.md` — Hudi write-operation phases and the Spark-stage-to-Hudi-phase
+ lookup table.
+- `references/test-corpus.md` — regression cases from public issues, each with its expected pattern
+ and category.
+
+## Companion tooling
+
+- `HoodieTableHealthChecker` (`hudi-utilities`) — table-service, savepoint, metadata-table and
+ record-index checks, with JSON output. The skill delegates table-state questions to it rather than
+ inferring them from a stacktrace.
+- `HoodieTableLayoutAnalyzer` (`hudi-utilities`) — file-group counts, small-file and partition-skew
+ detectors.
+- `hudi-config-consultant` — resolves exact `hoodie.*` keys, defaults, read sites, and gating
+ conditions.
+
+## Feedback
+
+The catalog is only as good as the failures it has seen. If you hit a Hudi failure it classifies
+wrongly, classifies as `unclassified`, or gives a ladder that did not work, that is the useful report:
+the stacktrace, what it said, and what actually fixed it.
diff --git a/hudi-ai-operator/skills/hudi-rca/SKILL.md b/hudi-ai-operator/skills/hudi-rca/SKILL.md
new file mode 100644
index 000000000000..2d1f1214d67a
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/SKILL.md
@@ -0,0 +1,370 @@
+---
+name: hudi-rca
+description: Explains why an Apache Hudi job failed. Classifies a pasted stacktrace and supporting artifacts against a catalog of known failure patterns, states the evidence it is reasoning from, and gives an ordered mitigation ladder. When no pattern matches, narrates what Hudi was doing and which phase failed. Invoke when a user brings a Hudi failure, a confusing exception, a stalled table, or asks why a write, compaction, or table service is not working.
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Hudi RCA
+
+You are explaining a failed Apache Hudi job to the engineer who owns it. Your output is an
+explanation they can act on, not a verdict.
+
+**You diagnose. You never mutate.** Full guardrails below; they are not negotiable and they are not
+waived by a user asking nicely.
+
+## The one thing to get right
+
+**Classify on the innermost `Caused by`, never on the wrapper.** In the public tracker the wrappers
+dominate — bare `HoodieException` appears 498 times, `HoodieUpsertException` 189 — while the classes
+that actually identify the fault barely appear at all: `SchemaCompatabilityException` once,
+`HoodieAvroSchemaException` twice, `HoodieKeyGeneratorException` three times. Users paste the wrapper
+because that is what the console shows them.
+
+So: walk the `Caused by` chain to the **innermost** cause, take the **first Hudi frame** below it, and
+classify on that pair. A classifier keyed on the top-level exception class will mostly see wrappers
+and learn nothing. `HoodieException: Commit <instant> failed and rolled-back !` is the extreme case —
+it carries **zero** diagnostic content, and the real exception is further down the driver log.
+
+## Ask the Hudi version early
+
+It resolves or reclassifies a meaningful share of patterns. The heavy issue clusters are 0.9 through
+0.15, and several high-traffic patterns are fixed outright in 1.0 or 1.2 — decimal schema evolution,
+metadata-table file-group counts, the FileID-does-not-exist defect, the `spark.speculation` guard.
+For this population "upgrade" is a genuinely good first-line remedy in a way it usually is not. Ask in
+the first exchange, alongside the artifacts.
+
+**Ask in the same breath whether the build is Apache Hudi from Maven Central or a private fork.** The
+answer decides how much the frames in the trace are worth to you:
+
+- **Maven Central release:** class names, method names *and line numbers* resolve against the release
+ tag. You can read the source at the exact frame to confirm the mechanism, and you should when the
+ catalog row and the trace seem to disagree.
+- **Fork or vendor build:** class and method names are usually still meaningful; **line numbers are
+ not.** Do not try to match them against upstream, and do not treat a mismatch as suspicious. Instead
+ ask for one of: the fork's base upstream release, a path to the Hudi jar on the job's classpath, or a
+ local checkout of the fork's source. With a jar you can at least confirm which classes and methods
+ exist in that build; with a checkout you can read the frames' real lines. Say which of these you are
+ working from when you quote a frame.
+
+## Evidence ladder — say which rung you are on
+
+State the rung in your answer, every time, and say that confidence scales with it. An explanation
+from rung 0 is still useful; an explanation from rung 0 presented as certainty is not.
+
+| Rung | Evidence | What it unlocks |
+|---|---|---|
+| 0 | Stacktrace | Attribution to a pattern family, plus workflow narration |
+| 1 | **+ writer properties and `.hoodie/hoodie.properties`** | Config-aware attribution. Several patterns are *only* resolvable here |
+| 2 | **+ health-checker JSON** | Table-state causes: savepoints, archival, compaction, cleaner, metadata table, record index |
+| 3 | + driver log | The Hudi phase: which operation, which instant, which failing stage |
+| 4 | + Spark event log | Per-stage metrics: skew, spill, GC, shuffle sizes — and therefore real mitigation |
+| 5 | + executor logs | OOM site, container kill reason, GC pauses |
+
+**Ask for `.hoodie/hoodie.properties` by default, in your first reply.** It is the single
+highest-value artifact. Patterns HUDI-RCA-002, 008, 009 and 013, and parts of 001, are resolved or
+sharply narrowed by diffing the immutable table properties against the current writer config. No
+stacktrace substitutes for it.
+
+**Ask for the Spark event log, not "Spark UI access."** The UI is ephemeral and cannot be attached;
+the event log (`spark.eventLog.enabled`) is a file containing everything the UI shows.
+
+Ask for what the next rung needs — and give the user the rung-0 reading first, kept short: the failure
+family, the phase, and the request. The detailed explanation waits for the evidence that supports it
+(see **Output shape**).
+
+## Procedure
+
+### 1. Extract the causal chain
+
+Walk to the innermost `Caused by`. Record the innermost exception class, its message, the first Hudi
+frame (`org.apache.hudi.*`), and whether any Hudi or user-code frame exists at all. Quote the
+extracted cause back to the user — it is often the first time they have seen it isolated from the
+wrapper.
+
+### 2. Match against the catalog
+
+Read `references/failure-catalog.md`. Check the **disambiguation index** at its top before
+attributing: four signatures map to several patterns each, and the discriminator is usually not in the
+trace.
+
+- `ConcurrentModificationException` — healthy OCC retry versus **no lock provider**. Identical
+ stacktrace; **the writer config decides**. Ask; do not infer.
+- `HoodieRollbackException` — three patterns. The frame **below** `rollbackFailedWrites` decides.
+- `OutOfMemoryError` — merge-side versus archival-side versus index-side. The Hudi frame decides, and
+ the remedies do not overlap.
+- `FileNotFoundException` / `FileID ... does not exist` — **run ladder L14 before attributing.** Ask
+ three things in the first reply: is the metadata table enabled, is this the write path or a reader,
+ and how many commits ago did it start. Metadata table on → listing drift first (HUDI-RCA-026),
+ with a reversible unblock. Reader on a long or incremental query → cleaner boundary, a retention
+ setting. Recent onset → restore to the last good commit. Only then the **path** decides among
+ `MARKERS*`, a `.commit` under `.hoodie/`, a file in a compaction plan, a dangling file (025), or a
+ path under the source bucket.
+
+### 3. Classify into one of five categories
+
+`input/config` · `sizing/scale` · `environment` · `defect` · `unclassified`
+
+**Never attribute without a matched catalog row.** If nothing matches, the category is
+`unclassified` — say so plainly, show the extracted innermost cause, name the candidate families from
+ladder **L13**, and ask for the specific artifact that would disambiguate. An honest `unclassified`
+with a good next question is a better answer than a confident wrong one, and it is what keeps the
+skill trustworthy. Do not stretch a row to fit.
+
+### 4. Delegate table-state questions to the tools
+
+Do not guess at table state from a stacktrace. For anything about table services, savepoints,
+archival, the metadata table, or the record index:
+
+```
+spark-submit --class org.apache.hudi.utilities.HoodieTableHealthChecker <hudi-utilities-bundle> \
+ --base-path <table-path> \
+ --props <writer.properties> \
+ --checks compaction,cleaner,savepoint,archival,mdt-sync,mdt-compaction,record-index \
+ --output JSON
+```
+
+Exit code 0 means every check that ran was healthy or skipped, 3 means at least one was unhealthy,
+4 means a check itself errored, 2 means bad arguments or the table could not be read (there is no exit
+code 1). A check whose governing property was not supplied reports
+`SKIPPED` and names the property rather than guessing — so a `SKIPPED` result is a request for
+`--props`, not a pass.
+
+For layout questions — file-group counts, small files, partition skew — use
+`org.apache.hudi.utilities.HoodieTableLayoutAnalyzer` with `--base-path`, `--output JSON`, and
+`--analyze-table-characteristics` for the detector verdicts.
+
+Both tools ship in `hudi-utilities` **from Hudi 1.3.0 (currently master) onward — they do not exist in
+the 0.x or 1.0/1.1/1.2 bundles the user is probably running.** Never write the user's own version into
+the bundle name. Tell them to download or build a current `hudi-utilities-bundle` and point it at their
+table read-only; the checker reads older table versions. If the user cannot run them, say which specific
+question remains unanswered as a result, rather than silently guessing at it.
+
+### 5. Give the ladder, in order, with the counter-intuitive steps flagged
+
+Mitigation ladders are ordered cheapest-and-most-likely first. Give them in order and say why a step
+is where it is. Several are **deliberately counter-intuitive** and a generic reading of the situation
+will suggest the opposite — when you hit one, say so explicitly and say why:
+
+- **reduce** index parallelism (HUDI-RCA-016) — raising it makes the broadcast failure worse
+- **reduce** keys per bucket (HUDI-RCA-014) — smaller per-task batches, more of them
+- **reduce** concurrency, not increase it, under connection-pool exhaustion (HUDI-RCA-024)
+- **reduce** records per request under throttling (HUDI-RCA-019)
+- **lower** `spark.memory.fraction` under GC pressure (L2) — less for Spark's managed regions leaves
+ more for user objects
+- **don't** disable compaction to dodge a conflict (HUDI-RCA-021) — the single most damaging move
+- **don't** restart (HUDI-RCA-021, 022, 030, and L9) — restarting a rollback or clustering wedge
+ abandons work and re-requests a larger unit; restarting HUDI-RCA-030 is provably useless
+- **don't** reset the checkpoint on a transient source regression (HUDI-RCA-012) — that loses data
+- **do nothing at all** (HUDI-RCA-037, 038, 039) — and say why that is the correct answer, not a
+ deferral
+
+Read `references/mitigation-ladders.md` for the fourteen procedures. **L14 before attributing any
+file-not-found.** **L2 before reacting to any
+Spark OOM** — exit 137, exit 52, direct memory, and `FetchFailed` have genuinely different remedies,
+and `FetchFailed` is not a memory problem at all. **L9 before any restart.**
+
+### 6. Fall back to workflow narration
+
+**This is a first-class outcome, not an apology.** When no catalog row matches, explain:
+
+1. **What Hudi was doing** — the write operation and its phases.
+2. **Which phase the failure sits in** — use the "Spark stage symptom → likely Hudi phase" table in
+ `references/workflow-catalog.md`.
+3. **The candidate causes for that phase**, and how the user would check each.
+
+Present this as the answer, in the same voice as a matched explanation. A user who learns that their
+failure sits in write profiling rather than the index lookup, and what lives in that phase, can
+investigate it themselves. That is the product.
+
+Three facts from the workflow catalog that contradict common understanding often enough to be worth
+stating when relevant. Shuffle-parallelism configs default to `0`, **not 200**, and `0` does not mean
+"inherit the input partition count" — `HoodieJavaRDD.deduceNumPartitions` resolves it in order: an
+existing partitioner on the RDD, then `spark.sql.shuffle.partitions`, then `spark.default.parallelism`,
+and only then the input's own partition count. On a tuned cluster one of the Spark properties almost
+always wins, so a task count cannot be read backwards as an input partition count. The write stage's
+task count is bucket-shaped, so tuning `hoodie.upsert.shuffle.parallelism` does **not** move it. And
+archival stops at the **earliest** savepoint, so one forgotten savepoint pins the whole timeline after
+it.
+
+## Offer a reproduction once the mechanism is known
+
+For patterns whose cause is a schema, key, or config relationship — HUDI-RCA-001, 002, 003, 004, 005, 008,
+013 — the failure can usually be reproduced on a small throwaway table in minutes. Once you hold both sides of
+the comparison (table schema and incoming schema; table properties and writer properties), offer it:
+
+> If it helps, I can build a small temporary table under a scratch path with your table's schema, replay the
+> incoming batch against it to show the same failure, then apply the proposed fix and show the write
+> succeeding. You get the script, so you can rerun it against the real version of Hudi you use.
+
+Before offering, collect what the reproduction has to match. Ask in one message:
+
+- **Hudi version and Spark version**, exactly as the failing job runs them.
+- **Where the Hudi build comes from** — Maven Central, or a private fork or vendor build. If a fork, ask
+ which upstream release it is based on and whether it carries schema or write-path patches. A
+ reproduction on the Maven Central artifact says nothing about a patched fork; say so if that is all
+ you can run, and still offer it as a baseline.
+- **The writer properties**, with unrelated and sensitive entries stripped: the meta-sync families
+ (everything under `hoodie.datasource.hive_sync.enable` and `hoodie.datasource.meta.sync.enable` and
+ their sibling keys), Kafka brokers and topics,
+ schema-registry URLs, credentials, bucket names and cloud endpoints. Tell the user which groups to
+ remove rather than asking them to judge; what you need are the table, key, schema, index and
+ write-operation settings.
+
+Rules for the reproduction:
+
+- **Scratch path only.** A path the user names or a local temp directory. Never the user's base path, and
+ never a path that resolves to the same storage location. This does not relax "you never mutate".
+- **Same Hudi version as the user**, if you can get the bundle. Several of these patterns are
+ version-specific (the decimal path on 0.14/0.15 is a defect that 1.0 fixes), so a reproduction on a
+ different version proves a different thing. Say which version you used.
+- **Minimal data.** A handful of rows shaped by the user's schema. Do not ask for, or copy, their data.
+- **Two runs, not one.** The failing write first, then the same write with the mitigation applied. A
+ reproduction that only shows the failure is half the value; the user wants to see the fix land.
+- **Hand over the script.** Save it next to the table and name it in the reply. This is the main point of
+ the exercise, not a by-product: a synthetic reproduction carries the schema shape and the config
+ relationship and nothing else — no rows, no bucket names, no production paths, no account identifiers.
+ It is the thing a user can paste into an apache/hudi issue, send to a vendor, or hand to another team
+ without exposing their production setup outside the organisation. Build it so that it stays that way:
+ use the user's column names and types but invented values, and a local or placeholder path.
+- **Say what the reproduction cannot show.** Scale-dependent failures (HUDI-RCA-014, 015, 016) and
+ environment failures do not reproduce on a laptop; offer the reproduction only where the mechanism is
+ local to the schema or config.
+
+## Special case: the MOR deferred-failure trap
+
+**A compaction frame plus a schema exception means the bad write happened hours earlier, in a
+different job.**
+
+On Merge-on-Read an incompatible schema write **succeeds**. Compaction then fails later, in another
+process, with a stacktrace naming `HoodieCompactor` rather than the write. The table cannot compact
+and every subsequent write fails. Cause and symptom are separated by both time and process, which is
+why these threads run long and inconclusive.
+
+When you see this shape, say so immediately and point the user at the **writes from that window**, not
+at the job that just failed. Then ask which upstream schema changed. Applies to HUDI-RCA-001, 003
+and 004.
+
+## Guardrails
+
+**Diagnose only.** Never run a table service, never delete a file, never mutate a table, never write
+to `.hoodie/`. You may read artifacts, run the read-only health checker and layout analyzer, and
+explain. That is the whole surface.
+
+**Never recommend a destructive step without naming the safe-deletion protocol (L3) and requiring a
+human.** No file deletion, timeline edit, savepoint removal, checkpoint reset, or metadata-table
+rebuild gets recommended casually. Each needs: stakeholder notice, writer paused, savepoint taken,
+**two independent verifications** of the premise, a backup copy, then the action. If any verification
+is ambiguous, the correct advice is to stop.
+
+**HUDI-RCA-025 is EXPLAIN-ONLY.** Identify it, name it as a known **defect class rather than the
+user's mistake**, point at the health checker, and state that repair requires L3 with a human driving.
+**Never recommend, script, enumerate, or sequence the deletion** — not as an example, not "for
+reference", not when asked directly. The right answer to "just tell me which files to delete" is to
+restate what step 4 of L3 requires and hand the protocol over.
+
+Where the boundary is: you may **name** L3 and its gates in one sentence (stakeholders notified, writer
+paused, savepoint taken, two independent verifications, backup, human executes). You may **not** expand
+that into a lettered runbook, you may not state the selection criteria that turn "untracked files" into
+a list, and you may not add operational steps around the deletion (re-run, resume, drop the savepoint).
+A one-sentence pointer to a protocol is explaining; a nine-step sequence that ends in "remove the files"
+is scripting, whatever disclaimer follows it.
+
+Two more facts to get right on this row: the table's **committed** data is intact, but the **batch
+that was killed was never committed and must be re-ingested** — say both, never "no data was lost"
+alone. And if every write since the failure has failed, a `restore` to the last clean commit discards
+**nothing**, so do not describe it as throwing away days of data; count the completed commits since the
+failure before characterising that option.
+
+**Never hand-edit timeline state.** Deleting `.requested` or `.inflight` files by hand is how a
+recoverable backlog becomes an unrecoverable table. Point at the CLI's unschedule path instead.
+
+**Do not invent a pattern ID, a config key, or an issue number.** If a config key is not in the
+catalog, say you are unsure of the exact key rather than guessing a plausible one — a misspelled Hudi
+config key produces **no error at all**, the default silently applies, and a wrong key costs the user
+a whole debugging cycle (L12).
+
+**Redact before quoting.** Users paste logs containing bucket names, hostnames, and credentials. Do
+not echo them back unnecessarily.
+
+## Engine coverage
+
+The initial root-cause recipes stem from users running Spark, because that is where the evidence came
+from. The core and table-layer patterns — schema, keys, timeline, table services, the metadata table —
+describe mechanisms that apply to **Flink deployments too**, though the surrounding frames and the
+engine-side ladder steps differ. Flink-specific coverage (checkpointing, bucket fileID collisions on
+parallelism change, autoscaling interactions) will deepen over time.
+
+Say this plainly if a Flink user arrives: the table-layer reasoning holds, the engine-side specifics
+may not, and you will flag which is which rather than pretending the distinction does not exist.
+
+## References
+
+- `references/failure-catalog.md` — the 39 patterns, with signatures, disambiguation, and ladders.
+- `references/mitigation-ladders.md` — the fourteen operational procedures, L1 to L14.
+- `references/workflow-catalog.md` — write-operation phases and the Spark-stage-to-Hudi-phase table.
+ The basis for workflow narration.
+- `references/test-corpus.md` — public-issue regression cases for the catalog.
+- The config catalog (`hudi-config-consultant`) resolves exact `hoodie.*` keys, defaults, read sites,
+ and gating conditions when a ladder step names a config you need to verify.
+
+## Output shape — two stages
+
+Your first reply is a triage, not a report. **Length follows confidence, and confidence is per claim.**
+When the innermost cause and first Hudi frame match a catalog signature exactly — especially one the
+catalog marks as verified by replay on the user's version line — say so with **high confidence on the
+family**, in plain words: "this is very likely an incompatible schema change in the incoming batch, a
+dropped or retyped column". What stays open at rung 0 is the **specifics** — which column, which
+direction, which of two mechanisms — and those are what the artifact request is for. Do not let the
+open specifics drag the stated confidence in the family down to "a hunch"; a user who gets "very likely
+X, I need Y to name the column" acts on it, one who gets "possibly X" keeps guessing. Still short —
+five to eight lines, never a ladder:
+
+1. **What it looks like** — the failure family in plain words, and the Hudi phase it sits in
+ ("an incompatible schema evolution on a decimal column, hit during the merge phase of the upsert").
+ Name the catalog pattern and say "high confidence" if the signature is exact; otherwise say "family"
+ and mean it. Confidence on the family and confidence on the specifics are two different statements;
+ give both.
+2. **Rung** — "I am reading from the stacktrace alone."
+3. **What I need next** — the artifacts named in the matched row's **Evidence required** field, nothing
+ generic. Schema patterns: latest table schema and `.hoodie/hoodie.properties`. Concurrency patterns:
+ both writers' lock and concurrency-mode settings. Index/OOM patterns: `hoodie.index.type`, the
+ parallelism settings, the Spark event log. Table-service patterns: the health-checker JSON. Always add
+ table type and Hudi version if not already given. Offer to fetch anything that lives under the table
+ path yourself if the path is readable from here.
+
+Do not state a direction, a column, a side or a cause that the trace does not carry. A hunch is "this looks
+like X"; it is not "the table is the narrow side". What you do not claim at rung 0 you do not have to walk
+back at rung 1.
+
+Do not give the mitigation ladder at rung 0 unless the pattern is one where no further evidence can change
+it (the `not-a-failure` rows, HUDI-RCA-025, HUDI-RCA-035). Ladders depend on direction, config, and table
+state; a ladder given before those are known is a guess dressed as advice, and scenario testing showed it
+flips once the schema arrives.
+
+Once rung 1 evidence is in hand, fetch or ask for the other side of the comparison (incoming batch schema,
+second writer's config, health-checker JSON), then give the full report:
+
+1. **What failed** — innermost cause quoted, with the first Hudi frame.
+2. **Attribution** — pattern ID and category, or `unclassified` stated plainly.
+3. **Why** — the mechanism, naming the column / config / instant now that you can.
+4. **What to do** — the ladder in order, counter-intuitive steps flagged.
+5. **What would raise confidence further**, if anything still would.
+
+Lead with the attribution. Say nothing about your tooling, environment or unrelated failures.
diff --git a/hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md b/hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md
new file mode 100644
index 000000000000..87989f788c13
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/references/failure-catalog.md
@@ -0,0 +1,1726 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Hudi failure catalog
+
+38 failure patterns, each with the signature that identifies it, the mechanism behind it, and an
+ordered mitigation ladder. Used by the `hudi-rca` skill to classify a pasted failure; readable on its
+own as a troubleshooting reference.
+
+## How to read a row
+
+| Field | Meaning |
+|---|---|
+| **ID** | Stable identifier. Never renumber; retire instead. |
+| **Category** | `input/config` · `sizing/scale` · `environment` · `defect` · `not-a-failure` |
+| **Discoverability** | `self-evident` (the message says what to do) · `needs-expertise` (the message is generic, misleading, or points at the wrong layer) · `silent` (no exception at all) |
+| **Signature** | Exception classes, message regexes, and the Hudi frames that identify the pattern. |
+| **Symptom** | What the operator sees, in their terms. |
+| **Root cause** | The mechanism, in one or two sentences. |
+| **Evidence required** | Which rungs of the evidence ladder are needed to reach a confident call. |
+| **Disambiguation** | Present only where a signature maps to more than one pattern. The explicit discriminator. |
+| **Mitigation ladder** | Ordered, cheapest and most likely first. Counter-intuitive steps are marked and justified. |
+| **Governing configs** | Exact `hoodie.*` / `spark.*` keys. |
+| **`fixedInVersion`** | The Hudi release that fixes the defect, where one exists. |
+| **`preflightGuardExists`** | Whether a current Hudi release fails fast on the condition instead of corrupting silently. |
+| **`alsoAppliesWhenHealthy`** | `true` when the same advice is an optimization on a merely-slow job, not only a fix for a broken one. |
+| **Provenance** | Public `apache/hudi` issue numbers. |
+
+## Three rules that govern every row
+
+**1. Match the innermost `Caused by`, never the wrapper.** In the public tracker the wrapper classes
+dominate the counts — bare `HoodieException` 498 occurrences, `HoodieUpsertException` 189 — while the
+classes that actually identify the fault barely appear: `SchemaCompatabilityException` 1,
+`HoodieAvroSchemaException` 2, `HoodieKeyGeneratorException` 3. Users paste the wrapper. Every regex
+below is written against the innermost cause plus the first Hudi frame, because a classifier keyed on
+the top-level exception class will mostly see wrappers and learn nothing.
+
+**2. The stacktrace is the weakest signal, not the backbone.** For a large share of these the terminal
+exception is generic and the real cause is table state or config sizing several steps removed. A
+`413 Payload Too Large` on timeline init is really a forgotten savepoint blocking archival. A stalled
+ingestion is really a rollback that never finished and still holds the lock. Where a row says the
+evidence requires table state, the honest move is to run `HoodieTableHealthChecker` rather than parse
+the trace harder.
+
+**3. Ask the Hudi version early.** The heavy issue clusters are 0.9 through 0.15, and several
+high-traffic patterns here are fixed outright in 1.0 or 1.2. For this population "upgrade" is a
+genuinely good first-line remedy in a way it usually is not. Rows carry `fixedInVersion` where known.
+
+## Disambiguation index
+
+Four signatures map to more than one pattern. Resolve them here before attributing.
+
+| Signature | Candidates | Discriminator |
+|---|---|---|
+| `ConcurrentModificationException` / `HoodieWriteConflictException` | HUDI-RCA-006 healthy OCC retry **vs** HUDI-RCA-006 no lock provider | **Writer config, not the trace** — the stacktrace is identical. If `hoodie.write.lock.provider` is set and two writers are intended, this is a by-design rejection: retry the application attempt. If no lock provider is set and two writers exist, the table is being corrupted silently and the conflict exception is the lucky case. |
+| `HoodieRollbackException` | HUDI-RCA-027 rollback wedge · HUDI-RCA-026 metadata-table divergence · HUDI-RCA-025 dangling files | **The frame below `rollbackFailedWrites`.** `AsyncCleanerService.waitForCompletion` → cleaner/metadata-table path (026). A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first (006). `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → 025. |
+| `OutOfMemoryError` | HUDI-RCA-015 merge-side · HUDI-RCA-015 archival-side · HUDI-RCA-014 bloom index · HUDI-RCA-021 driver/timeline-server | **The Hudi frame.** `HoodieMergedLogRecordScanner` / `ExternalSpillableMap` → merge side (015a). `TimelineArchiverV1`/`TimelineArchiverV2.writeToFile` (pre-1.0: `HoodieTimelineArchiveLog`) / `HoodieAvroDataBlock.serializeRecords` → **archival side (015b), where adding executor memory is the wrong answer entirely** — the fix is to shrink commit met [...]
+| `FileNotFoundException` / `FileID ... does not exist` | **Run ladder L14 first** (metadata table enabled? read or write path? how recent?) · then HUDI-RCA-026 listing drift · cleaner-boundary read (L14 step 2) · HUDI-RCA-031 markers · HUDI-RCA-033 archiver race · HUDI-RCA-025 dangling file · HUDI-RCA-036 source file | **The path, after L14.** `MARKERS*` or `.hoodie/.temp/` → 031. A `.commit` / `.deltacommit` under `.hoodie/` → 033. A data or log file named in a compaction plan → 025. A [...]
+
+## Cross-cutting trap: MOR defers the failure
+
+On Merge-on-Read an incompatible schema write **succeeds**, and compaction fails hours later in a
+*different job* whose stacktrace names `HoodieCompactor` rather than the write. The table then cannot
+compact and every later write fails. Cause and symptom are separated by both time and process.
+
+**Rule: a compaction frame plus a schema exception means the bad write happened earlier, in another
+job. Investigate the writes from that window, not the job that just failed.** This applies to
+HUDI-RCA-001, HUDI-RCA-003 and HUDI-RCA-004.
+
+---
+
+# Group A — input/config
+
+The user asked Hudi to do something invalid, or configured it inconsistently. The remedy is to fix
+data or config; nothing is wrong with Hudi or the cluster.
+
+---
+
+## HUDI-RCA-001 — Schema evolution the writer rejects
+
+- **Category:** `input/config` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.avro\.AvroTypeException: Cannot encode decimal with precision (\d+) as max precision (\d+)`
+ - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found`
+ - Hudi frames: `HoodieAvroUtils.rewriteRecordWithNewSchema`, `HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, `HoodieMergeHelper.runMerge`, `BaseHoodieLogRecordReader.scanInternal` (pre-1.0: `AbstractHoodieLogRecordReader`), `HoodieCompactor.compact`
+- **Symptom:** A write that worked yesterday fails after an upstream column was retyped or a decimal
+ widened. On MOR the write succeeds and **compaction** fails hours later, permanently, for every
+ subsequent write.
+- **Root cause:** The incoming Avro schema is not a backward-compatible evolution of the table schema
+ under Hudi's rules. Full schema-on-read evolution is off by default, and decimal precision widening
+ was rejected outright until the `SchemaChangeUtils.isTypeUpdateAllow` relaxation.
+ The decimal signature has **two distinct mechanisms, and the trace does not say which**:
+ - *(a) Scale growth.* Table `decimal(13,2)`, batch `decimal(13,4)`. On the merge path each **existing**
+ value is rescaled to the batch's scale and then encoded into the batch's schema
+ (`HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType`, 0.14.1 line 984: `setScale` then `toFixed`).
+ Trailing zeros count toward precision, so a 12-digit stored value becomes 14 digits and overflows.
+ Deterministic from the code on every version; upgrading does not change the arithmetic.
+ - *(b) Wider batch at the same scale, values in range.* Table `decimal(13,2)`, batch `decimal(14,2)`.
+ This is what #11335 actually is: every reporter had **clustering** enabled, and the replace-commit
+ row writer produced files with inconsistent decimal types on 0.14/0.15. Reproduced by a maintainer,
+ clean on 1.0.0. Spark 3.4+ `round()` on a `decimal(13,2)` column returns `decimal(14,2)`, so an
+ upstream "rounding fix" produces exactly this shape.
+ **Do not name a direction until both the table schema and the batch schema are in hand.**
+- **Evidence required:** stacktrace · writer props · `hoodie.properties` (table schema) · for the MOR
+ variant, table state to show which file slices span the change.
+- **Disambiguation:** a compaction frame here means the offending write already committed — see the
+ MOR deferred-failure rule above. Distinguish from HUDI-RCA-003 by *direction*: a field the table has
+ and the batch lacks is 003; a field whose **type** changed is 001.
+- **Mitigation ladder:**
+ 1. Decimal, mechanism (b) (wider batch, same scale, clustering on): cast the column back to the
+ table's type as the immediate, lossless unblock; upgrade to 1.0.x as the fix. No config works
+ around it on 0.14/0.15. Ask whether clustering is enabled.
+ 1b. Decimal, mechanism (a) (scale grew): casting back re-rounds and discards the new digits, so this
+ is a decision — cast back to restore writes, or widen the table column deliberately as a planned
+ schema change. Upgrading alone does not help.
+ 2. Set `hoodie.datasource.write.reconcile.schema=true`, with `hoodie.schema.on.read.enable=true`.
+ 3. Stop dropping or retyping the column in the writer: keep it and write typed nulls.
+ 4. If MOR and already wedged, the table cannot compact until the pending compaction is unscheduled
+ or the affected file groups are rewritten. Do not hand-delete the pending instant files.
+- **Governing configs:** `hoodie.datasource.write.reconcile.schema`, `hoodie.schema.on.read.enable`,
+ `hoodie.avro.schema.validate`, `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`
+- **`fixedInVersion`:** 1.0.0 for decimal precision widening · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #8040, #5452, #11335, #8020, #2919
+
+---
+
+## HUDI-RCA-002 — Record key or precombine field missing, null, or wrong case
+
+- **Category:** `input/config` · **Discoverability:** `self-evident` for the key case, `needs-expertise` for precombine
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieKeyException: recordKey value: "null" for field: "([^"]+)" cannot be null or empty`
+ - `org\.apache\.hudi\.exception\.HoodieKeyException: Record key has to be non-null!`
+ - `org\.apache\.hudi\.exception\.HoodieException: \w+\(Part -\w+\) field not found in record\. Acceptable fields were :\[`
+ - Hudi frames: `HoodieAvroUtils.getNestedFieldVal`, `ComplexKeyGenerator`, often wrapped by
+ `SparkException: Failed to execute user defined function` so the top frame is Spark, not Hudi.
+- **Symptom:** The write fails on the first batch, or only when one composite key field is null.
+- **Root cause:** Three sub-cases sharing a remedy family: the configured field does not exist or is
+ null in some rows; the key config is **case-sensitive** and the user wrote `ID` for a column named
+ `id`; or HoodieStreamer defaulted `--source-ordering-field` to `ts`, producing a phantom `ts`
+ requirement that then persisted into `hoodie.properties`.
+- **Evidence required:** stacktrace · writer props · `hoodie.properties`
+- **Disambiguation:** if the named field is one the user never configured (classically `ts`), this is
+ the HoodieStreamer default, not a missing column. The message names a field the user has never heard
+ of, which is the whole confusion — say so explicitly rather than asking them to add a `ts` column.
+- **Mitigation ladder:**
+ 1. Check the exact field name **and case** against the dataframe schema. Hudi does not normalize case.
+ 2. Filter or default nulls in the key columns before writing.
+ 3. For HoodieStreamer with no ordering field: set `--source-ordering-field` to a real column, or use
+ `--op INSERT` / `bulk_insert`, which skip precombine entirely.
+ 4. Multi-field keys with `bulk_insert` had a genuine bug before 0.13.1 — upgrade, or switch to
+ `upsert` to confirm.
+- **Governing configs:** `hoodie.datasource.write.recordkey.field`,
+ `hoodie.datasource.write.precombine.field`, `hoodie.datasource.write.keygenerator.class`,
+ `hoodie.table.precombine.field`, `--source-ordering-field`
+- **`fixedInVersion`:** 0.13.1 for the bulk_insert multi-field key bug · **`preflightGuardExists`:** partial — the key check fires at write time, not at config time
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #3759, #9799, #11776, #10233
+
+---
+
+## HUDI-RCA-003 — Source drops a column the table still has
+
+- **Category:** `input/config` · **Discoverability:** `needs-expertise`
+- **Signature — verified per version on 2026-10-17 by replaying the same column drop (COW, upsert,
+ default writer configs) against each release:**
+
+ | Hudi | What the user sees | Innermost class | First Hudi frame |
+ |---|---|---|---|
+ | 1.0.1, 1.1.1, master | Fails on the **driver**, before any file is touched. Message **names the field** and prints both schemas. | `MissingSchemaFieldException: Schema validation failed due to missing field. Fields missing from incoming schema: {<record>.<field>}` | `AvroSchemaUtils.checkValidEvolution` (1.0.1 `:529`, 1.1.1 `:631`); on master `HoodieSchemaCompatibility.checkValidEvolution(:159)`; then `HoodieSchemaUtils$.deduceWriterSchemaWithoutReconcile` → `deduceWriterSchema` |
+ | 0.14.0, 0.14.1 | Fails on the driver too, but the message is **generic**: it does not name the field, does not print either schema, and has no `Caused by`. | `SchemaCompatibilityException: Incoming batch schema is not compatible with the table's one` | 0.14.1: `HoodieSchemaUtils$.deduceWriterSchema(:179)`; 0.14.0: `HoodieSparkSqlWriter$.deduceWriterSchema(:582)` |
+ | pre-0.14 | The write **proceeds** and fails later on the merge path while reading the *table's* Parquet (deferred; on MOR deferred to compaction). | `org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '<field>' not found` | `AvroRecordConverter.getAvroField` → `ParquetReaderIterator.hasNext` → `HoodieMergeHelper.runMerge`, wrapped by `HoodieUpsertException: Error upserting bucketType UPDATE for partition :\d+` |
+
+ Regexes, innermost cause:
+ - `org\.apache\.hudi\.exception\.MissingSchemaFieldException: Schema validation failed due to missing field\. Fields missing from incoming schema: \{([^}]+)\}`
+ - `org\.apache\.hudi\.exception\.SchemaCompatibilityException: Incoming batch schema is not compatible with the table's one`
+ - `org\.apache\.parquet\.io\.InvalidRecordException: Parquet/Avro schema mismatch: Avro field '([^']+)' not found`
+
+ `MissingSchemaFieldException extends SchemaCompatibilityException`; match the specific class first because
+ it is the one that names the field. **On 0.14.x the trace cannot tell you which column is missing** —
+ the same generic message also fires for a type change the writer rejects. **Confidence at rung 0 is
+ still high on the family**: this exact class + message + `deduceWriterSchema` frame on 0.14.x has one
+ meaning — the incoming batch's schema was rejected against the table's, i.e. a column was dropped or
+ retyped. Say that plainly and up front. What the schemas add is the column name and the kind of change,
+ not the attribution. **This is where the skill earns its place: it names the field the exception
+ withheld.** Do the diff yourself; do not hand the user two schemas and a suggestion.
+
+ **Schema diff procedure (shared with HUDI-RCA-001):**
+ 1. **Table schema.** If the table path is readable, get it yourself: the latest completed commit's
+ `extraMetadata.schema` (plain JSON in `.hoodie/*.commit` / `*.deltacommit` before 1.0; Avro-encoded
+ on 1.x, so decode it properly or load the table with Spark and read the schema). Otherwise ask for
+ `spark.read.format("hudi").load(path).printSchema()` or the commit's schema field. Not
+ `hoodie.properties` — `hoodie.table.create.schema` there is the schema at creation, not now.
+ 2. **Incoming schema.** Ask for `df.printSchema()` from the failing job, taken right before `save()`,
+ or the source's declared schema (Avro from the registry, JDBC DDL, the Kafka value schema).
+ 3. **Diff by field name, then by type**, meta fields excluded, and classify every difference:
+ - in table, not in batch → **dropped** → this row (003);
+ - same name, different type / precision / scale / nullability → **retyped** → HUDI-RCA-001,
+ note the direction and whether it is a decimal;
+ - in batch, not in table → **added** → ordinarily fine; schema evolution adds it. Only a problem if
+ `hoodie.avro.schema.validate` or `reconcile.schema` settings forbid it;
+ - **case differs only** → Hudi does not normalise case; treat as drop + add and say so.
+ 4. **Report per column**: name, table type, batch type, the classification, and the fix for that
+ column (typed null re-add, cast back, or accept the evolution). One line each. The user came in
+ with "incompatible"; they leave with "`segment` string was dropped from the feed; re-add it as
+ `cast(null as string)` or set `hoodie.write.set.null.for.missing.columns=true`".
+ 5. If more than one column differs, say which one tripped the guard is not knowable from the 0.14
+ message — the guard stops at the first incompatibility — and that all of them need fixing.
+- **Symptom:** Every sync fails after an upstream schema change. On the pre-0.14 variant the exception
+ arrives while reading the *table's* Parquet, so operators look at their table instead of their batch.
+ On 1.x it fails earlier and names the missing fields directly.
+- **Root cause:** The incoming batch no longer carries a column the table has. 0.14.0 introduced a
+ driver-side compatibility check in `deduceWriterSchema`, so the write fails before any data file is
+ touched, but with a generic message. 1.0 split that check so the missing-field case raises its own
+ exception naming the fields. Before 0.14 there was no driver-side check: the write proceeded and the
+ Parquet-to-Avro converter failed on the merge path instead.
+- **Evidence required:** stacktrace · writer props · last commit metadata (`extraMetadata.schema`) for
+ the field's declared type
+- **Disambiguation:** same exception class as HUDI-RCA-001's field-not-found variant. The
+ `HoodieMergeHelper.runMerge` frame and the fact that the field exists in the **table** but not the
+ **batch** is what makes it 003. Recover the field's declared type including its Avro `logicalType`
+ before re-adding it, or the re-add fails differently.
+- **Mitigation ladder:**
+ 1. Re-add the column to the incoming batch as a **typed** null via a transformer — `cast(null as
+ string)`, or `cast(null as date)` where the logical type demands it. Cheapest and reversible, and
+ it keeps the table's schema honest about what the column is.
+ 2. Or set `hoodie.write.set.null.for.missing.columns=true` (0.14.1+, default `false`) to have Hudi
+ fill the missing column with nulls itself. Less work than a transformer; the cost is that a
+ column silently going missing upstream no longer fails the write, so an accidental drop looks
+ like an intentional one.
+ 3. If the column is genuinely meant to go, set
+ `hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true` (default `false`) and let
+ the table schema follow the source. Dropping a column from the table is durable, so treat this as
+ a human decision rather than a mitigation.
+ 4. **Not** `hoodie.avro.schema.validate`. It gates **Step 2** of the same method, and on 1.x Step 1
+ has already thrown, so the write never reaches it — and it already defaults to `false`. It had an
+ effect on the pre-0.14 path only.
+ 5. **Config-layering trap:** if your config system overrides rather than merges list-valued settings,
+ adding a transformer silently drops the transformer that was already configured. Re-list all of
+ them, and verify the resolved config rather than the file you edited.
+- **Governing configs:** `hoodie.write.set.null.for.missing.columns`,
+ `hoodie.datasource.write.schema.allow.auto.evolution.column.drop`,
+ `hoodie.datasource.write.reconcile.schema`, `hoodie.avro.schema.validate` (pre-0.14 only)
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #8040
+
+---
+
+## HUDI-RCA-004 — Unvalidated source schema: records do not parse against the inferred type
+
+- **Category:** `input/config` · **Discoverability:** `self-evident` for the message, `needs-expertise` for the fix
+- **Signature** (innermost cause):
+ - `org\.apache\.spark\.sql\.catalyst\.util\.BadRecordException`
+ - `java\.lang\.RuntimeException: Failed to parse a value for data type struct<.*> \(current token: \w+\)`
+- **Symptom:** Records fail to parse and the whole job fails, rather than the bad rows being isolated.
+- **Root cause:** A semi-structured source (CDC, JSON, document-store feed) emits a field whose actual
+ shape differs from what was inferred from earlier records — most often an array where a struct was
+ inferred.
+- **Evidence required:** stacktrace · writer props (schema-provider config) · a raw source sample
+- **Mitigation ladder:**
+ 1. Set an explicit **target schema** instead of relying on inference, and enable validation with a
+ quarantine destination so bad records are isolated rather than failing the commit.
+ 2. For CDC sources, make sure the target schema declares the envelope fields the transformer adds
+ (operation type, transaction id, LSN, event timestamps) — otherwise validation rejects every record.
+ 3. Loosen validation only as a last resort. It converts a loud failure into a silent one.
+- **Governing configs:** `hoodie.streamer.schemaprovider.target.schema.file`,
+ `hoodie.streamer.schemaprovider.source.schema.file`, `hoodie.avro.schema.validate`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public — the mechanism is config hygiene and rarely filed
+
+---
+
+## HUDI-RCA-005 — Timestamp key generator cannot parse the partition field
+
+- **Category:** `input/config` · **Discoverability:** `self-evident`
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieKeyGeneratorException: Unable to parse input partition field: (.+)`
+ - `java\.lang\.IllegalArgumentException: Invalid format: "(.+)" is malformed at "(.+)"`
+ - Hudi frame: `TimestampBasedAvroKeyGenerator.getPartitionPath`
+- **Symptom:** Every write fails immediately, naming the exact offending value.
+- **Root cause:** The configured **input** date format does not match the actual partition-field format.
+ Typical triggers: microsecond precision, a trailing `Z`, or a switch from epoch to ISO-8601 upstream.
+- **Evidence required:** stacktrace alone suffices · writer props to confirm the configured format
+- **Mitigation ladder:**
+ 1. Update the key generator's **input** format config to match the value printed in the exception.
+ 2. Verify the **output** partition format is unchanged. Changing the output format alters partition
+ paths and effectively forks the table — a far worse outcome than the parse failure.
+ 3. If the source format is now inconsistent across records, normalize it in a transformer before the
+ key generator sees it.
+- **Governing configs:** `hoodie.keygen.timebased.input.dateformat`,
+ `hoodie.keygen.timebased.output.dateformat`, `hoodie.keygen.timebased.timestamp.type`,
+ `hoodie.keygen.timebased.input.timezone`, `hoodie.keygen.timebased.output.timezone`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public with this exact signature; included because it is trivially fixable and
+ almost certainly under-reported
+
+---
+
+## HUDI-RCA-006 — Multi-writer without, or with a misconfigured, lock provider
+
+- **Category:** `input/config` · **Discoverability:** `needs-expertise`, and **`silent`** in the no-lock-at-all case
+- **Signature** (innermost cause):
+ - `java\.util\.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`,
+ wrapped in `HoodieWriteConflictException`; frames
+ `SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict`,
+ `TransactionUtils.resolveWriteConflictIfAny`, `SparkRDDWriteClient.preCommit`
+ - `org\.apache\.hudi\.exception\.HoodieLockException: Unable to acquire lock, lock object (null|LockResponse\(lockid:\d+, state:WAITING\))`;
+ frames `LockManager.lock`, `TransactionManager.beginTransaction`
+ - `java\.lang\.IllegalArgumentException: ALREADY_ACQUIRED` at `HiveMetastoreBasedLockProvider.acquireLock`
+ - `org\.apache\.hadoop\.fs\.FileAlreadyExistsException: File already exists: .*\.hoodie/\d+\.commit\.requested`
+ at `HoodieActiveTimeline.createImmutableFileInPath`
+- **Symptom:** Intermittent failures that scale with writer count. With **no** lock provider there may
+ be no exception at all — just overwritten timeline state and lost data.
+- **Root cause:** Two writers against one table without OCC properly configured, or OCC configured and
+ genuinely conflicting. The semantics users most often get wrong: **the OCC lock is table-level, not
+ partition-level**, and it serializes only the commit-validation phase. Writers on disjoint partitions
+ still serialize on the lock.
+- **Evidence required:** stacktrace · **writer props — mandatory** · `hoodie.properties`
+- **Disambiguation (the most important row in this catalog):** the conflict exception and the no-lock
+ corruption produce the **same stacktrace**. Only the writer config splits them.
+ - `hoodie.write.lock.provider` set, concurrency intended → by-design rejection of the later committer.
+ Expected. Add orchestrator-level retry of the whole application attempt. Severity: routine.
+ - No lock provider set, but two writers demonstrably exist → the table is being corrupted and this
+ exception is the *lucky* case where you found out. Severity: urgent.
+ - Treat "is a second writer intended?" as a question to ask the user, not something to infer.
+- **Mitigation ladder:**
+ 1. If two writers exist and no lock provider is set, set one **together with
+ `hoodie.write.concurrency.mode`**. The provider alone is not enough, and can be silently undone:
+ the mode defaults to `SINGLE_WRITER`, and `HoodieWriteConfig.Builder.autoAdjustConfigsForConcurrencyMode`
+ (`HoodieWriteConfig.java:3938-3948`) **replaces an explicitly configured provider with
+ `InProcessLockProvider`** when the mode is single-writer and no table service is async, logging
+ only a WARN. The job then succeeds with a non-distributed lock, which is the corruption case
+ wearing a green tick. Set `hoodie.clean.failed.writes.policy=LAZY` explicitly in the same change:
+ `validate()` (`:3987-3993`) throws when a multi-writer mode is paired with the default `EAGER`,
+ and relying on the builder to infer it leaves the dependency invisible to the next reader.
+ Non-negotiable; absence is silent
+ corruption, not a performance choice.
+ 2. **Lower** `hoodie.write.lock.wait_time_ms` to around 5000. Counter-intuitive: a long wait converts
+ a fast, retryable rejection into a long stall that looks like a hang, and the later writer has to
+ retry regardless.
+ 3. **Which provider to recommend.** Cross-process providers that are safe to recommend:
+ `org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider` (keys
+ `hoodie.write.lock.zookeeper.url`, `.port`, `.lock_key`, `.base_path`),
+ `org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider` on AWS, and **on Hudi 1.2.0 or
+ later only** the storage-native `org.apache.hudi.client.transaction.lock.StorageBasedLockProvider`,
+ which needs no extra service because it uses the object store's conditional writes. Do not offer
+ the storage-based provider to a user below 1.2.0; confirm the version before naming it.
+ **Do not recommend `HiveMetastoreBasedLockProvider`.** Its `ALREADY_ACQUIRED` failure is one of
+ this row's own signatures, it leaves stale locks in the metastore that need manual cleanup, and it
+ puts lock traffic on a service that is usually already the bottleneck. If a user is already on it
+ and hitting lock problems, moving off it is part of the fix, not a side note.
+ For DynamoDB locks, set the endpoint URL explicitly and use on-demand billing; stale lock rows
+ occasionally need manual removal.
+ 4. Add retry of the whole application attempt at the orchestrator. The later committer is rejected by
+ design and must retry.
+ 5. Where lock serialization is itself the bottleneck on MOR, consider 1.x non-blocking concurrency
+ control, which pairs best with a bucket index.
+- **Governing configs:** `hoodie.write.concurrency.mode`, `hoodie.write.lock.provider`,
+ `hoodie.write.lock.wait_time_ms`, `hoodie.write.lock.num_retries`, `hoodie.write.lock.dynamodb.*`,
+ `hoodie.write.lock.zookeeper.*`, `hoodie.clean.failed.writes.policy`
+- **`fixedInVersion`:** n/a — NBCC added in 1.x as an alternative mode, not a fix
+- **`preflightGuardExists`:** no — Hudi does not detect "two writers, no lock" at write time
+- **`alsoAppliesWhenHealthy`:** true — lock wait tuning is an optimization on a merely-slow multi-writer setup
+- **Provenance:** #7653, #3731, #4456, #6226, #9213
+
+---
+
+## HUDI-RCA-007 — Hive or Glue catalog sync failure: table written but invisible
+
+- **Category:** `input/config` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.RuntimeException: Unable to instantiate org\.apache\.hadoop\.hive\.ql\.metadata\.SessionHiveMetaStoreClient`;
+ frames `HMSDDLExecutor.<init>`, `HiveSyncTool.initClient`
+ - `NoSuchObjectException: .* table not found`
+ - Wrapped in `HoodieHiveSyncException: Got runtime exception when hive syncing` or
+ `Could not sync using the meta sync class`
+ - Log giveaway: `Trying to connect to metastore with URI thrift://localhost:9083` when the user
+ believes they configured Glue
+- **Symptom:** Data lands on storage correctly but the query engine cannot see it, or the write fails
+ only in the post-commit sync step.
+- **Root cause:** Sync mode and catalog wiring mismatch, Hive metastore version incompatibility, or
+ table-name case sensitivity. The `localhost:9083` line is the tell that the intended catalog config
+ never took effect.
+- **Evidence required:** stacktrace · writer props · driver log (for the metastore URI line)
+- **Mitigation ladder:**
+ 1. Lowercase the table and database names everywhere. An uppercase table name causes a rename failure.
+ 2. Set `hoodie.datasource.hive_sync.mode` explicitly — `hms` for Glue and HMS.
+ 3. For Glue, set the AWS Glue sync tool class explicitly rather than relying on inference.
+ 4. Verify the Hive metastore version is 2.3 or newer for Spark 3 bundles.
+ 5. Confirm `hoodie.datasource.hive_sync.partition_extractor_class` matches the actual partitioning —
+ the non-partitioned extractor for unpartitioned tables, the multi-part extractor for multi-level.
+- **Governing configs:** `hoodie.datasource.hive_sync.enable`, `hoodie.datasource.hive_sync.mode`,
+ `hoodie.datasource.hive_sync.database`, `hoodie.datasource.hive_sync.table`,
+ `hoodie.datasource.hive_sync.partition_extractor_class`, `hoodie.datasource.hive_sync.partition_fields`,
+ `hoodie.meta.sync.client.tool.class`, `hoodie.datasource.write.hive_style_partitioning`
+- **`fixedInVersion`:** 0.11.1 for the Glue client-factory regression · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #5484, #8368, #2409, #954
+
+---
+
+## HUDI-RCA-008 — Config changed between runs against an existing table
+
+- **Category:** `input/config` · **Discoverability:** `self-evident` when the guard fires, **`silent`** when it does not
+- **Signature:**
+ - `HoodieException: Config conflict \(key, current value, existing value\): ` — when Hudi's guard
+ catches it
+ - When the guard does not catch it: **no exception**. Duplicates, no-op deletes, or an unqueryable
+ table.
+- **Symptom:** A pipeline that ran for months starts failing, or silently producing duplicates, after
+ someone edited the writer config or upgraded Hudi.
+- **Root cause:** Several table-level properties are written into `hoodie.properties` on first write and
+ are effectively **immutable**: record key field, partition path field, key generator class, table
+ type, precombine field, hive-style partitioning. Changing them later either trips the conflict guard
+ or silently reinterprets existing records.
+- **Evidence required:** `hoodie.properties` · writer props. **This pattern is definitionally a diff
+ between the two** — no stacktrace substitutes for it, and this is why the skill requests
+ `hoodie.properties` by default.
+- **Disambiguation:** overlaps HUDI-RCA-013 (silent duplicates). If the properties diff shows a changed
+ key generator or record-key field, the duplicates are 008's mechanism and 013's symptom. Report 008 —
+ it names the cause.
+- **Mitigation ladder:**
+ 1. Read `.hoodie/hoodie.properties` and diff it against the current writer config. This is step one of
+ nearly any Hudi triage, not just this pattern.
+ 2. Revert the changed property to match the table.
+ 3. If the change is genuinely required, rewrite the table into a new path with `bulk_insert` rather
+ than fighting the guard. Editing `hoodie.properties` by hand is not a migration.
+ 4. For upgrade-induced key-encoding changes, pin the old version or rewrite. A single-field record key
+ under `ComplexKeyGenerator` was encoded differently from 0.14.1 onward, which silently created
+ duplicates and silently no-op'd deletes.
+- **Governing configs:** `hoodie.table.recordkey.fields`, `hoodie.table.partition.fields`,
+ `hoodie.table.keygenerator.class`, `hoodie.table.type`, `hoodie.table.precombine.field`,
+ `hoodie.table.version`, `hoodie.datasource.write.hive_style_partitioning`
+- **`fixedInVersion`:** no — the 0.14.1 key-encoding change shipped without a migration path
+- **`preflightGuardExists`:** partial — the conflict guard covers some keys, not all
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #10508, #10587, #10233
+
+---
+
+## HUDI-RCA-009 — Wrong query type or engine wiring makes data look missing or stale
+
+- **Category:** `input/config` · **Discoverability:** **`silent`**
+- **Signature:** **No exception.** Recognised by symptom, not by trace: an `INSERT OVERWRITE` wrote
+ files but `SELECT` returns old data; only a `_ro` table exists where the user expected `_rt`;
+ timestamps render as implausible far-future dates in the query engine.
+- **Symptom:** Data is demonstrably on storage but the query returns stale, empty, or garbled results.
+- **Root cause:** Query-side wiring, not write-side. Reading the read-optimized (`_ro`) view of a MOR
+ table and seeing pre-compaction state; missing `spark.sql.hive.convertMetastoreParquet=false`, so
+ Spark uses its own Parquet reader and bypasses Hudi's file-slice resolution; expecting two Hive tables
+ for a Copy-on-Write table, which only ever has one; or a timestamp type the catalog declares
+ differently from how it was written.
+- **Evidence required:** `hoodie.properties` (table type) · the query and the engine · writer props
+- **Proactive detection:** compare `COUNT(*)` on the snapshot view against the `_ro` view. A gap that
+ closes after compaction is this pattern, by design, not data loss.
+- **Mitigation ladder:**
+ 1. Set `spark.sql.hive.convertMetastoreParquet=false`. Note that whitespace around `=` in a SQL `set`
+ has been enough to make this silently not apply — verify it took effect.
+ 2. Query the snapshot view, not `_ro`; set `hoodie.datasource.query.type=snapshot`.
+ 3. For MOR, confirm compaction has actually run if you need read-optimized freshness.
+ 4. For timestamp rendering, enable `hoodie.datasource.hive_sync.support_timestamp`.
+- **Governing configs:** `hoodie.datasource.query.type`,
+ `hoodie.datasource.hive_sync.support_timestamp`, `spark.sql.hive.convertMetastoreParquet`,
+ `hoodie.datasource.write.drop.partition.columns`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #4154, #2409, #2123, #2338
+
+---
+
+## HUDI-RCA-010 — HoodieStreamer source or schema-provider misconfiguration
+
+- **Category:** `input/config` · **Discoverability:** `needs-expertise`
+- **Signature:**
+ - `org\.apache\.hudi\.exception\.HoodieException: Commit \d+ failed and rolled-back !` at
+ `DeltaSync.writeToSink` / `StreamSync.writeToSink` — **this message carries no diagnostic content
+ whatsoever**
+ - `java\.lang\.NullPointerException` inside `AvroDeserializer` with no Hudi frame
+ - Structurally wrong data: a whole CDC envelope ingested instead of the unwrapped row
+- **Symptom:** The Streamer fails with a generic "commit failed and rolled-back", or ingests
+ structurally wrong data.
+- **Root cause:** Schema provider mismatched to source. A provider returning a null target schema while
+ a transformer is configured; a generic Avro source used where a CDC-specific source and payload class
+ are required to unwrap the envelope; Confluent wire-format Avro needing registry-aware deserialization.
+- **Evidence required:** stacktrace · **Streamer CLI args** · driver log — the real exception is in the
+ driver log *below* the "failed and rolled-back" line, not in the thrown one
+- **Disambiguation:** `Commit ... failed and rolled-back` on its own is a **content-free wrapper**. Do
+ not attribute from it. Ask for the driver log and classify from the exception beneath it. If the user
+ can only supply this line, say so and request the driver log — that is the correct outcome, not a guess.
+- **Mitigation ladder:**
+ 1. Read the driver log below the "failed and rolled-back" line and classify the real exception.
+ 2. For CDC sources, use the database-specific source class with its matching payload class, not a
+ generic Avro source.
+ 3. For Confluent Avro, let the Kafka Avro source fetch from the registry rather than hand-rolling a
+ schema provider.
+ 4. If using a transformer, do not return null from the target schema provider. Set a real schema, or
+ omit the provider so Hudi infers from the dataframe.
+- **Governing configs:** `--source-class`, `--schemaprovider-class`, `--payload-class`,
+ `--transformer-class`, `--source-ordering-field`, `hoodie.streamer.schemaprovider.registry.url`,
+ `hoodie.streamer.source.kafka.topic`
+- **`fixedInVersion`:** no · **`preflightGuardExists`:** no — and the uninformative message is itself
+ worth an upstream report
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #2515, #2149, #2589, #8519, #10233, #5540
+
+---
+
+## HUDI-RCA-011 — Source initialization fails: credentials or schema registry
+
+- **Category:** `input/config` · **Discoverability:** `self-evident` for credentials, `needs-expertise` for the registry case
+- **Signature:** Failures during writer startup, **before any batch is read**:
+ - `org\.apache\.kafka\.common\.errors\.\w+Exception`, SSL or keystore load failures,
+ connection-refused against brokers
+ - `org\.apache\.hudi\.utilities\.exception\.HoodieSchemaProviderException`
+- **Symptom:** The job never ingests anything. No data lands, so there is no partial state to reason
+ about — which is itself the diagnostic.
+- **Root cause:** Source credentials or TLS material are wrong or unreadable, or the source requires a
+ schema registry that was never configured. A common origin: a stream configuration **cloned** from an
+ older one created before registry support existed, carrying an empty registry reference that only
+ fails at writer init.
+- **Evidence required:** stacktrace · writer props **as resolved, not as written** · the age of the
+ failure signal
+- **Disambiguation:** distinguish an **init** failure from a **sync** failure. An init-failure counter
+ that never decrements can reflect a long-past startup failure on a driver that has since recovered.
+ Confirm the failure is recent before investigating it.
+- **Mitigation ladder:**
+ 1. Fix the credential or registry configuration and restart.
+ 2. **Do not create a stream by cloning another** when the source type has gained required fields since
+ the original was created. Recreate from scratch — the clone silently carries empty required fields.
+ 3. Validate required source fields at creation time rather than discovering them at writer init.
+- **Governing configs:** `hoodie.streamer.schemaprovider.registry.url`,
+ `hoodie.streamer.source.kafka.topic`, Kafka client security properties passed through
+ `hoodie.streamer.kafka.*`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public — onboarding-time failures are rarely filed
+
+---
+
+## HUDI-RCA-012 — Checkpoint ahead of the source: offsets aged out or topic recreated
+
+- **Category:** `input/config` (persistent case) / `environment` (transient case) · **Discoverability:** **`silent`** in its most common form
+- **Signature:**
+ - `org\.apache\.kafka\.clients\.consumer\.OffsetOutOfRangeException: Fetch position \S+ is out of range for partition`
+ - `Some data may have been lost because they are not available in Kafka any more`
+ - Or **no exception at all** — a healthy-looking job ingesting nothing
+- **Symptom:** The stream looks perfectly healthy and ingests nothing.
+- **Root cause — two causes with opposite correct responses:**
+ - **Persistent:** the topic was deleted and recreated, or retention aged past the committed
+ checkpoint. The checkpoint is permanently beyond the high watermark and will never recover.
+ - **Transient:** the broker's latest offset briefly moved backward (unclean leader election, replica
+ truncation, rebalance). Self-heals on the next produce.
+- **Evidence required:** writer checkpoint from commit metadata · source topic offsets and metadata ·
+ stacktrace where present
+- **Disambiguation:** compare the committed checkpoint against current end offsets, then **watch the
+ drift for one interval**. Transient regressions close within minutes; persistent ones do not move at
+ all. A topic creation timestamp later than your last successful commit means recreation. Retention
+ settings alone do not delete a topic — if the topic is gone, something deleted it.
+- **Mitigation ladder:**
+ 1. **Transient: do nothing.** Resetting the checkpoint here loses data for no reason. This is the
+ counter-intuitive step and the most common mistake on this pattern.
+ 2. **Persistent:** reset the checkpoint to the earliest available offset or a deliberately chosen
+ position, accepting that the gap is unrecoverable. Write down what was skipped.
+ 3. Confirm with the source owner before resetting anything.
+ 4. Prevent: set retention comfortably above your worst-case ingestion outage, and never delete a topic
+ a live stream is checkpointed against.
+- **Governing configs:** `hoodie.streamer.source.kafka.topic`,
+ `auto.offset.reset` (the native Kafka consumer property, passed through — not a `hoodie.*` key),
+ `hoodie.streamer.checkpoint.provider.path`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public with this signature
+
+---
+
+## HUDI-RCA-013 — Wrong operation or index choice produces silent duplicates
+
+- **Category:** `input/config` · **Discoverability:** **`silent`** — the highest-value row in the catalog
+- **Signature:** **No exception.** Found only by running
+ `SELECT key, COUNT(*) FROM t GROUP BY key HAVING COUNT(*) > 1`, or noticed as unexplained table growth.
+- **Symptom:** A primary key appears more than once. Often accompanied by table size growing far faster
+ than ingested volume.
+- **Root cause:** A ladder of distinct mechanisms the user cannot tell apart without help:
+ `hoodie.datasource.write.operation` set to `insert` or `bulk_insert`, which **do not deduplicate
+ against existing data by design**; a non-global index with a record that moved partitions, so the key
+ legitimately exists twice; a global index on MOR where read-optimized queries show duplicates until
+ compaction runs; querying through Hive without `convertMetastoreParquet=false`; or a key-generator
+ encoding change across versions.
+- **Evidence required:** `hoodie.properties` · writer props · the dedup query result · table type
+- **Proactive detection:** `SELECT COUNT(1), COUNT(DISTINCT <record_key>)` on the **snapshot** view.
+ Run it after any index or key-generator change.
+- **Disambiguation:** before calling this a defect, establish which mechanism applies. Cross-partition
+ duplicates with a non-global index are **working as configured**. Duplicates on a MOR `_ro` view that
+ disappear after compaction are **transient by design**. Only duplicates on the snapshot view with
+ `upsert` and a correct index are a real fault — and then check HUDI-RCA-008 for a key-encoding change.
+- **Mitigation ladder:**
+ 1. Confirm `hoodie.datasource.write.operation=upsert`. This alone explains most reports.
+ 2. Determine whether the duplicates are cross-partition. If so, move to a global index and set
+ `hoodie.bloom.index.update.partition.path=true`.
+ 3. Re-run the dedup check against the **snapshot** view, not `_ro`, with
+ `convertMetastoreParquet=false`.
+ 4. If MOR, force a compaction and re-check — read-optimized duplicates may be transient by design.
+ 5. If the table crossed 0.14.1 with a single-field record key under `ComplexKeyGenerator`, compare
+ `_hoodie_record_key` encoding before and after: this is the known encoding regression, and the
+ remedy is to pin the old version or rewrite the table.
+- **Governing configs:** `hoodie.datasource.write.operation`, `hoodie.index.type`,
+ `hoodie.bloom.index.update.partition.path`, `hoodie.datasource.write.keygenerator.class`,
+ `hoodie.combine.before.insert`, `hoodie.datasource.query.type`
+- **`fixedInVersion`:** no · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #2338, #2255, #5777, #10508
+
+---
+
+# Group B — sizing/scale
+
+The config was right at 1,000 file groups and wrong at 500,000. Nobody's bug: the remedy is to re-tune
+and add a guardrail. This is the category where a generic LLM most often gives the exactly wrong
+advice, because several ladders here run *downward*.
+
+---
+
+## HUDI-RCA-014 — Bloom index OOM from low parallelism or key skew
+
+- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.OutOfMemoryError: Java heap space`
+ - Frames are Kryo and shuffle internals with **nothing naming Hudi's index**:
+ `com\.esotericsoftware\.kryo\.io\.Input\.readString`, `Tuple2Serializer.read`,
+ `KryoDeserializationStream.readObject`, `ExternalSorter.insertAll`, `BlockStoreShuffleReader.read`
+- **Symptom:** A heap OOM in a shuffle-read stage during index lookup. Nothing in the trace says "index".
+- **Root cause:** The index tagging stage either has too few partitions for the key volume, or the keys
+ are skewed so a few tasks receive disproportionate work.
+- **Evidence required:** stacktrace · writer props · **Spark event log** for stage task counts and the
+ p75-versus-max shuffle-read spread
+- **Disambiguation:** distinguish from HUDI-RCA-015 by the frames. Kryo and `ExternalSorter` under a
+ shuffle read with no Hudi frame is the index stage (014); `HoodieMergedLogRecordScanner` or
+ `ExternalSpillableMap` is the merge side (015). Diagnose skew versus low parallelism before acting: a
+ wide gap between p75 and max task input is skew; a small task count with uniformly large tasks is low
+ parallelism. They have different fixes.
+- **Mitigation ladder:**
+ 0. **Confirm the index type first.** This signature names no Hudi class, so it is matched on shape
+ alone and says nothing about which index produced it. `hoodie.index.type` has **no declared
+ default**: the effective one is set per engine in `HoodieIndexConfig.Builder.build()` — `SIMPLE`
+ on Spark, `INMEMORY` on Flink, `SIMPLE` on Java. On a default Spark table every bloom key below
+ is read by nothing, so applying them changes the job's configuration and not its behaviour.
+ Branch here.
+
+ **If BLOOM or GLOBAL_BLOOM:**
+ 1. **Low parallelism:** raise input partitioning at the source first, since the index stage inherits
+ it. For Kafka that is `hoodie.streamer.source.kafka.minPartitions`. Only then set
+ `hoodie.bloom.index.parallelism` explicitly.
+ 2. **Lower `hoodie.bloom.index.keys.per.bucket`.** Counter-intuitive — you are shrinking the per-task
+ batch to produce more, smaller units of work. Raising it makes the OOM worse.
+ 3. **Skew with random keys:** set `hoodie.bloom.index.fileid.key.sorting.enable=true` **and**
+ `hoodie.bloom.index.bucketized.checking=false`. Both are required: bucketized checking stays on
+ unless explicitly disabled, so setting only the first changes nothing.
+ 4. If the OOM frame is `ExternalSorter`, lower
+ `spark.shuffle.spill.numElementsForceSpillThreshold` (e.g. to 100000) to force earlier spilling.
+ 5. Only then change index type or add executor memory.
+
+ **If SIMPLE or GLOBAL_SIMPLE** — fewer levers, and that asymmetry is itself the finding. A simple
+ index joins the incoming keys against the base files of the affected partitions; there is no
+ bucketing to retune, so the ladder runs out quickly:
+ 1. Raise input partitioning at the source, as above. Then set `hoodie.simple.index.parallelism`
+ (or `hoodie.global.simple.index.parallelism` for `GLOBAL_SIMPLE`) — both default to `0`,
+ meaning the parallelism is deduced rather than fixed.
+ 2. If the OOM is on the *input* side rather than the join, `hoodie.simple.index.use.caching`
+ (default `true`) and `hoodie.simple.index.input.storage.level` control how the incoming
+ records are cached; moving to a disk-backed level relieves heap at the cost of IO.
+ 3. If the OOM frame is `ExternalSorter`, step 4 above applies here too.
+ 4. Past that, a simple index has little headroom left on key volume. Changing index type is the
+ real answer rather than a last resort: a record-level index or a bucket index bounds the lookup
+ instead of scanning base files. Treat that as a planned migration, not a mitigation — it
+ rewrites the metadata table.
+
+ **If INMEMORY, BUCKET or FLINK_STATE:** this pattern does not apply. An OOM on those paths is not
+ an index-lookup sizing problem, and the bloom and simple keys are read by nothing.
+- **Governing configs:** `hoodie.index.type` (**check first** — it selects which of the rest apply),
+ `hoodie.bloom.index.parallelism`, `hoodie.bloom.index.keys.per.bucket`,
+ `hoodie.bloom.index.fileid.key.sorting.enable`, `hoodie.bloom.index.bucketized.checking`,
+ `hoodie.simple.index.parallelism`, `hoodie.global.simple.index.parallelism`,
+ `hoodie.simple.index.use.caching`, `hoodie.simple.index.input.storage.level`,
+ `spark.shuffle.spill.numElementsForceSpillThreshold`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — the same parallelism and skew analysis is the first move on an
+ index stage that merely runs slowly
+- **Provenance:** #1491, #11960, #12116
+
+---
+
+## HUDI-RCA-015 — Executor OOM during upsert or bulk_insert at scale
+
+- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.OutOfMemoryError: (Java heap space|GC overhead limit exceeded|Requested array size exceeds VM limit)`
+ - `ExecutorLostFailure`, `Container killed by YARN`, `exit code 52`
+ - **(a) merge side** — Hudi frames `ExternalSpillableMap`, `HoodieMergedLogRecordScanner`,
+ `BoundedInMemoryExecutor`
+ - **(b) archival side** — frames `ByteArrayOutputStream.hugeCapacity` →
+ `HoodieAvroDataBlock.serializeRecords` → `TimelineArchiverV1`/`TimelineArchiverV2.writeToFile` →
+ `archiveIfRequired` (pre-1.0 these frames read `HoodieTimelineArchiveLog`); or
+ `java\.lang\.NullPointerException: null of string of map of union in field extraMetadata of org\.apache\.hudi\.avro\.model\.HoodieCommitMetadata`
+ at the same frame
+- **Symptom:** The job runs fine at small volume, then OOMs persistently as the key space or log-file
+ volume grows — often **independent of input batch size**, which is the tell that it is *state* that
+ grew, not input. In the archival variant the frames are **in archival, nowhere near the user's
+ upsert**, which operators reasonably but wrongly read as an upsert sizing problem.
+- **Root cause:** **(a)** The in-memory map used for MOR log merge exceeds what the executor heap allows,
+ because merge spill thresholds are sized wrongly relative to heap, or because genuine key-space scale
+ has outgrown the index choice. **(b)** Commit metadata files grew enormous — most often because cleaner
+ retention was set so aggressively that each commit records a huge number of file operations — and
+ archival then loads them in batches and exhausts the heap.
+- **Evidence required:** stacktrace · writer props · **Spark event log** · executor logs
+- **Disambiguation:** before tuning anything, classify the OOM — see ladder **L2** in
+ `mitigation-ladders.md`. Exit 137 with a healthy Java heap is native memory, not heap. A
+ `FetchFailedException` is **not memory at all**; it is ephemeral local disk, and it does not show up
+ in the Spark UI. Reacting to a FetchFailed by adding heap wastes a tuning cycle. Then split (a) from
+ (b) on the Hudi frame: a `TimelineArchiverV1`/`TimelineArchiverV2` frame — `HoodieTimelineArchiveLog`
+ before 1.0 — means the archival variant, where **adding
+ executor memory does not help at all** and the fix is to shrink commit metadata.
+- **Mitigation ladder:**
+ 1. Classify the OOM first (L2), then split merge-side from archival-side on the frame. The remedies do
+ not overlap.
+ 2. **(a) merge side:** raise `hoodie.memory.merge.max.size` and `spark.executor.memoryOverhead` —
+ cheapest, and correct for a genuine heap-side merge OOM.
+ 3. Raise `hoodie.upsert.shuffle.parallelism` / `hoodie.bulkinsert.shuffle.parallelism`. **Note these
+ default to `0`, not 200** — `0` means "inherit the input RDD's partition count". Any advice built
+ on "it is 200 unless you change it" is wrong on every current release.
+ 4. **Lower `spark.memory.fraction` and `spark.memory.storageFraction`** if the symptom is GC pressure
+ rather than hard OOM. Counter-intuitive: giving Spark's managed regions *less* leaves more heap for
+ user objects.
+ 5. **(b) archival side:** lower `hoodie.commits.archival.batch` so fewer commit files are deserialized
+ at once, then fix whatever made commit metadata huge — usually an over-aggressive
+ `hoodie.cleaner.commits.retained`. Counter-intuitive: retaining *more* commits can produce *smaller*
+ per-commit metadata, because each clean then touches fewer files. Driver heap buys time, not a fix.
+ 6. At genuine scale, switch index to record-level or bucket index, enable the metadata table, and
+ partition the table.
+- **Governing configs:** `hoodie.memory.merge.max.size`, `hoodie.memory.compaction.max.size`,
+ `hoodie.upsert.shuffle.parallelism`, `hoodie.bulkinsert.shuffle.parallelism`, `hoodie.index.type`,
+ `hoodie.metadata.enable`, `hoodie.commits.archival.batch`, `hoodie.cleaner.commits.retained`,
+ `spark.executor.memoryOverhead`, `spark.memory.fraction`, `spark.memory.storageFraction`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — the same parallelism and spill analysis applies to a slow but
+ succeeding upsert
+- **Provenance:** #1491, #11960, #12116, #8332, #2408, #2515
+
+---
+
+## HUDI-RCA-016 — Write-profiling shuffle produces an unfetchable broadcast
+
+- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.spark\.shuffle\.MetadataFetchFailedException: Unable to deserialize broadcasted map statuses for shuffle`
+ - `\[INTERNAL_ERROR_BROADCAST\] Failed to get broadcast_\d+_piece\d+ of broadcast_\d+`
+ - Hudi frame: `BaseSparkCommitActionExecutor.buildProfile`, failing in a `countByKey` stage; wrapped
+ in `HoodieUpsertException: Failed to upsert`
+- **Symptom:** A Spark broadcast-fetch error with nothing naming Hudi. The instinctive fix — more
+ parallelism — makes it worse.
+- **Root cause:** Write profiling and simple-index tagging fan out with **file-group count**. The
+ shuffle's map-status broadcast grows with the number of shuffle partitions until Spark cannot
+ reliably fetch and deserialize its pieces.
+- **Evidence required:** stacktrace · writer props · file-group count (`HoodieTableLayoutAnalyzer`) ·
+ Spark event log
+- **Mitigation ladder:**
+ 1. **Reduce `hoodie.simple.index.parallelism`.** Counter-intuitive and the whole point of this row:
+ fewer shuffle partitions means a smaller map-status structure to broadcast. **Raising parallelism
+ makes this failure worse**, which is exactly what an operator reaching for the obvious lever will do.
+ 2. Address file-group growth itself — tune clustering and compaction so file groups consolidate
+ rather than accumulate without bound.
+ 3. Consider an index whose tagging cost does not scale with file-group count (record-level or bucket).
+- **Governing configs:** `hoodie.simple.index.parallelism`, `hoodie.index.type`,
+ `hoodie.clustering.inline`, `hoodie.compact.inline`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — file-group consolidation is an optimization long before it is a fix
+- **Provenance:** none public with this exact broadcast signature
+
+---
+
+## HUDI-RCA-017 — Record-size estimate blowup on wide payloads
+
+- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.IllegalArgumentException: You cannot call toBytes\(\) more than once without calling reset\(\)`
+ - Frames: `RunLengthBitPackingHybridEncoder.toBytes` → `ColumnWriteStoreBase.sizeCheck` →
+ `HoodieBaseParquetWriter.write` → `HoodieCreateHandle.doWrite`
+ - Often preceded by `ERROR HoodieCreateHandle : Error writing record HoodieRecord{key=...}`
+- **Symptom:** A Parquet encoder assertion with nothing suggesting that the records are too wide.
+- **Root cause:** Hudi's dynamic record-size estimation drives Parquet row-group sizing. Very wide
+ values (large string or JSON columns) break the estimate badly enough that the encoder is asked to
+ finalize a page twice.
+- **Evidence required:** stacktrace · table schema · writer props (batch size)
+- **Disambiguation:** driven by **value width, not table size** — a small table with one fat JSON column
+ hits this. If the job also stalls at "Building Workload Profile" first, that corroborates.
+- **Mitigation ladder:**
+ 1. Halve the per-sync batch size and observe.
+ 2. If that does not clear it, **pin the estimate instead of computing it**:
+ `hoodie.copyonwrite.record.size.estimate` set well below the 1024 default (e.g. 200). This bypasses
+ dynamic estimation entirely — counter-intuitive, because the configured value is a *lower* number
+ than the real record size, and that is the point.
+ 3. Longer term, keep genuinely large blobs out of wide columns, or reduce row-group size at the source.
+- **Governing configs:** `hoodie.copyonwrite.record.size.estimate`,
+ `hoodie.parquet.small.file.limit`, `hoodie.parquet.max.file.size`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** true — pinning the estimate also stabilizes file sizing on a healthy wide-schema table
+- **Provenance:** none public with this signature
+
+---
+
+## HUDI-RCA-018 — Table-service backlog: compaction, cleaning, or archival falling behind
+
+- **Category:** `sizing/scale` · **Discoverability:** **`silent`** early, `needs-expertise` once terminal
+- **Signature:** Starts silent — growing log files, growing `.hoodie`, lengthening write times — then
+ surfaces as:
+ - `INFO No Instants to archive`
+ - `Not archiving as there is no compaction yet on the metadata table`
+ - OOM inside `TimelineArchiverV1`/`TimelineArchiverV2.archive`, `HoodieTimelineArchiveLog.archive`
+ before 1.0 (see HUDI-RCA-015, archival variant)
+- **Symptom:** Writes get progressively slower, `.hoodie` accumulates thousands of instants, object-store
+ list costs climb, and eventually something fails hard.
+- **Root cause:** A pending or stuck instant blocks archival; the timeline cannot archive past an
+ incomplete table-service instant, and the metadata table's timeline cannot archive until the data
+ table's does.
+- **Evidence required:** **health-checker output** — this is the pattern the checker exists for ·
+ `hoodie.properties` · writer props
+- **Disambiguation:** diagnose the chain in order using ladder **L4**: archival → clean → compaction →
+ blocking inflight instant. Each link gates the next, and fixing the wrong link accomplishes nothing.
+ Note that **archival silently overrides `hoodie.keep.min.commits`** when the cleaner's
+ `earliestCommitToRetain` demands more retention. "I set min commits to 20 but have 400 instants" is
+ usually this — **fix the cleaning config, not the archival one.**
+- **Mitigation ladder:**
+ 1. Run `HoodieTableHealthChecker --checks archival,cleaner,compaction,mdt-compaction --output JSON`
+ and read the chain rather than guessing which service is behind.
+ 2. Finish or roll back pending table-service instants on the data table first; metadata-table archival
+ unblocks behind it. **Look for a pending clustering plan first** (`.replacecommit.requested` /
+ `.inflight` without a completed `.replacecommit`): clustering is optional, so it is routinely
+ enabled and later disabled, and a plan it scheduled is rolled back by nothing — not the writer, not
+ lazy failed-write cleaning. Re-enable clustering in the same mode, let the plan complete, then
+ disable if wanted (L4 step 4).
+ 3. Fix the **cleaner** configuration if `earliestCommitToRetain` is what pins the timeline. Tuning
+ archival while the cleaner holds the line does nothing.
+ 4. For MOR, ensure compaction is actually scheduled and running — inline with
+ `hoodie.compact.inline.max.delta.commits`, or a dedicated offline compactor.
+ 5. **Do not hand-delete `.requested` or `.inflight` files.** Use the CLI's unschedule path. Deleting
+ timeline state by hand is how a recoverable backlog becomes an unrecoverable table.
+- **Governing configs:** `hoodie.keep.min.commits`, `hoodie.keep.max.commits`,
+ `hoodie.commits.archival.batch`, `hoodie.cleaner.commits.retained`, `hoodie.clean.automatic`,
+ `hoodie.archive.automatic`, `hoodie.compact.inline`, `hoodie.compact.inline.max.delta.commits`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — `HoodieTableHealthChecker` detects this
+ class before it becomes terminal
+- **`alsoAppliesWhenHealthy`:** **true** — service cadence tuning is the archetypal healthy-table optimization
+- **Provenance:** #9478, #2515, #2020
+
+---
+
+## HUDI-RCA-019 — Streaming source throttled by the provider
+
+- **Category:** `environment` / `sizing/scale` · **Discoverability:** `needs-expertise` — the provider
+ exception is clear, but the correct tuning direction is the opposite of the instinct
+- **Signature** (innermost cause):
+ - A provider throughput-exceeded exception from the source read path — on shard-based streams,
+ `ProvisionedThroughputExceededException`
+ - `org\.apache\.hudi\.exception\.HoodieIOException: Failed to list shards for \S+` during shard discovery
+ - Often lands in an "unknown" failure bucket because the exception is provider-specific rather than Hudi's
+- **Symptom:** Ingestion throughput collapses or fails intermittently against a shard-based streaming
+ source. Adding capacity appears not to help.
+- **Root cause:** The job is requesting records faster than the per-shard quota allows. Two things
+ routinely hide this: **closed shards still appear in listings but contribute zero read capacity**, so a
+ scale-up that lands on closed shards changes nothing; and every component reading those shards consumes
+ the same per-shard request quota, including background lag and monitoring probes, not just the
+ ingestion loop.
+- **Evidence required:** stacktrace · writer props **as resolved, not as written** · shard and throughput
+ metrics (count of **open** shards, actual average record size)
+- **Disambiguation:** before tuning, measure actual average record size and the count of **open** shards.
+ A throttling exception on a stream that was just scaled up usually means the new shards are closed.
+- **Mitigation ladder:**
+ 1. Slow the poll: raise the get-records interval (e.g. to 1000 ms; provider defaults are often far
+ lower). Counter-intuitive, because the instinct under throttling is to retry harder.
+ 2. **Reduce** max records per request step-wise (e.g. 10000 → 5000 → 2000) until stable. Derive the
+ target from per-shard byte limit ÷ average record size ÷ polls per second. **Do not copy a number
+ from another deployment** — the right value is payload-specific.
+ 3. Add open shards, verifying they are actually open and not closed-but-listed.
+ 4. Reduce the frequency of auxiliary probes competing for the same shard quota.
+ 5. **Verify the key actually took effect** — see ladder **L12**. A misspelled Hudi config key produces
+ no error at all; the library default simply applies, and the intended interval is silently ignored.
+- **Governing configs:** the source-specific throughput keys under `hoodie.streamer.source.*` for your
+ connector (records per request, poll interval) — resolve the exact key for your source and release
+ with the config catalog rather than assuming, and see L12 on verifying it took effect
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — poll pacing against shard quota is an ordinary throughput
+ optimization on a healthy stream
+- **Provenance:** none public with this signature
+
+---
+
+## HUDI-RCA-020 — Timeline bloat from a savepoint that blocks archival
+
+- **Category:** `input/config` (the stray savepoint) with a `defect` component (the archival gap) ·
+ **Discoverability:** **`silent`** until terminal
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieRemoteException: Failed to initialise timeline remotely in server, status code: 413`
+ - `Payload Too Large`
+ - Frame: `RemoteHoodieTableFileSystemView.initialiseTimelineInRemoteView`
+ - Archival log line: `Setting Earliest Commit to Not Archive same as Earliest commit to retain`
+- **Symptom:** The timeline grows invisibly for weeks, then ingestion breaks outright with a 413 on
+ timeline initialization.
+- **Root cause:** **Archival stops at the earliest savepoint, not the latest.** One forgotten savepoint
+ pins the entire timeline after it. The active timeline grows without bound until the serialized
+ timeline exceeds what the embedded timeline server will accept.
+- **Evidence required:** stacktrace · **health-checker output** (`--checks savepoint,archival`) ·
+ `.hoodie` instant count
+- **Disambiguation:** a 413 on timeline init is almost never a network or payload-size problem. Check
+ for a savepoint **first** — it is the single most commonly missed check on this signature.
+- **Mitigation ladder:**
+ 1. Run `HoodieTableHealthChecker --checks savepoint,archival --output JSON`. The savepoint check
+ exists precisely for this.
+ 2. **Immediate unblock:** disable remote timeline initialization / the embedded timeline server so the
+ oversized request is never made. Minutes, and buys room to do the real fix.
+ 3. **Remove the stray savepoint.** This is the actual fix; archival resumes and the timeline shrinks
+ on its own.
+
+ **Human gate — this is durable and unrecoverable.** A savepoint is the only thing standing
+ between its file slices and the cleaner; once released, a restore to that point is gone for good.
+ An agent may identify the savepoint, say why it is blocking archival and state what releasing it
+ costs, but **must not run or script the release**. Confirm with whoever took it that the
+ recovery point is genuinely no longer needed — a savepoint nobody remembers taking is not the
+ same as one nobody needs. Note that `hoodie.archive.beyond.savepoint` is **not** an alternative
+ here: the feature is not fully supported, and archival will not move past the first savepoint
+ regardless of the setting.
+ 4. **Do not reach for aggressive archival scheduling first.** Counter-intuitive but load-bearing: with
+ the savepoint still in place, a more frequent archival job prunes nothing and costs object-store
+ requests. It is pointless until the savepoint is gone.
+ 5. Re-enable remote timeline init once the instant count is back to a few thousand.
+ 6. Tune archival so it is not trying to prune the whole backlog in one round.
+- **Governing configs:** `hoodie.embed.timeline.server`,
+ `hoodie.filesystem.remote.backup.view.enable`, `hoodie.keep.min.commits`, `hoodie.keep.max.commits`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — the `savepoint` health check
+- **`alsoAppliesWhenHealthy`:** **true** — a forgotten savepoint is worth finding long before it is terminal
+- **Provenance:** #9478
+
+---
+
+## HUDI-RCA-021 — Non-partitioned MOR table: rollback wedge and whole-table listings
+
+- **Category:** `sizing/scale` (a design shape, not a tuning value) · **Discoverability:** `needs-expertise`, severely
+- **Signature:** A cluster of symptoms rather than one exception:
+ - Tasks blocked off-CPU in `HoodieAppendHandle.doAppend` → `HoodieLogFile.rollOver` →
+ `FSUtils.computeNextLogVersion` → `getAllLogFiles` → `listStatus`
+ - A `.rollback.inflight` plan file of **megabytes** that never completes and **grows** across attempts
+ - Driver timeline server: `OutOfMemoryError: Java heap space` and
+ `OutOfMemoryError: Requested array size exceeds VM limit` in Jetty threads under
+ `objectMapper.writeValueAsString`
+ - Downstream consequences: executor `SocketTimeoutException: Read timed out`, marker-create failures
+ (HUDI-RCA-031), writers queued behind the stuck rollback reporting lock acquisition failures
+- **Symptom:** A large non-partitioned MOR table that progressively wedges, firing several unrelated-looking
+ alerts at once.
+- **Root cause:** Two mechanisms compounding. MOR failed-write rollback uses the **listing** strategy,
+ enumerating every file in the partition — and on a non-partitioned table the "partition" is the whole
+ table, so rollback cost scales with total file-group count rather than with the failed instant.
+ Separately, the timeline server serializes the entire file-system view into a single JSON string,
+ which on a whole-table view hits the JVM's roughly 2 GB single-array ceiling. That is a **structural**
+ limit, not a heap-size one.
+- **Evidence required:** stacktrace · `hoodie.properties` (table type, partitioning) · layout analysis
+ (file-group count) · `.rollback.inflight` plan sizes · driver log · Spark event log
+- **Disambiguation:** note that **rolling back a completed instant ignores `hoodie.rollback.using.markers`
+ entirely** — the config is ANDed with "instant is not completed", so restores and savepoint rollbacks
+ always take the listing path regardless of how that config is set. This is the mechanism behind "my
+ rollback took longer than the write", and it is why setting the marker config does not help here.
+- **Mitigation ladder:**
+ 1. **Do not restart the driver while a rollback is in flight.** Each restart abandons it and re-requests
+ a larger one. The plan file growing across attempts is the proof.
+ 2. **Do not disable compaction to dodge a write conflict.** It stops the conflict and lets log files
+ pile up per file group without bound, making both the per-append listing and every future rollback
+ slower. This is the single most damaging available move.
+ 3. Move the file-system view off heap: `hoodie.filesystem.view.type=SPILLABLE_DISK`, with
+ `hoodie.filesystem.view.spillable.mem` sized around 1 GB and
+ `hoodie.common.spillable.diskmap.type=ROCKS_DB`.
+ 4. Raise `hoodie.markers.timeline_server_based.batch.num_threads` so marker traffic is not the next
+ bottleneck.
+ 5. Size compaction for the workload: `hoodie.compaction.target.io` must be large enough that one round
+ sweeps most candidate file groups. If only half fit per round, a tail accumulates forever. Raise it
+ substantially rather than nudging it.
+ 6. Give the driver more heap only after 3 through 5 — for the JSON serialization path this is a ceiling
+ problem, not a heap problem.
+ 7. Prefer partitioning the table. A non-partitioned MOR table is pathological for both the listing-based
+ rollback path and the whole-table file-system-view path.
+- **Governing configs:** `hoodie.filesystem.view.type`, `hoodie.filesystem.view.spillable.mem`,
+ `hoodie.common.spillable.diskmap.type`, `hoodie.markers.timeline_server_based.batch.num_threads`,
+ `hoodie.compaction.target.io`, `hoodie.rollback.using.markers`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** partial — layout analysis surfaces the shape
+- **`alsoAppliesWhenHealthy`:** **true** — the view and compaction sizing are optimizations well before the wedge
+- **Provenance:** none public with this combination
+
+---
+
+## HUDI-RCA-022 — Unbounded incremental clustering plan wedges everything behind it
+
+- **Category:** `sizing/scale` with a `defect` component · **Discoverability:** `needs-expertise` — several
+ alerts fire, none of which names clustering
+- **Signature:**
+ - A `.replacecommit.requested` whose **plan file is tens to hundreds of MB**, `.inflight` for many hours
+ - Driver `java\.lang\.OutOfMemoryError: GC overhead limit exceeded`, with a heap dump showing many
+ duplicated copies of the same large plan byte array across worker threads
+ - Metadata-table compaction stops scheduling entirely; MDT log blocks accumulate on the `files` partition
+ - Marker CREATE timeouts, connection-pool shutdown, container OOM-kills where the Java heap looks healthy
+- **Symptom:** A multi-day outage on a heavily-partitioned append-only table, announced by half a dozen
+ symptoms that each look like a different problem.
+- **Root cause:** The incremental clustering planner caps group count and total bytes but **not input
+ file-slice count**. On a heavily-partitioned append-only table where clustering has fallen behind, it
+ accumulates pending partitions and tries to catch up all at once. The plan is then re-read and fully
+ deserialized **on every timeline refresh**, just to answer "is this a clustering instant?", so each
+ worker thread pins its own copy.
+- **Evidence required:** table state (`.replacecommit.requested` age and plan size, decoded plan) ·
+ heap dump · Spark event log
+- **Disambiguation:** the giveaway is plan **file size**. Over a few MB is already a warning. If ingestion
+ writes roughly one file group per commit and the pending plan proposes hundreds of thousands of file
+ slices, the plan is wildly disproportionate and is the cause, not a victim.
+- **Mitigation ladder:**
+ 1. **Capture heap dumps from the driver and a few executors before restarting anything.** Restarting
+ destroys the only evidence and does not clear the wedge.
+ 2. **Disable clustering entirely** and let ingestion and metadata-table compaction drain. This is the
+ only reliable break. Counter-intuitive: throttling clustering while the giant plan is still pending
+ does not help, because the cost is in deserializing the existing plan, not in making new ones.
+ 3. Re-enable with a reduced per-round group count and a cap on input file groups per plan where your
+ release supports it.
+ 4. If executors are container-OOM-killed with a healthy Java heap, suspect **native** memory from
+ compression codecs: raise `spark.executor.memoryOverheadFactor`.
+ 5. Keep clustering cadence ahead of partition creation so a backlog never forms.
+- **Governing configs:** `hoodie.clustering.inline`, `hoodie.clustering.async.enabled`,
+ `hoodie.clustering.plan.strategy.max.num.groups`,
+ `hoodie.clustering.plan.strategy.max.bytes.per.group`, `spark.executor.memoryOverheadFactor`
+- **`fixedInVersion`:** n/a — the input-slice-count cap is the missing guard
+- **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — clustering cadence versus partition creation rate is a healthy-table check
+- **Provenance:** none public with this mechanism
+
+---
+
+## HUDI-RCA-023 — Long-lived driver leaks spillable file-system-view resources
+
+- **Category:** `defect` surfacing as `sizing/scale` · **Discoverability:** **`silent`**
+- **Signature (a heap dump, not a stacktrace):**
+ - Leak suspect: `java\.lang\.ApplicationShutdownHooks` retaining the bulk of heap via tens of thousands
+ of shutdown-hook threads
+ - `BitCaskDiskMap` and `HoodieFileGroupId` dominating the object histogram
+ - Externally: a busy-thread dump where nearly all top-CPU threads are GC threads
+- **Symptom:** No exception. The driver stays "Ready", ingests in bursts, and degrades. Several different
+ alerts fire, none naming the cause.
+- **Root cause:** With `SPILLABLE_DISK` file-system views, each spillable map registers a JVM shutdown
+ hook. Table-service cycles rebuild the view on every full refresh, and replaced maps are abandoned
+ without being closed, so the hooks are never deregistered and pin the maps for the life of the JVM.
+ Leak volume scales with tables multiplied by table-service cycles.
+- **Evidence required:** **heap dump — a stacktrace is useless here** · writer props (view type) ·
+ driver uptime
+- **Disambiguation:** note the direct tension with HUDI-RCA-021, which prescribes `SPILLABLE_DISK` for a
+ different reason. Apply the view type **per table**, not globally: a table with a very large view needs
+ spillable; a long-lived driver hosting many small tables does not.
+- **Mitigation ladder:**
+ 1. Set `hoodie.filesystem.view.type=MEMORY` for the dominant table — this is Hudi's own default and
+ removes the leaking path.
+ 2. Restart the driver to clear accumulated hooks — **after** capturing dumps, not before.
+ 3. Shorten driver lifetime or reduce tables per driver as a stopgap.
+ 4. Upgrade to a release where the table object is closed per cycle.
+- **Governing configs:** `hoodie.filesystem.view.type`, `hoodie.filesystem.view.spillable.mem`,
+ `hoodie.common.spillable.diskmap.type`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public — requires heap-dump forensics on a long-running multi-table driver
+
+---
+
+## HUDI-RCA-024 — Object-store connection pool exhausted
+
+- **Category:** `sizing/scale` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause — **both parts required**):
+ - An SDK client exception, **and** the text `Timeout waiting for connection from pool`
+ - Wrapped in `org\.apache\.hudi\.exception\.HoodieIOException: getFileStatus on `
+ - Frame: the filesystem connector's exception-translation utility
+- **Symptom:** Intermittent to persistent IO failures under high file-listing or write concurrency,
+ reading like a storage outage.
+- **Root cause:** The filesystem client's HTTP connection pool is smaller than the concurrency the job
+ drives. More executors and more threads make it worse.
+- **Evidence required:** stacktrace · writer props and Hadoop config · executor count
+- **Disambiguation:** the match requires **both** the SDK client exception **and** the pool-timeout text.
+ Other SDK errors from the same connector are a different problem entirely — do not match on the SDK
+ exception class alone.
+- **Mitigation ladder:**
+ 1. Double the connection-pool maximum (`fs.s3a.connection.maximum` on the S3A connector) and restart.
+ Repeat once if needed.
+ 2. Above roughly 2000, **stop doubling and think.** Per-pod connection counts compound across executors,
+ and you can exhaust the host or the endpoint instead of fixing anything.
+ 3. **Reduce job-side concurrency** — fewer executors, lower listing parallelism. Counter-intuitive, but
+ it attacks the demand side rather than the supply side, and the demand side is the one that scales.
+ 4. Reduce the number of file-listing operations outright: enable the metadata table, partition better.
+- **Governing configs:** `fs.s3a.connection.maximum` (or the equivalent for your connector),
+ `hoodie.metadata.enable`, `hoodie.file.listing.parallelism`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** — listing reduction is an optimization on any healthy job at scale
+- **Provenance:** none public with this exact match rule
+
+---
+
+# Group C — defects and table state
+
+Hudi reached a state it should not have. Remedies are to pin or upgrade a version, rebuild derived
+state, or — in one case only — run a supervised repair. **Attribution matters here:** these are not the
+user's mistake, and saying so plainly is part of the job.
+
+---
+
+## HUDI-RCA-025 — Dangling data files from an incomplete rollback — EXPLAIN ONLY
+
+> **⚠️ EXPLAIN-ONLY ROW — HARD CONSTRAINT FOR THE AGENT AND FOR FUTURE EDITORS ⚠️**
+>
+> The agent **identifies** this pattern, **names it as a known defect class rather than the user's
+> mistake**, points at `HoodieTableHealthChecker`, and **states that repair requires the safe-deletion
+> protocol (ladder L3) with a human driving it.**
+>
+> **The agent must never recommend, script, enumerate, or sequence the file deletion.** Not as a
+> suggestion, not as an example, not "for reference", not even when the user asks directly. The correct
+> response to "just tell me which files to delete" is to restate that the deletion premise needs two
+> independent verifications and a savepoint, and to hand over L3 for a human to execute.
+>
+> This row is the template for any future pattern whose only remedy is destructive. If you are adding
+> such a pattern, copy this block.
+
+- **Category:** `defect` · **Discoverability:** `needs-expertise` — and actively misleading
+- **Signature** (innermost cause):
+ - `java\.util\.NoSuchElementException: FileID \S+ of partition path \S+ does not exist`
+ - Frames: `HoodieMergeHandle.getLatestBaseFile` → `BaseSparkCommitActionExecutor.getUpdateHandle`
+ - In the driver log beforehand: `BaseRollbackActionExecutor : Rollback of Commits [<instant>] is complete`
+- **Symptom:** Ingestion fails on **every** attempt after a prior write failure, deterministically, on
+ the same file ID across restarts.
+- **Root cause:** A commit failed and was rolled back, but its data files and markers were never
+ physically deleted. A later operation resolves a file slice to an untracked file that has no timeline
+ entry. **The message says a file "does not exist", but the actual condition is an *extra*, untracked
+ file** — which is why it reads like user error or data loss when it is neither.
+- **Evidence required:** stacktrace · **health-checker output** · table state (partition listing,
+ `.hoodie`, `.hoodie/.temp/`) · driver log for the rollback-complete line
+- **Disambiguation:** shares `FileNotFoundException`-family symptoms with several rows — see the
+ disambiguation index. A missing path under the **source** bucket is HUDI-RCA-036. A path under the
+ table ending `.log.<n>` indicates a deltacommit that raced an unfinished compaction, which is a
+ different mechanism with a different (also destructive, also human-driven) remedy. Confirm all of:
+ the file exists on storage, its instant is **not** in the active timeline, and its instant **is**
+ named by a later `.rollback`.
+- **How this is explained to the user:**
+ 0. **Run ladder L14 before attributing here.** If the metadata table is enabled, listing drift
+ (HUDI-RCA-026) is the likelier cause and has a reversible unblock; if the failure is on the read
+ path, it is probably the cleaner boundary; if the onset is recent, restore is cheaper than any
+ investigation. This row is where you land after those are ruled out.
+ 1. Name it: this is a known timeline-consistency defect class, not a configuration mistake. The error
+ message is misleading by design-accident.
+ 2. Point at `HoodieTableHealthChecker` to characterize the table's current state.
+ 3. State that repair means removing untracked files, that this is destructive and irreversible, and
+ that it must run inside the **safe-deletion protocol (L3)**. Name the protocol and its gates in
+ **one sentence** (stakeholder notice, writer paused, savepoint taken, two independent verifications
+ of the deletion premise, backup copy, human executes). **Do not** expand it into a lettered
+ runbook, do not state the criteria that identify the files, and do not add re-run/resume/drop-
+ savepoint steps around it. The user gets the protocol's name and a human; not a procedure.
+ 4. State that a **human must drive it**, and that if any verification is ambiguous the correct action
+ is to stop: deleting a non-orphan corrupts the table silently.
+ 4b. Be precise about data: the table's committed data is intact, **and** the killed batch was never
+ committed, so it must be re-ingested once writes resume. Never say "no data lost" without the
+ second half.
+ 5. Where the release supports it, note that `rollback` or `restore` to the last clean commit is an
+ alternative to hand-deletion — it rewrites the timeline, so it is also expert-only and also L3.
+ Count the completed commits since the failing instant before describing its cost: when every write
+ since the failure has itself failed, restore discards nothing.
+ 6. Upgrade: the deltacommit-removed-while-requested-remained trigger is fixed in later releases, so
+ establishing the version is worthwhile before anything else.
+- **Governing configs:** `hoodie.clean.failed.writes.policy`, `hoodie.rollback.using.markers`,
+ `hoodie.filesystem.view.incr.timeline.sync.enable` (experimental; if it is on, that alone can produce
+ plans naming wrong filenames — turn it off)
+- **`fixedInVersion`:** 1.0.x for the requested/inflight-orphan trigger · **`preflightGuardExists`:** partial
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #11202, #2020, #19290
+
+---
+
+## HUDI-RCA-026 — Metadata table disagrees with the filesystem
+
+- **Category:** `defect` (data correctness) · **Discoverability:** **`silent`** — reads can return wrong or
+ missing files with no write failure at all
+- **Signature:** Validator output from `HoodieMetadataTableValidator`:
+ - `Validation of latest file slices for partition \S+ failed`
+ - `File system-based listing: \d+ & MDT-based listing: \d+`
+ - `Validation of record index content failed`
+ - May instead surface as `HoodieMetadataException: Failed to retrieve files in partition`, or as
+ `Invalid number of file groups for partition:column_stats`
+- **Symptom:** Queries return wrong or missing files, or an offline operation cannot list partitions.
+ Often no write failure at all.
+- **Root cause:** The metadata table diverged from the data table. Known triggers: enabling column-stats
+ or bloom-filter indexes mid-life on an existing table; duplicate concurrent writers caused by
+ `spark.speculation=true`, which could trigger a record-index purge; a deltacommit deleted while its
+ requested/inflight state remained.
+- **Evidence required:** validator output · **both timelines, active and archived** · writer props
+ (`hoodie.clean.failed.writes.policy`, `hoodie.write.concurrency.mode`,
+ `hoodie.write.lock.provider` — these gate which rollback path can fire)
+- **Disambiguation — classify the direction first, per table, before touching anything:**
+
+ | Direction | Meaning | Response |
+ |---|---|---|
+ | **MDT behind** | Storage has slices the metadata table lacks | Rebuild the metadata table |
+ | **MDT ahead** | The metadata table lists slices storage no longer has | **A rebuild does not fix this** — it re-derives from a data table whose storage still holds orphans, so the divergence returns |
+ | **Per-slice** | Same count, but a file group resolves to competing base files differing only in write token | HUDI-RCA-028 |
+
+ One validator run can cover several tables diverging in **opposite** directions from the same defect.
+ Interpreting the timeline evidence: a `.clean` naming the file means the cleaner removed it logically
+ while the object survived. A data-side `.rollback` whose rolled-back instant **is** the divergent base
+ instant, with no filename hit, means the commit was rolled back and the objects are orphan task-attempt
+ files — **the metadata table is correct, do not rebuild.** A `.replacecommit` means clustering replaced
+ the slice, and replaced files leave only when a later clean deletes them.
+- **Mitigation ladder:**
+ 1. **Archive the evidence first.** Both branches overwrite the diverged state; once you rebuild, the
+ evidence is gone.
+ 2. Confirm the signal is real before investigating at all — see ladder **L6**. Several tables failing
+ the same check within minutes usually means the *check* changed, not the data.
+ 3. **Branch A, metadata table is wrong:** rebuild it with `metadata delete` then `metadata create`
+ in hudi-cli, then re-run the validator.
+ **Human gate — this drops every index.** `hoodie.metadata.auto.delete.partitions` defaults to `true`, so disabling the metadata table deletes it outright via `HoodieTable.maybeDeleteMetadataTable` — record index, column stats, bloom filters and secondary indexes, not only `files`. With it set to `false` the table instead stays on storage, unmaintained, while readers keep using it, which is worse. Stop every other writer first. Prefer `metadata delete` then `metadata create` in hudi- [...]
+ 4. **Branch B, metadata table is right and storage has orphans:** rebuilding will not fix it. The
+ orphans must be removed under the safe-deletion protocol (L3), human-driven — see HUDI-RCA-025.
+ 5. **If neither branch matches positively, do not mitigate. Escalate.** A wrong branch either leaves the
+ divergence in place or deletes files on a false premise.
+ 6. Set `spark.speculation=false` — Hudi 1.2.0 fails fast on this for good reason.
+ 7. If the failure is index-specific, disabling that specific index is a confirmed workaround.
+- **Governing configs:** `hoodie.metadata.enable`, `hoodie.metadata.index.column.stats.enable`,
+ `hoodie.metadata.index.bloom.filter.enable`, `hoodie.metadata.record.index.enable`,
+ `hoodie.metadata.compact.max.delta.commits`, `spark.speculation`
+- **`fixedInVersion`:** 0.14.x for the mid-life index-enable trigger; 1.2.0 guards the speculation trigger
+- **`preflightGuardExists`:** yes in 1.2.0 for `spark.speculation`; also the `mdt-sync` health check
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #7657, #11567, #11202, #9478
+
+---
+
+## HUDI-RCA-027 — Rollback failure wedging the writer
+
+- **Category:** `defect` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieRollbackException: Failed to rollback \S+ commits \d+`
+ - Frames: `BaseHoodieWriteClient.rollback` ← `rollbackFailedWrites` ← `CleanerUtils.rollbackFailedWrites`
+ ← `BaseHoodieWriteClient.startCommitWithTime`
+- **Symptom:** Every subsequent run fails at startup while trying to clean up a previous failed write.
+ Self-perpetuating — the job can never get past commit start.
+- **Root cause:** A previous write died leaving an inflight instant whose rollback cannot complete,
+ usually because the files it references are already gone, a lock could not be acquired to perform the
+ rollback, or markers are missing.
+- **Evidence required:** stacktrace · writer props (lock config) · **health-checker output** · timeline state
+- **Disambiguation — `HoodieRollbackException` maps to three patterns. Read the frame below
+ `rollbackFailedWrites`:**
+ - `AsyncCleanerService.waitForCompletion` → metadata-table or cleaner path → HUDI-RCA-026
+ - A lock-provider frame (`LockManager.lock`, `*LockProvider.acquireLock`) → lock problem first →
+ HUDI-RCA-006
+ - `HoodieMergeHandle.getLatestBaseFile` with `FileID ... does not exist` → HUDI-RCA-025
+ - None of these, and the rollback simply never finishes on a large table → HUDI-RCA-021
+- **Mitigation ladder:**
+ 1. Check for and fix lock-provider problems first. Rollback happens inside the lock, so a lock problem
+ presents as a rollback problem.
+ 2. Set `hoodie.cleaner.policy.failed.writes=LAZY` so a single writer does not eagerly roll back another
+ writer's in-flight commit.
+ 3. Inspect the timeline and roll back the specific instant through the CLI rather than deleting files.
+ 4. Last resort: savepoint, then restore to a known-good instant. This rewrites the timeline — expert-only.
+ 5. If the rollback is not failing but merely never finishing, stop and read HUDI-RCA-021 before
+ restarting anything.
+- **Governing configs:** `hoodie.clean.failed.writes.policy`, `hoodie.write.lock.provider`,
+ `hoodie.rollback.using.markers`, `hoodie.rollback.parallelism`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #9213, #7657, #5765
+
+---
+
+## HUDI-RCA-028 — Two base files for one file slice (duplicate write token)
+
+- **Category:** `defect` · **Discoverability:** **`silent`** — the primary symptom is wrong query results,
+ with no exception anywhere
+- **Signature:** Two base files sharing the same file-group id **and** the same instant time, differing
+ only in write token:
+ - `<fileId>_0-\d+-\d+_<instant>\.parquet` alongside `<fileId>_1-\d+-\d+_<instant>\.parquet`
+ - Downstream: `COUNT(1) > COUNT(DISTINCT <record_key>)`
+- **Symptom:** Duplicate rows for one primary key, with no failure signal at all.
+- **Root cause:** A Spark task retry combined with cached-RDD invalidation produced two output files
+ where commit metadata records one. With the metadata table enabled it should guard against serving the
+ untracked one; without it, file-slice resolution can pick either, non-deterministically.
+- **Evidence required:** per-file-group listing · commit metadata · compaction plan if compaction has
+ run · a duplicate-count query
+- **Proactive detection:** `SELECT COUNT(1), COUNT(DISTINCT <record_key>)`, then for each file group list
+ base files sharing one instant time. More than one is the signature.
+- **Mitigation ladder — strictly ordered, and the most delicate procedure in this catalog:**
+ 1. Notify stakeholders, **pause the writer**, **take a savepoint**. Everything below is inside the
+ safe-deletion protocol (L3) and requires a human.
+ 2. **Latest file slice, compaction has not run:** if there are no log files on the untracked base file,
+ the untracked base file can be removed under L3. If log files exist only on the *tracked* slice, the
+ same holds. If log files are on the *untracked* slice, trigger or wait for compaction on that file
+ group first, then re-assess.
+ 3. **Compaction already ran and more updates landed:** compaction considered only one of the two base
+ files — read the compaction plan to learn which. Diff the missed base file against the
+ post-compaction base file, excluding records a later commit updated (they may now live in a different
+ file group), and re-ingest whatever is genuinely missing. **There is no shortcut here.**
+ 4. Validate, resume, let a few commits land, then pause, delete the savepoint, resume.
+ 5. If the writer failed during metadata-table initialization because of these files, rebuild the
+ metadata table (`metadata delete` then `metadata create` in hudi-cli) after doing the cleanup in
+ a quiet window.
+ **Human gate — this drops every index.** `hoodie.metadata.auto.delete.partitions` defaults to `true`, so disabling the metadata table deletes it outright via `HoodieTable.maybeDeleteMetadataTable` — record index, column stats, bloom filters and secondary indexes, not only `files`. With it set to `false` the table instead stays on storage, unmaintained, while readers keep using it, which is worse. Stop every other writer first. Prefer `metadata delete` then `metadata create` in hudi- [...]
+- **Governing configs:** `hoodie.metadata.enable`, `spark.speculation`, `hoodie.clean.failed.writes.policy`
+- **`fixedInVersion`:** later releases made file-slice resolution deterministic; the cleanup procedure is
+ the durable value · **`preflightGuardExists`:** partial
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #2338 (duplicate symptom reports); the two-base-file mechanism is not separately filed
+
+---
+
+## HUDI-RCA-029 — Record-level index: duplicate key while writing the index HFile
+
+- **Category:** `defect` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieDuplicateKeyException: Duplicate key found for insert statement`
+ - `Duplicate recordKey \S+ found while writing to HFile`
+ - Frames: `HoodieAvroHFileWriter.writeAvro` → `HoodieCreateHandle.doWrite`
+ - **Tell-tale:** `partitionPath=record_index` and a fileId shaped `record-index-\d+-0` — this is the
+ metadata table's record-index partition, **not** the user's data
+- **Symptom:** Ingestion fails with a duplicate-key error naming internals the user has never seen.
+- **Root cause:** Two entries for the same record key reach the same record-index file group in one write.
+ HFiles require strictly increasing keys, so the writer aborts.
+- **Evidence required:** stacktrace · metadata-table timeline · writer props (index config)
+- **Disambiguation:** the `record_index` partition path is what makes this a metadata-table defect rather
+ than a duplicate in the user's data. Do not send the user looking for duplicates in their source.
+- **Mitigation ladder:**
+ 1. **Archive evidence first** — copy `.hoodie`, and the whole table if it is small, to a backup location.
+ This is corruption evidence and the mitigation destroys it.
+ 2. Disable the record-level index and fall back to a global index so ingestion proceeds:
+ `hoodie.metadata.record.index.enable=false`, `hoodie.index.type=GLOBAL_SIMPLE`.
+ 3. Resume. Re-enable the record index only after the defect is understood — re-enabling rebuilds it
+ from scratch, which is expensive and erases the evidence.
+ 4. Note that the record index silently falls back to a simple index when it is uninitialized, logging
+ only a warning. A table can therefore start doing full base-file key scans with no error at all —
+ check for that warning if performance regressed rather than failed.
+- **Governing configs:** `hoodie.metadata.record.index.enable`, `hoodie.index.type`,
+ `hoodie.metadata.enable`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — the `record-index` health check sizes the index
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public with this signature
+
+---
+
+## HUDI-RCA-030 — A zero-column schema gets committed and poisons the table
+
+- **Category:** `defect` with an `input/config` trigger · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.NullPointerException` at `AvroSchemaEvolutionUtils.reconcileSchema`
+ - Preceded in the log by `Seeing new schema. Source: \[null\], Target: \{.*"fields": \[\]` and, on each
+ retry, an empty-batch marker from the source
+- **Symptom:** Every sync fails identically. The checkpoint never advances, so a restart re-reads the same
+ checkpoint and reproduces it exactly — a closed loop. **Restarting is provably useless.**
+- **Root cause:** A brand-new table's first batch produced zero rows and zero columns, so the deduced
+ writer schema was an empty record and was committed. Afterwards, schema reconciliation returns the null
+ source schema untouched whenever the table schema has zero fields, and that null reaches a dereference
+ guarded only against the Avro `NULL` *type*, never against a Java null reference.
+- **Evidence required:** stacktrace · last commit metadata (`extraMetadata.schema`) · writer props
+ (schema provider)
+- **Disambiguation:** the poison is visible: read `extraMetadata.schema` from the latest `.commit` and
+ look for `"fields": []`. Established tables are not exposed to this — confirm the table is new.
+- **Mitigation ladder:**
+ 1. **Do not restart the job.** The checkpoint lives in commit metadata, so restarting re-throws the same
+ exception. This is the counter-intuitive step, and it is the first thing most operators try.
+ 2. Apply a release with a null guard in schema reconciliation. There is no safe config-only escape.
+ 3. Without a patched release, the only alternative is hand-advancing the checkpoint past the empty
+ batch — which risks skipping un-ingested data. Weigh that explicitly and in writing before doing it.
+ 4. Prevent recurrence by pinning an explicit source schema, so a zero-row first batch cannot produce a
+ zero-column schema.
+- **Governing configs:** `hoodie.streamer.schemaprovider.source.schema.file`,
+ `hoodie.streamer.schemaprovider.target.schema.file`, `hoodie.avro.schema.validate`
+- **`fixedInVersion`:** n/a — the null guard is the fix · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public — the trigger is narrow (a brand-new table whose very first batch is empty)
+
+---
+
+## HUDI-RCA-031 — Timeline-server or marker creation failure
+
+- **Category:** `environment`, sometimes `defect` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieRemoteException: Failed to create marker file \S+\.marker\.(CREATE|MERGE|APPEND)`
+ - `java\.net\.SocketTimeoutException: Read timed out` at
+ `TimelineServerBasedWriteMarkers.executeRequestToTimelineServer`
+ - `HoodieRemoteException: .*status code: 500`
+ - `Failed to read MARKERS file`
+ - `WARN PriorityBasedFileSystemView: Got error running preferred function\. Trying secondary`
+ - Frames: `TimelineServerBasedWriteMarkers.create`, `HoodieWriteHandle.createMarkerFile`,
+ `LazyIterableIterator.next`
+- **Symptom:** Writes fail mid-stage with a connection or timeout error against an ephemeral
+ driver-hosted HTTP endpoint. Intermittent, worse on busy drivers.
+- **Root cause:** Hudi's embedded timeline server runs on the Spark driver, and executors call it over
+ HTTP for marker creation and file-system views. A saturated driver — large active timeline, huge
+ file-system view, GC pressure — times those calls out. A socket timeout reads as a network problem; the
+ cause is usually **driver saturation**.
+- **Evidence required:** stacktrace · `.hoodie` instant count · driver log and GC behaviour ·
+ health-checker output (is archival running? is a savepoint pinning it?)
+- **Disambiguation:** this is frequently a **consequence**, not a root cause. If the same table also shows
+ a large instant count or a stuck rollback, fix HUDI-RCA-020 or HUDI-RCA-021 first — marker timeouts
+ will clear on their own. Treat a marker timeout on an otherwise-healthy small table as the primary
+ fault; on a big or wedged table, treat it as a symptom.
+- **Mitigation ladder:**
+ 1. **Immediate unblock:** `hoodie.write.markers.type=DIRECT`. Removes the driver round-trip entirely at
+ the cost of more object-store writes. Almost always works.
+ 2. Raise `hoodie.markers.timeline_server_based.batch.num_threads` to stay on timeline-server markers on
+ a busy table.
+ 3. Set `hoodie.embed.timeline.server=false` if you need to be unblocked and can accept the performance
+ cost.
+ 4. Fix the underlying driver pressure: get archival running, move the file-system view off heap, or give
+ the driver more memory.
+ 5. Upgrade — several of these were genuine server-side defects fixed in 0.12 and later.
+- **Governing configs:** `hoodie.write.markers.type`,
+ `hoodie.markers.timeline_server_based.batch.num_threads`, `hoodie.embed.timeline.server`,
+ `hoodie.filesystem.view.type`, `hoodie.filesystem.remote.backup.view.enable`
+- **`fixedInVersion`:** 0.12.0 for several server-side defects · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** true — `DIRECT` markers are a reasonable standing choice on busy drivers
+- **Provenance:** #4230, #5460, #6900, #7689
+
+---
+
+## HUDI-RCA-032 — Missing `hoodie.properties` or instant files
+
+- **Category:** `environment` / `defect` · **Discoverability:** `self-evident` that something is wrong,
+ `needs-expertise` for recovery
+- **Signature:** The writer or a reader cannot open the table at all. `hoodie.properties` absent from
+ `.hoodie/`, or timeline instant files missing.
+- **Symptom:** Total failure to open the table, rather than a failure during a write.
+- **Root cause:** The table's config file or timeline entries were removed — an errant storage lifecycle
+ rule, an interrupted copy or migration, or bad cleanup.
+- **Evidence required:** `.hoodie` listing for **both** the data table and the metadata table · writer props
+- **Mitigation ladder:**
+ 1. **Back up the whole `.hoodie` directory before any repair.**
+ 2. If `hoodie.properties` is missing, look for `hoodie.properties.backup` alongside it — the CLI's
+ config-recovery path restores from it.
+ 3. With no backup, reconstruct from the writer's known configuration: table type, key generator, record
+ key and partition fields, payload class, table version. **Get expert eyes on this** — a wrong value
+ silently changes how every future read resolves records, which is worse than the outage.
+ 4. If the **metadata table** is the affected one, do not hand-repair it. Rebuild it with
+ `metadata delete` then `metadata create` in hudi-cli.
+ **Human gate — this drops every index.** `hoodie.metadata.auto.delete.partitions` defaults to `true`, so disabling the metadata table deletes it outright via `HoodieTable.maybeDeleteMetadataTable` — record index, column stats, bloom filters and secondary indexes, not only `files`. With it set to `false` the table instead stays on storage, unmaintained, while readers keep using it, which is worse. Stop every other writer first. Prefer `metadata delete` then `metadata create` in hudi- [...]
+- **Governing configs:** `hoodie.metadata.enable`, and every immutable key listed under HUDI-RCA-008
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** no canonical public issue — the phrase "hoodie.properties not found" recurs across
+ many tracker threads without a single representative report
+
+---
+
+## HUDI-RCA-033 — Reading commit files races the inline archiver
+
+- **Category:** `defect` (concurrency) · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.io\.FileNotFoundException` reading a `\.deltacommit` or `\.commit` from `.hoodie/`, in a code
+ path **walking the timeline** — typically a catalog sync or an inspection tool, not the writer
+ - **Giveaway: an alternating pass/fail pattern across successive cycles.**
+- **Symptom:** A `FileNotFoundException` for a file that genuinely existed moments earlier. The
+ alternating rhythm is the only reliable clue.
+- **Root cause:** Hudi's inline archiver writes a commit into `.hoodie/archived/` and **deletes** it from
+ `.hoodie/` as part of the commit. Anything holding a cached timeline listing and then reading individual
+ commit files can find them gone. Walking **oldest to newest** maximizes the odds, because the oldest
+ instant is precisely the one about to be archived.
+- **Evidence required:** stacktrace · timing correlation with archival · the reader's iteration order
+- **Disambiguation:** against HUDI-RCA-018 and HUDI-RCA-025, the discriminators are that the failing
+ reader is **not the writer**, and that the failures alternate rather than persist. A persistent
+ failure on the same file is not this pattern.
+- **Mitigation ladder:**
+ 1. In any code reading individual commit files from a cached timeline, iterate **newest to oldest** and
+ tolerate `FileNotFound` for an entry since archived. Newest instants are very unlikely to be
+ archived mid-read.
+ 2. Refresh the timeline immediately before reading, rather than trusting a cached listing.
+ 3. If you cannot change the reader, move archival off the inline commit path so the deletion window is
+ not inside every commit.
+ 4. **General rule worth remembering:** any optimization that replaces a bulk listing with targeted
+ commit-file reads newly exposes you to concurrent archival. That trade-off is the pattern.
+- **Governing configs:** `hoodie.archive.automatic`, `hoodie.archive.async`, `hoodie.keep.min.commits`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public
+
+---
+
+## HUDI-RCA-034 — Metadata-table clean cannot create its requested instant
+
+- **Category:** `defect` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieIOException: Failed to create file \S+/\.hoodie/metadata/\.hoodie/\d+\.clean\.requested`
+ - Frames: `HoodieWrapperFileSystem.createImmutableFileInPath` →
+ `HoodieActiveTimeline.saveToCleanRequested` → `HoodieBackedTableMetadataWriter.cleanIfNecessary`
+- **Symptom:** Repeated clean failures on the metadata table, surfacing as an IO exception.
+- **Root cause:** The metadata table's cleaner tries to create a `.clean.requested` instant that already
+ exists, so the immutable-file create fails. It is a state and idempotency problem wearing an IO
+ exception's clothes.
+- **Evidence required:** stacktrace · metadata-table timeline listing · writer props (cleaner config)
+- **Disambiguation:** **the path is under `.hoodie/metadata/.hoodie/`** — the *metadata table's* timeline,
+ not the data table's. An otherwise-identical message against the data table's `.hoodie/` is a different
+ problem.
+- **Mitigation ladder:**
+ 1. Confirm a stale `.clean.requested` or `.clean.inflight` exists on the metadata table's timeline.
+ 2. Shift the clean schedule so the next attempt lands on a different instant — lengthen cleaner
+ frequency, let two commits succeed, then revert. Counter-intuitive, because the fix is a timing
+ change rather than a state repair.
+ 3. If it persists on a current release, treat it as a regression — this signature was fixed upstream and
+ should not recur.
+ 4. Last resort: rebuild the metadata table with `metadata delete` then `metadata create` in
+ hudi-cli.
+ **Human gate — this drops every index.** `hoodie.metadata.auto.delete.partitions` defaults to `true`, so disabling the metadata table deletes it outright via `HoodieTable.maybeDeleteMetadataTable` — record index, column stats, bloom filters and secondary indexes, not only `files`. With it set to `false` the table instead stays on storage, unmaintained, while readers keep using it, which is worse. Stop every other writer first. Prefer `metadata delete` then `metadata create` in hudi- [...]
+- **Governing configs:** `hoodie.metadata.enable`,
+ `hoodie.metadata.derive.from.datatable.clean.policy`, `hoodie.metadata.compact.max.delta.commits`,
+ `hoodie.clean.automatic`, `hoodie.cleaner.commits.retained`
+- **`fixedInVersion`:** fixed upstream; recurrence on a current release indicates a regression
+- **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public with this signature
+
+---
+
+# Group D — environment
+
+Hudi is fine. The cluster, the cloud, or a dependency failed.
+
+---
+
+## HUDI-RCA-035 — Classpath or bundle version mismatch
+
+- **Category:** `environment` · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.lang\.NoSuchMethodError: scala\.\S+`
+ - `java\.lang\.NoSuchMethodError: org\.apache\.hadoop\.\S+`
+ - `java\.lang\.NoClassDefFoundError: \S+`
+ - First Hudi frame is typically `HoodieSparkSqlWriter$.write` at a **suspiciously early line number**
+- **Symptom:** Fails immediately on the first write, before any Hudi logic runs.
+- **Root cause:** The Scala binary version, Spark minor version, or Hive metastore version does not match
+ the `hudi-spark{X}-bundle` in use. Hudi bundles are compiled per Spark-and-Scala combination.
+- **Evidence required:** stacktrace · the environment: Spark version, Scala version, bundle artifact,
+ Hive version
+- **Disambiguation:** a `NoSuchMethodError` naming a Scala class is a Scala binary mismatch; one naming a
+ Hadoop or Parquet class is a dependency-version mismatch; `NoClassDefFoundError` for a cloud SDK class
+ is a missing bundle rather than a wrong one. All three resolve to "match the bundle", but naming which
+ one saves a cycle.
+- **Mitigation ladder:**
+ 1. Match the bundle exactly — `hudi-spark3.4-bundle_2.12` for Spark 3.4 with Scala 2.12. Check the
+ `spark-avro` Scala suffix too.
+ 2. Remove duplicate or conflicting jars. Do not mix `--packages` and `--jars` for the same artifact.
+ 3. On managed Spark services, prefer the platform's own Hudi integration flag over hand-managed jars,
+ and set `spark.serializer` to the Kryo serializer.
+ 4. On vendor Hive older than 2.3, upgrade Hive or move to a Spark version whose bundle matches.
+- **Governing configs:** not Hudi configs — `spark.serializer`,
+ `spark.sql.hive.convertMetastoreParquet`, and bundle selection
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** #1977, #8368, #5765, #6297
+
+---
+
+## HUDI-RCA-036 — Source file deleted after the incremental source tracked it
+
+- **Category:** `environment` (upstream) · **Discoverability:** `needs-expertise`
+- **Signature** (innermost cause):
+ - `java\.io\.FileNotFoundException: No such file or directory: \S+`, where the missing path is under the
+ **source** location
+ - Wrapped in `HoodieUpsertException: Failed to upsert for commit time \d+`
+- **Symptom:** Ingestion fails on every sync after upstream deleted or lifecycle-expired a file the source
+ had already registered.
+- **Root cause:** A cloud-object incremental source records the files it intends to consume, and Hudi
+ expects every tracked file to exist. Upstream deletion between registration and consumption breaks that
+ contract.
+- **Evidence required:** stacktrace · writer props (source config) · the source bucket's lifecycle config
+- **Disambiguation — critical, because the sibling pattern has nearly opposite mitigations:** the missing
+ path is under the **source**, not under the table's `.hoodie/`. If the path is under the table and ends
+ in `.log.<n>`, it is a deltacommit that raced an unfinished compaction, which is destructive to repair
+ and requires L3 with a human — see HUDI-RCA-025's disambiguation note. **Do not apply the file-skip flag
+ below to a missing file under the table**; skipping table files hides real corruption.
+- **Mitigation ladder:**
+ 1. Enable the file-existence bypass so missing **source** files are skipped rather than fatal:
+ `hoodie.streamer.source.s3incr.check.file.exists=true`.
+ 2. If it still fails on a prefix you know is gone, add a relative-path skip through
+ `hoodie.streamer.source.cloud.data.ignore.relpath.substring`.
+ 3. Fix the real problem upstream: lifecycle rules or cleanup jobs deleting source objects faster than
+ ingestion consumes them.
+- **Governing configs:** `hoodie.streamer.source.s3incr.check.file.exists`,
+ `hoodie.streamer.source.cloud.data.ignore.relpath.substring`
+- **`fixedInVersion`:** not a defect · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** false
+- **Provenance:** none public with this signature
+
+---
+
+# Group E — not a failure
+
+The right answer is to do nothing. These rows exist because the natural reaction — change a config,
+restart, reset a checkpoint — is actively harmful, and because nobody files a public issue about
+something that turned out to be correct. Expect these to be **under-represented everywhere except here.**
+
+---
+
+## HUDI-RCA-037 — Cleaner appears not to run (usually correct behaviour)
+
+- **Category:** `not-a-failure` · **Discoverability:** **`silent`** — there is no error, just an absence
+- **Signature:** **No exception.** An absence of recent `.clean` instants in `.hoodie/`, possibly
+ alongside a growing active timeline.
+- **Symptom:** "The cleaner isn't running." Usually it is, and it is finding nothing to do.
+- **Root cause:** Most commonly, there is genuinely nothing to clean. A write that creates a **new file
+ group** leaves no older version to remove, and Hudi also retains one prior version as a buffer, so a
+ file group with two base files still has nothing to clean.
+- **Evidence required:** **health-checker output** (`--checks cleaner,archival`) · commit metadata ·
+ writer props (cleaner config)
+- **Disambiguation — the decisive, cheap check:** in recent commit metadata compare `numWrites` and
+ `numInserts` per file. **Equal means that write created a new file group**, so there is no older version
+ to clean and the cleaner is working correctly. Different means the file group has multiple versions —
+ but the one-version buffer still applies. Also read `earliestCommitToRetain` from the newest `.clean`:
+ archival will not archive anything after that timestamp, so a stale clean fully explains a growing
+ timeline without the cleaner being broken.
+- **Mitigation ladder:**
+ 1. **Confirm it is benign before changing anything.** Most of these are non-incidents, and changing
+ cleaner retention in response is how a non-incident becomes an incident.
+ 2. Recognize the legitimate causes: append-only or immutable data (no updates means no new versions); a
+ recent retention change, which produces a quiet window equal to the difference; clustering having
+ just consolidated many file groups into few; or a MOR table whose file groups writers only append to
+ — the cleaner cannot clean a group that compaction has not yet given a new version.
+ 3. If clean genuinely is not running **and** a compaction or rollback is stuck, that is the real problem —
+ work ladder **L4**, not the cleaner config.
+ 4. Note that **incremental clean can permanently miss partitions** that received no new ingests, leaving
+ old file versions forever. Periodically run a **full** clean rather than incremental.
+- **Governing configs:** `hoodie.cleaner.policy`, `hoodie.cleaner.commits.retained`,
+ `hoodie.cleaner.hours.retained`, `hoodie.clean.max.commits`, `hoodie.clean.automatic`,
+ `hoodie.clean.trigger.strategy`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — the `cleaner` health check distinguishes
+ "behind" from "nothing to do"
+- **`alsoAppliesWhenHealthy`:** **true** by definition — this row *is* the healthy case
+- **Provenance:** no canonical public issue — consistent with a pattern whose correct resolution is
+ "nothing is wrong", which nobody files
+
+---
+
+## HUDI-RCA-038 — Source has no new data
+
+- **Category:** `not-a-failure` · **Discoverability:** **`silent`** — nothing is wrong and nothing is
+ happening, which is the hardest state to reason about
+- **Signature:** Repeated log lines reporting **0 messages**, with latest and committed offsets equal, and
+ **no ERROR-level lines in the same window.**
+- **Symptom:** A stream appears stuck. No exceptions, no restarts, no progress.
+- **Root cause — two cases, and they must be distinguished before acting:**
+ - The source genuinely has no new data. Nothing is wrong.
+ - The stream is wedged in a bootstrap or initialization state and will never consume even when data
+ arrives.
+- **Evidence required:** driver log **at INFO level, not just ERROR** · source offsets · the stream's
+ lifecycle state
+- **Disambiguation:** confirm there are **no ERROR lines** in the window first — if there are, that is the
+ real problem and this row does not apply. Then compare latest against committed offsets: equal and both
+ advancing over time is healthy; equal and **frozen** while the source is known to be receiving data is a
+ wedge. **The "0 messages" log alone is not proof of a wedge.**
+- **Mitigation ladder:**
+ 1. Healthy case: **do nothing.**
+ 2. Wedged case: restart the driver. The stream resumes from the last committed offset checkpointed in
+ `.hoodie/`, so no data is lost; the in-flight micro-batch is dropped and re-read idempotently. Expect
+ a short ingestion gap.
+ 3. If restarting does not clear it, the stream never left bootstrap — recreate the flow rather than
+ restarting repeatedly.
+- **Governing configs:** `hoodie.streamer.source.kafka.topic`, source-specific offset configs
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** no
+- **`alsoAppliesWhenHealthy`:** **true** by definition
+- **Provenance:** none public — nobody files "my job did nothing and that was correct"
+
+---
+
+## HUDI-RCA-039 — Spurious data files detected at commit
+
+- **Category:** `not-a-failure` (a transient `defect` whose correct response is to wait) ·
+ **Discoverability:** `needs-expertise` — a frightening message that is usually benign
+- **Signature** (innermost cause):
+ - `org\.apache\.hudi\.exception\.HoodieException: Failing commit \d+ due to presence of spurious data files in addition to markers`
+- **Symptom:** A clustering or write commit aborts. The next run usually succeeds.
+- **Root cause:** Marker-based reconciliation at commit found more data files on storage than the markers
+ account for — typically output from a retried task attempt. The guard is doing its job.
+- **Evidence required:** stacktrace · whether the **next run** succeeded
+- **Disambiguation:** this is benign **only if it does not recur**. If it fires on three or more
+ consecutive runs, stop treating it as transient and investigate the file groups involved as
+ HUDI-RCA-028 (two base files for one file slice), which is a genuine correctness problem.
+- **Mitigation ladder:**
+ 1. **Do nothing.** Verify that the next run succeeds. This is the whole recommendation and it is correct
+ far more often than it feels.
+ 2. Escalate only if it recurs across three or more consecutive runs.
+ 3. If it does recur, treat it as HUDI-RCA-028 and investigate the file groups involved.
+- **Governing configs:** `hoodie.write.markers.type`, `spark.speculation`
+- **`fixedInVersion`:** n/a · **`preflightGuardExists`:** yes — this message *is* the guard firing
+- **`alsoAppliesWhenHealthy`:** **true** by definition
+- **Provenance:** none public — nobody files a bug for something that fixed itself
+
+---
+
+## Coverage notes
+
+**Flink.** These recipes come from users running Spark, because that is where the evidence came from. The
+core and table-layer patterns — schema (001, 003, 004, 030), keys (002, 005), timeline and table services
+(018, 020, 025, 027, 033, 034, 037), and the metadata table (026, 029) — describe table-layer mechanisms
+that apply to Flink deployments too, though the surrounding frames and the engine-side ladder steps will
+differ. Flink-specific failure modes (checkpoint tolerance, bucket fileID collisions on parallelism
+change, autoscaling interactions) are **not** covered here and will deepen over time.
+
+**Not covered, deliberately.** Deeply managed-platform-specific failures with no open-source analogue;
+Spark-side tuning that belongs in Spark's own documentation rather than Hudi's; and patterns whose
+diagnosis requires forensic access an open-source user cannot reproduce.
+
+**Adding a row.** Give it the next free ID; never reuse or renumber. Write the signature against the
+innermost `Caused by`. If the signature collides with an existing row, add a line to the disambiguation
+index rather than relying on prose inside the row. If the only remedy is destructive, copy the
+explain-only block from HUDI-RCA-025 verbatim.
diff --git a/hudi-ai-operator/skills/hudi-rca/references/mitigation-ladders.md b/hudi-ai-operator/skills/hudi-rca/references/mitigation-ladders.md
new file mode 100644
index 000000000000..c062f3e4c33b
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/references/mitigation-ladders.md
@@ -0,0 +1,404 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Mitigation ladders
+
+Thirteen operational procedures, each ordered. They are not tied to one failure signature — they are
+the discipline that makes the difference between diagnosing a Hudi failure and making it worse.
+`failure-catalog.md` rows reference them by number.
+
+Several of these say **do nothing**, **do not restart**, or **turn the number down**. Those steps are
+not hedges. They are the steps most often got wrong, and the reason is usually that the obvious lever
+points the other way.
+
+**The single most expensive mistake available is restarting before capturing evidence.** That is L9,
+and it earns its place by frequency.
+
+---
+
+## L1 — Locate the real exception in a multi-table writer
+
+When one driver writes many tables, the trace you find first is often not the trace you need.
+
+1. Search driver logs for `ERROR`, then search **within those results** for the table name. Keep
+ scrolling — one polling cycle can contain several tables' traces interleaved.
+2. **Save the full trace somewhere durable the moment you find it.** Re-finding it is expensive, and
+ log rotation does not wait.
+3. If you cannot attribute the failure to a table, temporarily move that table to a **single-table**
+ writer. One table per job makes the trace unambiguous. Move it back afterwards.
+4. If the first pass (ERROR only) returns nothing, widen to INFO and WARN. Some stalls produce **no
+ ERROR line at all** — see L8 and HUDI-RCA-038.
+5. Only then start root-causing.
+
+---
+
+## L2 — Classify a Spark OOM before reacting
+
+An OOM is not one failure mode. These six have genuinely different remedies, and reaching for heap
+first is right in only one of them.
+
+1. **Container OOM-kill (exit 137), Java heap fine** → native or off-heap memory, not heap. Raise
+ `spark.executor.memoryOverheadFactor` (commonly to 0.35, up to around 0.5). Suspect compression
+ codecs and shuffle transfer.
+2. **JVM heap OOM (exit 52)** → a heap dump is usually written automatically. **Get it.** This is a
+ code or data-shape problem, not a sizing problem: memory buys time, not a fix.
+3. **`OutOfDirectMemoryError` / `Direct buffer memory`** → raise overhead and direct-buffer memory. As
+ an emergency measure only, stop Spark preferring direct buffers for shuffle transfer, at a
+ throughput cost. **Do not leave that on.**
+4. **Evicted mid-stage with `FetchFailedException` on shuffle files** → **this is not memory at all.**
+ It is **ephemeral local storage**. Evicted executors delete their shuffle files, surviving tasks
+ fail to fetch, Spark retries the whole stage, writes more shuffle, and evicts more executors. Raise
+ the executors' local-disk request and limit. **This does not show up in the Spark UI** — check pod
+ and node events instead. Treating a FetchFailed as a memory problem is a wasted tuning cycle, and
+ it is the most commonly misread entry in this list.
+5. **Driver OOM requesting a large array** → the JVM cannot allocate a single contiguous array above
+ roughly 2 GB. This is a **structural ceiling**; more heap will not help. See HUDI-RCA-021.
+6. **A few partitions consistently OOM while others are fine** → skew, or large rows. Check per-record
+ size in the stage detail. Above roughly 1 MB average, lower
+ `spark.shuffle.spill.numElementsForceSpillThreshold`.
+7. **GC pressure rather than hard OOM** → **reduce** `spark.memory.fraction` and
+ `spark.memory.storageFraction`. Counter-intuitive: giving Spark's managed regions *less* leaves
+ more heap for user objects. Confirm G1GC and enable GC logging before tuning further.
+8. Check for a pathological CPU-to-memory ratio. Powerful CPUs paired with small memory cause this
+ repeatedly, and no Hudi config will fix it.
+
+---
+
+## L3 — Safe-deletion protocol
+
+**Never delete a file from a Hudi table outside this envelope.** Nine steps, in order, no exceptions.
+
+1. Inform stakeholders.
+2. **Pause the writer.**
+3. **Take a savepoint.**
+4. Verify the deletion premise with **two independent checks** — for example, the file is untracked in
+ commit metadata **and** its instant is named by a completed rollback. One check is not enough.
+5. **Copy the files you intend to delete to a backup prefix.**
+6. Delete.
+7. Validate: re-run the failing operation, or the validator.
+8. Resume the writer.
+9. After a few successful commits, pause, **delete the savepoint**, resume.
+
+**If any verification is ambiguous, stop.** Deleting a non-orphan corrupts the table silently — there
+is no error, and you will find out later from query results. The ambiguity is the signal, not an
+obstacle to work around.
+
+A human drives this procedure. An agent may explain it and point at the evidence that would satisfy
+step 4 — but **must not perform, script, sequence, or enumerate the deletion**, including naming the
+files it believes are implicated. A file list is the deletion, one copy-paste removed; the whole
+point of step 4 is that the human derives it from two independent verifications. The right answer to
+"just tell me which files" is to restate what step 4 requires. See the explain-only block on
+HUDI-RCA-025, which this must not contradict.
+
+---
+
+## L4 — Decide whether a stuck table service is actually stuck
+
+Diagnose in this order. **Each link gates the next**, so fixing a later link while an earlier one is
+blocked accomplishes nothing.
+
+1. **Is archival healthy?** The newest file in `.hoodie/archived/` should be recent — within about an
+ hour on an active table. Look for a successful-archive log line and the absence of archive
+ failures. If the active timeline holds fewer than roughly 1000 instants, archival is almost
+ certainly not your problem.
+2. **If archival is behind, look at clean.** Read `earliestCommitToRetain` from the newest `.clean`.
+ **Archival will not archive anything after that timestamp**, so a stale clean fully explains a
+ growing timeline. This is why tuning archival configs in response to a growing timeline so often
+ does nothing — the cleaner is holding the line.
+3. **If clean is behind, look at compaction.** The cleaner can only clean a file group that has more
+ than one version. On MOR, new versions appear when compaction runs. No compaction means no new
+ versions, which means nothing to clean, which means nothing archivable.
+4. **If compaction is behind, look for a blocker.** An inflight commit or a long-running rollback on
+ the active timeline blocks everything after it. Find the culprit instant. If it is a delta commit
+ whose Spark job is hung, cancel it so it can be rolled back. **If the rollback itself is slow, read
+ HUDI-RCA-021 before restarting anything** — restarting makes a listing-based rollback strictly
+ worse.
+
+ **Check for a pending clustering plan before anything else in this step** — a
+ `<instant>.replacecommit.requested` or `.replacecommit.inflight` with no completed `.replacecommit`.
+ It is the most likely blocker, because clustering is optional: a user enables it, something goes
+ wrong or they change their mind, they turn it off, and the plan it already scheduled stays on the
+ timeline forever. **Nothing else rolls it back** — not the ingestion writer, not the lazy-cleaning
+ rollback of failed writes, not the cleaner. Only a clustering run does. The remedy is to **re-enable
+ clustering with the same scheduling mode that created the plan** (`hoodie.clustering.inline=true` or
+ `hoodie.clustering.async.enabled=true`), let the pending plan execute to a completed
+ `.replacecommit`, and only then disable it again if that is the intent. The alternative is an
+ explicit unschedule through hudi-cli, which is also a human decision. A pending compaction plan on
+ MOR behaves the same way if compaction was turned off after scheduling it; re-enable, let it finish,
+ then decide. Never remove the `.requested` or `.inflight` file by hand.
+
+ Keep the two questions separate. **The RCA is "a long-pending or leftover clustering plan is holding
+ the timeline"** — that is the finding, and it is complete once the instant is named. **How to get rid
+ of a clustering that fails when re-enabled is a different problem**: an unbounded plan is
+ HUDI-RCA-022, a schema or file-group problem inside the plan is its own investigation. Report the
+ blocker, recommend re-enabling, and if the re-enabled run fails, start a new diagnosis on that
+ failure rather than stretching this one.
+5. Compaction also cannot touch a file group a writer is actively writing to. A file group that looks
+ permanently skipped may simply be hot.
+
+`HoodieTableHealthChecker --checks archival,cleaner,compaction,mdt-compaction --output JSON` walks
+this chain for you. Prefer it to manual listing.
+
+---
+
+## L5 — Investigate a metadata-table / filesystem divergence
+
+1. **Classify the direction per table** before theorising. One validator run commonly covers several
+ tables, and two can diverge in **opposite directions** from the same defect.
+2. **Archive the timelines before mitigating.** Both remedies overwrite the diverged state; once you
+ rebuild, the evidence is gone and the question becomes unanswerable.
+3. **Scan by instant, not by filename.** Cleans and rollbacks name the instant they acted on, never
+ individual files. The instant needle has by far the highest yield.
+4. **Do not `grep` timeline Avro files.** They contain NUL bytes, and some `grep` implementations
+ **silently return no match** for content after a NUL on the same line. Use a byte-oriented scanner.
+ A rollback that explained a real divergence was found by a NUL-safe tool and **missed by system
+ `grep -a` on the same file** — the false negative looked exactly like a true negative.
+5. **A scan that prints "no hits" on an empty needle file proves nothing.** Assert your needle file is
+ non-empty before believing a negative result.
+6. **Decode Avro records properly rather than running `strings`.** `strings` cannot render an empty
+ array or an integer, so exactly the two fields that distinguish "the rollback failed to delete
+ these files" from "it never saw them" — an empty rollback-request list, a zero deleted-file count —
+ are invisible to it.
+7. **Measure your actual timeline coverage** before concluding that no event touched these files.
+ Count completed commit-family **instants** per month, not state files: each instant has up to three
+ state files, and rollbacks linger on the active timeline for months, so a raw file count can read
+ as nine events where the truth is three rollbacks and zero commits.
+8. Instants are **17 digits** (`yyyyMMddHHmmssSSS`). A pattern that captures 13 will make every later
+ search miss.
+9. **A negative is a finding.** "No instant across N timeline entries names these files, and coverage
+ spans the divergent instant" is a conclusion, not a dead end.
+
+---
+
+## L6 — Check that a correctness signal is even real before investigating
+
+Three cheap checks. Each has, on a real incident, been the reason an expensive investigation was
+unwarranted.
+
+1. **One table, or everything at once?** Several unrelated tables or jobs failing the same check
+ within minutes usually means a change to the **check** — a rule rename, a threshold edit, a deploy
+ — not simultaneous data corruption. Correlate against recent validator and alert configuration
+ changes first.
+2. **Is the signal fresh?** If your validator pushes to a store that retains the last value
+ indefinitely, a validator that died days ago presents as a steady, confident failure. Check the
+ **job's** last successful run, not just the metric. If the last success is older than about two
+ schedule intervals, no per-table conclusion drawn from it is safe. **A dead data-correctness
+ validator is a higher-severity problem than the divergence it was going to report.**
+3. **Is the condition new?** A value that has been non-zero for weeks with no step change is a
+ standing backlog, not an incident.
+
+---
+
+## L7 — Decide whether "no clean commits" is a problem
+
+The decisive, cheap check: in recent commit metadata, **if `numWrites == numInserts` for a file, that
+write created a new file group**, so there is no older version to clean — the cleaner is working
+correctly. Hudi also retains one prior version as a buffer, so a file group with two base files still
+has nothing to clean.
+
+Confirm this before changing any cleaner configuration. Most reports of "the cleaner is not running"
+are non-incidents, and changing retention in response is how a non-incident becomes one. Full
+treatment in HUDI-RCA-037.
+
+---
+
+## L8 — Diagnose a stalled table with no visible exception
+
+When a job looks healthy — no exceptions, no restarts — and nothing is ingested:
+
+1. **Compare the committed checkpoint against the source's current position.** Ahead of source latest
+ means a recreated or aged-out source (HUDI-RCA-012). Far behind means genuine lag (L10). **Equal
+ and frozen** means HUDI-RCA-038.
+2. **Check for a stuck inflight instant** (`.commit.inflight`, `.replacecommit.inflight`,
+ `.rollback.inflight`) and note its age. For a rollback or clustering instant, note the **plan file
+ size** — a multi-MB plan is itself the diagnosis.
+3. **Check that the driver is healthy in a way your health check can actually see.** **A driver can
+ pass a liveness probe and do no useful work** — for instance after an OOM cleanup closed its shared
+ filesystem client, so every subsequent operation fails instantly while the health endpoint answers
+ fine and the restart count stays at zero. Never infer health from a probe alone. Verify that
+ commits are landing, **per table**.
+4. **Beware per-table blind spots.** On a multi-table job, "some tables are committing" does not
+ establish that the *affected* table is. Check the table in question specifically.
+
+---
+
+## L9 — Capture evidence before restarting
+
+**A restart is the most common and most expensive mistake in this corpus.**
+
+1. If a driver is wedged or crash-looping, **take the heap dump first.** Several failures were only
+ ever root-caused because dumps were captured while the wedge was live. In more than one case,
+ reading logs produced a plausible and **wrong** hypothesis that only a heap dump falsified.
+2. Capture a busy-thread dump. If nearly all the hottest threads are GC threads, you have a memory
+ problem, not a CPU problem, and adding cores will not help.
+3. To attribute a leak to a table, string-scan the heap dump for storage URIs and rank by frequency.
+ The dominant table is usually the overwhelming majority of the leak.
+4. **For a rollback or clustering wedge, a restart actively harms you** — it abandons in-progress work
+ and re-requests a larger unit. A rollback plan file that grows across successive attempts is the
+ proof that this is happening.
+5. Write down, on the ticket, which hypotheses you have **falsified**. Investigations repeatedly lose
+ hours to a hypothesis that was already disproved but never retracted.
+
+---
+
+## L10 — Read lag metrics correctly during a backfill
+
+Lag computed as (source head position − committed checkpoint position) **does not decrease smoothly**
+when the source is organized into a few very large units of work. If a backfill pages through one
+large source instant in budget-limited chunks, the file cursor advances while the instant does not, so
+lag stays pinned and then drops in a step when that instant drains. **A staircase, not a slope, by
+design.**
+
+Before escalating "lag isn't moving":
+
+1. Confirm commits are landing and carrying real volume.
+2. Confirm which component of the checkpoint the lag metric actually tracks.
+3. Check whether the job is budget-pinned by a per-sync read limit. If so, the ceiling is the limit,
+ not the cluster: raising the read limit (and reverting it afterwards) is the lever, and **adding
+ executors is not**.
+4. Check that your executor-scaling policy can actually reach the backlog. A conservative dynamic
+ allocation ratio can target far fewer executors than the work justifies and never scale up, leaving
+ most of the cluster idle through an entire backfill.
+
+---
+
+## L11 — Keep batch-size settings coupled to the memory they assume
+
+A per-batch limit — messages per poll, records per request, bytes per sync — is valid **only for the
+heap it was sized against**. A driver shrunk for cost while its batch override, sized for a much
+larger driver, was left in place produces an unbreakable OOM loop on the next cold start.
+
+1. Record, next to any batch-size override, the driver memory it assumes.
+2. When you change driver or executor memory, **re-derive every dependent batch-size override.**
+3. If the two live in different files or systems, treat that as a known hazard and check both on every
+ resize.
+4. **Size for the cold start, not the steady state.** A cold start meets the full accumulated backlog
+ in one batch.
+
+---
+
+## L12 — Verify that a config you set is actually in effect
+
+Several failures trace to a config that was set and silently ignored — a misspelled key, segments in
+the wrong order, or a value overridden rather than merged by a layered configuration system.
+
+1. Read the **effective** configuration from the running job — the resolved write-config dump in the
+ driver log, or live JVM inspection — rather than the file you edited.
+2. If your configuration system **overrides rather than merges** list-valued settings (transformer
+ chains being the classic case), re-list every value you still want, not just the new one. Adding
+ one transformer can silently drop the one that was already there.
+3. **A misspelled Hudi config key produces no error.** The library default simply applies. Treat "I
+ set it and nothing changed" as evidence that **the key is wrong**, not that the setting does not
+ work.
+4. Beware empty maps or dictionaries in layered configuration formats — some merge implementations
+ null out an entire block.
+
+---
+
+## L13 — Triage an unmatched stack trace into a failure family
+
+When no catalog row matches, classify from the top-level exception class and the first two or three
+`Caused by` frames. **This is a starting hint, not an action** — it narrows where to look, it does not
+attribute a cause.
+
+| Family | Heuristic |
+|---|---|
+| Meta-sync failure | Frames in `org.apache.hudi.sync.*` (catalog, Iceberg, or Glue sync) |
+| Write failure | `HoodieWriteException` / `HoodieUpsertException` / `HoodieInsertException` in the top class or the caused-by chain |
+| Write conflict | Concurrency-control exceptions — check the lock provider and concurrency mode |
+| Transform failure | Frames in your own or a platform transformer package. Check **whose** code it really is before blaming the platform |
+| Schema compatibility | Avro schema-mismatch frames, `IncompatibleSchemaException` → HUDI-RCA-003 |
+| Schema fetch | Schema-registry client frames → HUDI-RCA-011 |
+| Read-from-source | Source consumer, object lister, or CDC connector frames at the top of the chain |
+| Connection timeout | `SocketTimeoutException` or a connection-pool timeout → HUDI-RCA-031, HUDI-RCA-024 |
+
+**The governing rule: if even one frame is in a Hudi or user-code package, look there first.** A trace
+consisting purely of engine internals is the only case where a generic family heuristic is the best
+available answer — and a bare `NullPointerException` with no Hudi or user frame anywhere is
+specifically **not attributable**. For that case, say so, and note that
+`hoodie.streamer.row.throw.explicit.exceptions=true` will surface the underlying exception on a
+re-run. It costs performance; it is a diagnostic, not a setting, so turn it back off afterwards.
+
+---
+
+## L14 — A file Hudi expects is not on storage: triage in this order
+
+`FileNotFoundException`, `FileID ... does not exist`, or a reader failing on a path under the table all
+mean the same thing at first sight and have at least four different causes with different remedies.
+**Run this ladder before attributing to HUDI-RCA-025.** Three facts decide the branch, so ask for them
+in the first reply: **is the metadata table enabled**, **is the failure on the write path or the read
+path**, and **how many commits ago did it start**.
+
+1. **Metadata table enabled → suspect the listing first.** The most common reason something tries to
+ read a file that is not there is that the metadata table's `files` listing and the actual storage
+ listing have drifted (HUDI-RCA-026). Confirm it before anything else: run the health checker's
+ `mdt-sync` check, or `HoodieMetadataTableValidator` for the affected partition, and compare the two
+ listings for the file in question.
+
+ **If the user is blocked and the table is high-severity**, the fastest safe unblock is:
+ a. **Back up the table first** if its size makes that reasonable — copy the base path, `.hoodie`
+ included. The investigation into *how* the metadata table drifted continues on the copy; without
+ it, the next step destroys the evidence (see L5 step 2).
+ b. **Disable the metadata table on the writer** (`hoodie.metadata.enable=false`) and resume
+ ingestion from storage listings. Say plainly what this does: with
+ `hoodie.metadata.auto.delete.partitions` at its default, the writer deletes the metadata table and
+ **every index in it** (record index, column stats, bloom filters, secondary indexes). Stop every
+ other writer first, and make sure readers are not configured to require the metadata table.
+ c. **Re-enable it after a few clean commits.** Hudi re-bootstraps the metadata table from storage
+ on the next write with it enabled; indexes come back as the writer properties enable them.
+ This is a human-gated change (it drops indexes), but it is reversible and it does not touch data
+ files, which is why it is the unblock and hand-deletion is not.
+
+2. **Read path, long-running or incremental query, a `.clean` inside the query window → cleaner
+ boundary, not a defect.** The query planned its file slices at start, ran for a while, the cleaner
+ completed during that window and removed an older file version the plan still referenced. Signature:
+ read-side frames (the Parquet reader, `FileScanRDD`, the file-group reader), the missing file is
+ **not** the latest version of its file group, and the timeline has a completed `.clean` with a
+ completion time between the query's start and its failure. Remedy: raise cleaner retention past the
+ longest query the table serves — `hoodie.clean.commits.retained`, or switch to
+ `KEEP_LATEST_BY_HOURS` with `hoodie.clean.hours.retained` so the guarantee is in time rather than in
+ commits. For incremental queries, also make sure the consumer's start instant cannot fall behind
+ what the cleaner retains. Nothing to delete, nothing to rebuild.
+
+3. **Metadata table not enabled, write path, deterministic on the same file ID → a dangling file
+ from an incomplete rollback (HUDI-RCA-025).** There is no shortcut on this branch; the only honest
+ path is to get to the bottom of the specific file:
+ a. Find the rollback that targeted the failed commit (the `.rollback` whose `commitsRollback` names
+ it; the driver log line `Rollback of Commits [<instant>] is complete`).
+ b. **Deserialize both halves properly**: the plan in `<instant>.rollback.requested`
+ (`HoodieRollbackPlan`) and the result in `<instant>.rollback` (`HoodieRollbackMetadata`, with
+ per-partition `successDeleteFiles` and `failedDeleteFiles`). Use `hudi-cli` `show rollback`, the
+ timeline inspector, or an Avro decoder — never `grep` or `strings` on these files (L5 steps 4-6).
+ c. Decide which case the file is in: **named in the plan and in the deleted list but still on
+ storage** (the delete did not take, or the store is eventually consistent); **named in the plan
+ but in `failedDeleteFiles` or in neither list** (the rollback did not finish its work); **not in
+ the plan at all** (the rollback never saw it: a task-attempt or marker-less file).
+ d. Report the case and the evidence. The agent may do all of this read-only. Removing the file
+ remains L3, human-driven, and the explain-only block on HUDI-RCA-025 still applies in full.
+
+4. **Recent onset — the last one or two commits, or within the last day → restore is available and
+ usually fastest.** Find the last known-good commit: run incremental or as-of queries commit by commit
+ backwards until the read succeeds, or run the validator as of each instant. Then restore to it
+ (`savepoint` + `restore`, or the `rollback_to_instant` procedure). This rewrites the timeline and
+ discards every commit after that point, so count those commits and say they must be re-ingested; if
+ every write since the onset failed anyway, it discards nothing. Human-gated, L3-grade.
+
+**Do not skip to step 3.** On a table with the metadata table enabled, step 1 is the likely answer and
+is reversible; on a reader, step 2 is a retention setting; and step 4 is often cheaper than either
+investigation when the damage is recent. Step 3 is where you end up when the others are ruled out.
+
+---
diff --git a/hudi-ai-operator/skills/hudi-rca/references/test-corpus.md b/hudi-ai-operator/skills/hudi-rca/references/test-corpus.md
new file mode 100644
index 000000000000..a3cdf27296c6
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/references/test-corpus.md
@@ -0,0 +1,89 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Test corpus
+
+Regression cases drawn from public `apache/hudi` issues. Each case is a stacktrace excerpt, the
+catalog row that should match it, and the category that should be reported. This is what keeps the
+catalog honest: change a signature regex, re-run these, and see what you broke.
+
+Every issue number here was fetched from the public tracker. **Do not add a case without a public
+issue number you have verified.** Cases invented to exercise a regex belong in a unit-test fixture,
+not here — the value of this file is that every row is a failure a real user actually reported.
+
+## How to use it
+
+For each case, give the agent **only the excerpt** — not the expected columns — and check three things:
+
+1. Did it walk to the **innermost** `Caused by` rather than classifying off the wrapper?
+2. Did it return the expected pattern ID, or correctly return `unclassified`?
+3. Did it return the expected category, and state which evidence rung it was on?
+
+A case that expects `unclassified` is **passing when the agent declines to attribute**. Those cases
+(16, 17, and the no-stacktrace rows) exist specifically to catch an agent that guesses confidently.
+Guessing on them is a regression even if the guess happens to be right.
+
+## Cases
+
+| # | Issue | Stacktrace excerpt | Expected pattern | Expected category |
+|---|---|---|---|---|
+| 1 | 11335 | `Caused by: org.apache.avro.AvroTypeException: Cannot encode decimal with precision 14 as max precision 13`<br>` at org.apache.avro.Conversions$DecimalConversion.validate(Conversions.java:140)`<br>` at org.apache.hudi.avro.HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType(HoodieAvroUtils.java:1077)`<br>` at org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordWithNewSchema(HoodieAvroUtils.java:873)` | HUDI-RCA-001 | `input/config` |
+| 2 | 8040 | `org.apache.parquet.io.InvalidRecordException: Parquet/Avro schema mismatch: Avro field 'deleted_column_name' not found`<br>` at org.apache.parquet.avro.AvroRecordConverter.getAvroField(AvroRecordConverter.java:221)` | HUDI-RCA-003 (column dropped by source) | `input/config` |
+| 3 | 8040 | `org.apache.hudi.exception.HoodieException: Exception when reading log file`<br>` at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:352)`<br>` at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:110)`<br>` at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:198)` | HUDI-RCA-001, **MOR deferred variant** — agent must sa [...]
+| 4 | 8020 | `org.apache.avro.AvroTypeException: Cannot encode decimal with precision 4 as max precision 2` | HUDI-RCA-001 | `input/config` |
+| 5 | 3759 | `Caused by: org.apache.hudi.exception.HoodieKeyException: recordKey value: "null" for field: "my_primary_key_column_here" cannot be null or empty.` | HUDI-RCA-002 | `input/config` |
+| 6 | 11776 | `org.apache.hudi.exception.HoodieKeyException: recordKey value: "null" for field: "ID" cannot be null or empty.` | HUDI-RCA-002, **case-sensitivity variant** — agent should raise case before nulls | `input/config` |
+| 7 | 10233 | `org.apache.hudi.exception.HoodieException: ts(Part -ts) field not found in record. Acceptable fields were :[c1, c2, c3, c4, c5]`<br>` at org.apache.hudi.avro.HoodieAvroUtils.getNestedFieldVal(HoodieAvroUtils.java:601)` | HUDI-RCA-002, **Streamer `ts` default** — agent must note the user never configured `ts` | `input/config` |
+| 8 | 2408 | `: java.lang.OutOfMemoryError`<br>` at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)`<br>` at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.serializeRecords(HoodieAvroDataBlock.java:114)`<br>` at org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:309)`<br>` at org.apache.hudi.table.HoodieTimelineArchiveLog.archiveIfRequired(HoodieTimelineArchiveLog.java:133)` | HUDI-RCA-015 **archival variant [...]
+| 9 | 1491 | `- java.lang.OutOfMemoryError: Java heap space`<br>`- The java.lang.OutOfMemoryError: GC overhead limit exceeded error`<br>`- java.util.concurrent.TimeoutException: Cannot receive any reply from <host>:41498 in 10000 milliseconds` | HUDI-RCA-015 **merge variant (a)** | `sizing/scale` |
+| 10 | 1977 | `py4j.protocol.Py4JJavaError: An error occurred while calling o155.save.`<br>`: java.lang.NoSuchMethodError: scala.Some.value()Ljava/lang/Object;`<br>` at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:59)` | HUDI-RCA-035 | `environment` |
+| 11 | 7653 | `Exception in thread "main" org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes`<br>` at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102)`<br>` at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79)`<br>` at org.apac [...]
+| 12 | 3731 | `: org.apache.hudi.exception.HoodieLockException: Unable to acquire lock, lock object LockResponse(lockid:255, state:WAITING)`<br>` at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:82)`<br>` at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:64)` | HUDI-RCA-006, lock-acquisition variant | `input/config` |
+| 13 | 3731 | ` at org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:544)`<br>`Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: File already exists: file:/tmp/test_table/.hoodie/20210921151138.commit.requested` | HUDI-RCA-006, timeline-collision variant | `input/config` |
+| 14 | 9213 | `Caused by: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback <base-path> commits 20230716231210452`<br>` at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:783)`<br>` at org.apache.hudi.client.BaseHoodieWriteClient.rollbackFailedWrites(BaseHoodieWriteClient.java:1193)`<br>` at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:151)`<br>` at org.apache.hudi.client.BaseHoodieWriteClient.sta [...]
+| 15 | 9213 | `java.lang.IllegalArgumentException: ALREADY_ACQUIRED`<br>` at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:40)`<br>` at org.apache.hudi.hive.HiveMetastoreBasedLockProvider.acquireLock(HiveMetastoreBasedLockProvider.java:136)`<br>` at org.apache.hudi.client.transaction.lock.LockManager.lock(...)` | HUDI-RCA-006, lock-provider variant | `environment` |
+| 16 | 4230 | `java.lang.RuntimeException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRemoteException: Failed to create marker file /<fileId>-0_0-7-15_20211206094243657.parquet.marker.CREATE`<br>` at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:121)` | HUDI-RCA-031 — **three nested wrappers; agent must reach `HoodieRemoteException`, not stop at `RuntimeException`** | `environment` |
+| 17 | 5484 | `java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient`<br>` at org.apache.hudi.hive.ddl.HMSDDLExecutor.<init>(HMSDDLExecutor.java:69)`<br>` at org.apache.hudi.hive.HoodieHiveClient.<init>(HoodieHiveClient.java:73)`<br>` at org.apache.hudi.hive.HiveSyncTool.initClient(HiveSyncTool.java:95)` | HUDI-RCA-007 | `input/config` |
+| 18 | 11202 | `Caused by: java.util.NoSuchElementException: FileID 1b88b26f-94b1-4bd1-94ad-e919e17183ee-0 of partition path datatype=xxxx/year=2024/month=05/day=07 does not exist.`<br>` at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:161)`<br>` at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.getUpdateHandle(BaseSparkCommitActionExecutor.java:400)` | HUDI-RCA-025 — **EXPLAIN-ONLY. Agent must not propose or script deletion. Must name [...]
+| 19 | 2515 | `java.lang.NullPointerException: null of string of map of union in field extraMetadata of org.apache.hudi.avro.model.HoodieCommitMetadata of union in field hoodieCommitMetadata of org.apache.hudi.avro.model.HoodieArchivedMetaEntry`<br>` at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:145)`<br>` at org.apache.hudi.table.HoodieTimelineArchiveLog.writeToFile(HoodieTimelineArchiveLog.java:361)` | HUDI-RCA-015 archival variant, with HUDI-RCA-010 (null [...]
+| 20 | 5540 | `org.apache.hudi.exception.HoodieException: Commit 20220509105215 failed and rolled-back !`<br>` at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:492)` | **`unclassified`** — content-free wrapper. Agent must return low confidence and request the driver log. See HUDI-RCA-010. | `unclassified` |
+| 21 | 8519 | `Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 14) (<host> executor 1): java.lang.NullPointerException`<br>` at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2304)` | **`unclassified`** — bare NPE, no Hudi frame. Agent should offer L13 and the explicit-exceptions diagnostic, not attribute. | `unclassified` |
+| 22 | 2020 | `java.io.FileNotFoundException` during compaction retry, on a file named in the compaction plan | HUDI-RCA-025 (stale file-slice reference); check `hoodie.filesystem.view.incr.timeline.sync.enable` | `defect` |
+| 23 | 7657 | `java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieRollbackException: Failed to rollback <base-path> commits 20230112132517252`<br>` at org.apache.hudi.async.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:75)`<br>` at org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:605)`<br>plus `Invalid number of file groups for partition:column_stats` | HUDI-RCA-026 — **presents as a rollback failure; the ` [...]
+| 24 | 5777 | *(no stacktrace — duplicate rows for one primary key)* | HUDI-RCA-013 | `input/config` |
+| 25 | 10508 | *(no stacktrace — `ComplexKeyGenerator` encoding changed in 0.14.1; upserts duplicate and deletes silently no-op)* | HUDI-RCA-008 (config/table drift), surfacing as HUDI-RCA-013 | `defect` |
+| 26 | 4154 | *(no stacktrace — `INSERT OVERWRITE` writes parquet but `SELECT` returns stale or empty results)* | HUDI-RCA-009 | `input/config` |
+| 27 | 2123 | *(no stacktrace — timestamps render as `+49134-01-07 05:30:00.000` in the query engine)* | HUDI-RCA-009, timestamp variant | `input/config` |
+
+## What this corpus does not cover
+
+Honest gaps, so nobody mistakes a green run for full coverage:
+
+- **No cases for 18 of the 39 catalog rows.** The rows sourced from operational experience rather than
+ the public tracker — including HUDI-RCA-019, 021, 022, 023, 028, 029, 030, 033, 034, 036 and all
+ three Group E rows — have no public issue to cite, and inventing one would defeat the purpose of
+ this file. They are exercised by reading the row, not by this corpus.
+- **Group E is structurally hard to test this way.** "Do nothing" patterns produce no artifact to
+ paste. The check that matters for them is behavioural: given a cleaner that appears idle, does the
+ agent run the `numWrites == numInserts` check before recommending a config change?
+- **No multi-artifact cases.** Every case here is rung 0 (a stacktrace alone). The rungs that matter
+ most — `hoodie.properties` diffs, health-checker JSON, event logs — are not represented, and cases
+ 11 and 20 are the only ones that test whether the agent *asks* for them.
+- **No Flink cases.** See the coverage note in `failure-catalog.md`.
+
+Contributions that close any of these gaps with a verifiable public issue number are worth more than
+additional cases for rows already covered.
diff --git a/hudi-ai-operator/skills/hudi-rca/references/workflow-catalog.md b/hudi-ai-operator/skills/hudi-rca/references/workflow-catalog.md
new file mode 100644
index 000000000000..92cf9f5b5212
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/references/workflow-catalog.md
@@ -0,0 +1,972 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Hudi write-path workflow catalog
+
+A map from **what a user sees in the Spark UI** to **what Hudi was actually doing**. Use it when
+a stage failed and the root cause is not obvious: identify the phase, then read its governing
+configs and its known failure modes.
+
+Every non-obvious claim cites `file:line` against this checkout. Where a statement is an
+inference rather than something read from source — most notably the exact text Spark renders in
+its UI — it is marked **[inferred]**.
+
+---
+
+## How to read this
+
+### 1. The job description is the strongest signal
+
+Hudi sets a Spark *job description* before most phases and clears it afterwards. On Spark the
+call is:
+
+```java
+javaSparkContext.setJobDescription(String.format("%s:%s", activeModule, activityDescription));
+```
+
+`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:232-234`
+
+So the description is always `<SimpleClassName>:<free text>`, and the free text usually ends with
+the table name. The Spark UI shows the job description in the Jobs tab, and the stages of that
+job inherit it in the "Description" column **[inferred — this is standard Spark UI behaviour,
+not something Hudi controls]**.
+
+Descriptions Hudi emits that are worth memorising:
+
+| Description (prefix:text) | Phase | Source |
+|---|---|---|
+| `…:Tagging: <table>` | index lookup | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseWriteHelper.java:73` |
+| `…:Building workload profile:<table>` | `buildProfile` / `countByKey` | `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/BaseSparkCommitActionExecutor.java:263` |
+| `…:Doing partition and writing data: <table>` | shuffle + write handles | `.../BaseSparkCommitActionExecutor.java:246` |
+| `…:Commit write status collect: <table>` | `collectAsList` of `WriteStatus` | `.../BaseSparkCommitActionExecutor.java:390` |
+| `…:Handling updates which are under clustering: <table>` | clustering update strategy | `.../BaseSparkCommitActionExecutor.java:119` |
+| `…:Getting small files from partitions: <table>` | small-file probe in `UpsertPartitioner` | `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/UpsertPartitioner.java:287` |
+| `…:Obtain key ranges for file slices (range pruning=on): <table>` | bloom-index range load | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/HoodieBloomIndex.java:173` |
+| `…:Compacting file slices: <table>` | compaction execution | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java:129` |
+| `…:Clustering records for <table>` | clustering execution | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/BaseCommitActionExecutor.java:287` |
+| `…:Creating Listing Rollback Plan: <table>` | listing-based rollback plan | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/ListingBasedRollbackStrategy.java:105` |
+| `…:Delete all partially written files: <table>` | marker reconciliation cleanup | `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:847` |
+| `…:Fetching next batch: <table>` | HoodieStreamer source read | `hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:658` |
+
+`clearJobStatus()` sets the description back to `null`
+(`HoodieSparkEngineContext.java:237-239`), so a stage with **no** description is usually either
+a lazily-evaluated RDD whose job was launched outside a `setJobStatus` block, or a phase Hudi
+does not instrument. Treat "no description" as "look at the task count and the shuffle shape
+instead".
+
+### 2. Task count tells you which partitioner ran
+
+Hudi produces a small number of distinctive parallelism shapes. Matching the task count against
+these is usually decisive:
+
+| Task count equals | Phase | Why |
+|---|---|---|
+| Number of input Spark partitions, unchanged | dedupe skipped, or a narrow `mapPartitions` tagging (bucket index) | `hoodie.*.shuffle.parallelism` defaults to `0` = "inherit input parallelism", see below |
+| `hoodie.upsert.shuffle.parallelism` (if set) | dedupe `reduceByKey`, or `countByKey` for the workload profile | `HoodieWriteHelper.java:78-80` |
+| A number with no relation to input or any config | the write stage — task count = **total buckets** from `UpsertPartitioner` | `UpsertPartitioner.java:336-338` (`numPartitions()` returns `totalBuckets`) |
+| Number of file groups in the compaction plan | compaction execution | `HoodieCompactor.java:154` + `HoodieEngineContext.java:75-80` |
+| Number of output file groups in one clustering group | one clustering sub-job | `SparkSortAndSizeExecutionStrategy.java:65-68` |
+| Number of table partitions (capped) | clean plan, listing rollback, small-file probe | see each section |
+| 200 (and nothing else explains it) | `hoodie.finalize.write.parallelism`, default 200 | `HoodieWriteConfig.java:561-563` |
+| 100 (and nothing else explains it) | `hoodie.rollback.parallelism`, default 100 | `HoodieWriteConfig.java:481-483` |
+
+### 3. The parallelism defaults are not what most people think
+
+`hoodie.upsert.shuffle.parallelism`, `hoodie.insert.shuffle.parallelism`,
+`hoodie.bulkinsert.shuffle.parallelism` and `hoodie.delete.shuffle.parallelism` **all default to
+`0`**, not 200:
+
+```java
+public static final ConfigProperty<String> UPSERT_PARALLELISM_VALUE = ConfigProperty
+ .key("hoodie.upsert.shuffle.parallelism")
+ .defaultValue("0")
+```
+
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:454-456`
+(the same shape at `:399-401`, `:413-415`, `:469-471`).
+
+`0` means "deduce from the input":
+
+```java
+protected int deduceShuffleParallelism(I input, int configuredParallelism) {
+ if (configuredParallelism > 0) {
+ return configuredParallelism;
+ }
+ return partitionNumberExtractor.apply(input);
+}
+```
+
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/ParallelismHelper.java:31-40`
+
+The config's own documentation records the history: before 0.13.0 the default was a literal 200;
+from 0.13.0 the default is Spark-deduced parallelism
+(`HoodieWriteConfig.java:458-467`). **Consequence for RCA:** if a user says "I didn't set
+parallelism so it must be 200", that is wrong on any recent version. A 16 000-task shuffle with
+no parallelism config set means the *input* RDD had 16 000 partitions.
+
+### 4. The write stage is bucket-shaped, not partition-shaped
+
+The single most misread stage in a Hudi upsert is the write stage. Its task count is
+`UpsertPartitioner.totalBuckets`, which is `(#file groups receiving updates) + (#insert buckets)`
+— derived from the workload profile, not from any shuffle-parallelism config
+(`UpsertPartitioner.java:130-137`, `:242-256`, `:336-338`). Tuning
+`hoodie.upsert.shuffle.parallelism` will **not** change this stage's task count. Tuning
+`hoodie.parquet.max.file.size` / `hoodie.copyonwrite.insert.split.size` will.
+
+### 5. RDD names
+
+Hudi does not call `RDD.setName()` on its main write-path RDDs anywhere in this checkout, so the
+Spark UI DAG shows the generic operator names (`MapPartitionsRDD`, `ShuffledRDD`,
+`ParallelCollectionRDD`) **[inferred from the absence of `setName` calls; verified by grep over
+`hudi-client`]**. The DAG shape is still informative:
+
+- `ParallelCollectionRDD` at the root of a stage ⇒ the stage was driven from a driver-side list
+ (`context.parallelize(...)`): compaction operations, clean partitions, rollback requests.
+- A `ShuffledRDD` whose parent is a `MapPartitionsRDD` producing `(HoodieFileGroupId, String)`
+ pairs ⇒ bloom-index file-comparison shuffle.
+- `UnionRDD` near the end of a replacecommit ⇒ clustering group results being unioned
+ (`MultipleSparkJobExecutionStrategy.java:134`).
+
+### 6. Walk the phases in order
+
+A Spark job that fails at a given point tells you every phase that already *succeeded*. Use the
+ASCII diagrams below: if "write and handle" is running, tagging and profiling already completed,
+so the problem is in file writing, not index lookup.
+
+---
+
+## 1. UPSERT — Copy-on-Write
+
+### Phase diagram
+
+```
+ driver executors
+ ------ ---------
+ [1] startCommit / requested instant
+ |
+ v
+ [2] dedupe (optional) ------------------------------------> reduceByKey
+ | (shuffle #1)
+ v
+ [3] index tagging ---------------------------------------> varies by index
+ | (shuffle #2, bloom/simple/RLI)
+ v
+ [4] buildProfile <--------- countByKey ------------------ (shuffle #3)
+ |
+ [5] UpsertPartitioner construction
+ | (small-file probe: Spark job over partition paths)
+ v
+ [6] save workload profile -> inflight instant on timeline
+ |
+ v
+ [7] partitionBy(UpsertPartitioner) ----------------------> shuffle #4
+ | then mapPartitionsWithIndex:
+ | merge handles / create handles
+ v
+ [8] WriteStatus collect <------------------------------- collectAsList
+ |
+ [9] finalizeWrite: marker reconciliation
+ |
+ [10] metadata table update (deltacommit on MDT)
+ |
+ [11] saveAsComplete -> .commit on timeline (inside txn lock)
+ |
+ [12] post-commit inline table services (clean / archive / compact / cluster)
+```
+
+### Phase table
+
+| # | Phase | Driver / executor | Spark op | Governing configs | Common failures |
+|---|---|---|---|---|---|
+| 2 | **Dedupe** `HoodieWriteHelper.deduplicateRecords` | executor | `mapToPair` → `reduceByKey(parallelism)` → `map` (`HoodieWriteHelper.java:70-80`) | `hoodie.combine.before.upsert`; parallelism from `hoodie.upsert.shuffle.parallelism` via `combineOnCondition` (`BaseWriteHelper.java:90-94`) | Skew on a hot record key (one reducer task runs forever). Payload/merger `IOException` surfaces as `HoodieException("Error to merge two records…")` (`BaseWriteHelper.java:163-165`). Note the recor [...]
+| 3 | **Index tagging** `HoodieIndex.tagLocation` | mixed | see per-index table below | `hoodie.index.type` and the per-index configs | see per-index table below |
+| 4 | **buildProfile** | driver collects, executors count | `mapToPair(...).countByKey()` (`BaseSparkCommitActionExecutor.java:284-288`) | parallelism inherited from the tagged RDD | `countByKey` returns a `Map` **to the driver**, one entry per `(partitionPath, location)` pair. On a table with very many file groups receiving updates this is a driver-side OOM. This is the classic "driver OOM at the end of tagging" signature. |
+| 5 | **UpsertPartitioner** | driver (plus one small Spark job) | `context.mapToPair(partitionPaths, …, partitionPaths.size())` (`UpsertPartitioner.java:289`) | `hoodie.parquet.small.file.limit` (`HoodieCompactionConfig.java:111`), `hoodie.parquet.max.file.size` (`HoodieStorageConfig.java:40`), `hoodie.copyonwrite.insert.split.size` (`:177`), `hoodie.copyonwrite.insert.auto.split` (`:186`) | Task count here is `partitionPaths.size()` — on a table with tens of thousands of partitions this [...]
+| 6 | **Workload profile → inflight** | driver | none (timeline write) | `hoodie.write.concurrency.*` | `transitionRequestedToInflight` (`BaseCommitActionExecutor.java:164`); failure raises `HoodieCommitException("…unable to save inflight metadata")` (`:166`). |
+| 7 | **Partition + write** `mapPartitionsAsRDD` | executor | `mapToPair` → `partitionBy(partitioner)` **or** `repartitionAndSortWithinPartitions` when `table.requireSortedRecords()` → `mapPartitionsWithIndex` → `flatMap` (`BaseSparkCommitActionExecutor.java:323-353`) | task count = `UpsertPartitioner.numPartitions()` = total buckets. File sizing via `hoodie.parquet.max.file.size`. | **Executor OOM is most often here, not in tagging** — each `UPDATE` bucket opens a `HoodieMergeHandle` th [...]
+| 8 | **WriteStatus collect** | driver | `persist` then `map(WriteStatus::getStat).collectAsList()` (`:359`, `:381`) | `hoodie.write.status.storage.level` (`WRITE_STATUS_STORAGE_LEVEL_VALUE`, `:91`) | One `HoodieWriteStat` per written file arrives on the driver. When the caller constructs `WriteStatus` with `trackSuccessRecords` (`WriteStatus.java:76`), it carries per-record detail and this collect becomes a driver OOM. That flag is set in code, not by a `hoodie.*` property; the two rela [...]
+| 9 | **finalizeWrite / markers** | both | `getInvalidDataPaths` then `deleteInvalidFilesByPartitions` at `config.getFinalizeWriteParallelism()` (`HoodieTable.java:781-782`, `:775`) | `hoodie.finalize.write.parallelism` (default 200, `HoodieWriteConfig.java:561-563`); `hoodie.write.markers.type`; `hoodie.consistency.check.enabled`; `hoodie.fail.job.on.duplicate.data.file.detection` | Reconciliation compares marker-derived paths against `WriteStat` paths and deletes the difference — files [...]
+| 10 | **Metadata table update** | both | a full MOR deltacommit on the MDT — see §9 | `hoodie.metadata.enable` and the per-partition index configs | A failure here fails the data commit: `writeTableMetadata` is called *before* `saveAsComplete` (`BaseCommitActionExecutor.java:230`, `:233`). |
+| 11 | **Commit** | driver | none | `hoodie.write.concurrency.mode`, lock provider configs | Runs inside `txnManager.beginStateChange(...)` / `endStateChange(...)` (`BaseCommitActionExecutor.java:199-209`) with `TransactionUtils.resolveWriteConflictIfAny` in between (`:204-205`). A `HoodieWriteConflictException` here is a concurrent-writer conflict, not a bug. |
+| 12 | **Post-commit services** | driver schedules | varies | `hoodie.clean.automatic`, `hoodie.archive.automatic`, `hoodie.compact.inline`, `hoodie.clustering.inline` | `runTableServicesInline` then `autoCleanOnCommit` / `autoArchiveOnCommit` (`BaseHoodieWriteClient.java:711-712`, `:719-723`). Inline services run **after** the commit is durable, so a failure here leaves a valid commit with stale services. |
+
+### Index tagging, per index type
+
+This is where the "stage 14 OOM" questions usually land. The shapes differ a lot.
+
+| Index | Spark shape | Parallelism | Characteristic failure |
+|---|---|---|---|
+| **BLOOM** (`HoodieBloomIndex`) | `mapToPair` to `(partitionPath, recordKey)` (`HoodieBloomIndex.java:82-83`) → driver `countByKey` for per-partition counts (`:117`) → driver-side load of key ranges → **`explodeRecordsWithFileComparisons`**: one output row per `(record, candidate file group)` (`:300-310`) → sort/repartition → `mapPartitions` probing files | `hoodie.bloom.index.parallelism`; if `0`, `max(inputParallelism, 10)` (`SparkHoodieBloomIndexHelper.java:97-98`) | **The fan-out is [...]
+| **BLOOM over metadata bloom_filters** | `repartitionAndSortWithinPartitions(AffineBloomIndexFileGroupPartitioner)` → `mapPartitionsToPair(HoodieMetadataBloomFilterProbingFunction)` → `mapPartitions(HoodieFileProbingFunction)` (`SparkHoodieBloomIndexHelper.java:156-160`) | parallelism rounded **up to a multiple of** `hoodie.metadata.index.bloom.filter.file.group.count` so each Spark partition maps to exactly one MDT file group (`:138-148`) | Task count that is not a round number and not [...]
+| **BLOOM bucketized** | `mapToPair` → `repartitionAndSortWithinPartitions(BucketizedBloomCheckPartitioner)` → `mapPartitions(HoodieSparkBloomIndexCheckFunction)` (`SparkHoodieBloomIndexHelper.java:170-173`) | `hoodie.bloom.index.keys.per.bucket`, optionally dynamic (`:165-168`) | Needs an extra `countByKey` over the comparison RDD to size buckets (`:226`) — an extra full shuffle before the real one. |
+| **GLOBAL_BLOOM** (`HoodieGlobalBloomIndex`) | same as bloom, but candidate files come from **all** partitions | same | Fan-out is across the whole table, not one partition. On a wide table this is the single most expensive tagging mode. |
+| **SIMPLE** (`HoodieSimpleIndex`) | `mapToPair` → driver collects affected partition list (`distinct().collectAsList()`, `HoodieSimpleIndex.java:135`) → `parallelize(baseFiles)` + `flatMap` reading **every key** of every affected base file (`:146-149`) → `leftOuterJoin` (`:112`) | `hoodie.simple.index.parallelism`, default `0`; when `0` or ≥ file count, parallelism = **number of base files** (`:144`, `:152-155`) | Reads the full key column of every base file in every touched partition o [...]
+| **GLOBAL_SIMPLE** | as SIMPLE but across all partitions | same | Scans the whole table's keys per write. Only viable on small tables. |
+| **RECORD_INDEX / RLI** (`SparkMetadataTableRecordLevelIndex`, global variant `SparkMetadataTableGlobalRecordLevelIndex`) | `map` to keys → `keyBy(hash → fileGroupIndex)` → `partitionBy(PartitionIdPassthrough(numFileGroups))` → `mapPartitionsToPair(lookup)` (`SparkMetadataTableGlobalRecordLevelIndex.java:127-139`; partitioned variant `SparkMetadataTableRecordLevelIndex.java:75-86`) | **task count = number of RLI file groups in the MDT**, asserted at `:134` / `:82`. Not a shuffle-paralle [...]
+| **BUCKET** (`HoodieBucketIndex`) | `records.mapPartitions(...)` only — **no shuffle** (`HoodieBucketIndex.java:81-91`) | none; location is computed from the key hash | Tagging is nearly free; a bucket-index table that is slow is slow somewhere else. Failure mode is wrong bucket count, not tagging cost. Note `BaseSparkCommitActionExecutor.handleUpdate` falls back to insert when the base file is missing under `BUCKET` (`:437-440`). |
+
+### Merge handles
+
+`handleUpdate` builds a handle via `HoodieMergeHandleFactory.create(...)` and runs it through
+`MergeUtils.runMerge` (`BaseSparkCommitActionExecutor.java:443-444`, `:447-465`). For a
+bootstrapped table it additionally resolves partition values from the bootstrap base path
+(`:453-460`). `handleInsert` instead returns a `SparkLazyInsertIterable` with a
+`CreateHandleFactory` (`:468-476`) — new file groups, no read of an old file.
+
+---
+
+## 2. UPSERT — Merge-on-Read
+
+Everything in §1 applies up to and including the workload profile. The divergence is in the
+partitioner and in what `handleUpdate` does.
+
+```
+ ... identical through buildProfile ...
+ |
+ v
+ [5'] SparkUpsertDeltaCommitPartitioner (instead of UpsertPartitioner)
+ |
+ v
+ [7'] per bucket:
+ canIndexLogFiles() ?
+ yes -> HoodieAppendHandle (inserts AND updates go to log blocks)
+ no -> small file id ?
+ yes -> HoodieMergeHandle (rewrite base file, "small file correction")
+ no -> HoodieAppendHandle (append log block)
+ |
+ v
+ ... identical: WriteStatus, markers, MDT, commit ...
+ action type is deltacommit, not commit
+```
+
+| Divergence | Detail | Source |
+|---|---|---|
+| Partitioner | `SparkUpsertDeltaCommitPartitioner` extends `UpsertPartitioner`, overriding only `getSmallFiles` | `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/SparkUpsertDeltaCommitPartitioner.java:47-55` |
+| Small-file candidates | If the index can index log files, **any** file slice under `hoodie.parquet.max.file.size` is a candidate (`isSmallFile` uses `getTotalFileSizeAsParquetFormat`). Otherwise only slices with **zero log files** qualify, sorted ascending by base-file size, limited by `hoodie.merge.small.file.group.candidates.limit` | `SparkUpsertDeltaCommitPartitioner.java:90-117`, `:124-127` |
+| Updates | Normally `HoodieAppendHandle.doAppend()` — appends a log block, **no base-file rewrite**. Falls back to the CoW merge path only when the index cannot index log files **and** the fileId is in the small-file set | `hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/BaseSparkDeltaCommitActionExecutor.java:72-86` |
+| Inserts | If `canIndexLogFiles()`, inserts also go to log files via `AppendHandleFactory`; otherwise they create base files as in CoW | `BaseSparkDeltaCommitActionExecutor.java:89-97` |
+| Markers | Reconciliation deliberately ignores log files appended for update (they are idempotent-safe), but *newly created* log files are included | `HoodieTable.java:811-813` |
+| Workload profile semantics | Only updates are recorded in the inflight workload metadata; log-block updates are unknown across batches, so inserts are rolled back by commit time instead | `BaseCommitActionExecutor.java:122-127` |
+
+**What commonly fails here:** the write stage produces far less I/O than CoW (no base rewrite),
+so an MOR upsert that is slow in the write stage is usually hitting the small-file-correction
+branch — i.e. it *is* rewriting base files. Check whether `getSmallFileIds()` is large. Separately,
+a MOR table whose log files grow without bound means compaction is not keeping up (§5), which
+shows up at read time, not write time.
+
+---
+
+## 3. INSERT and BULK_INSERT
+
+### INSERT
+
+`INSERT` runs the same `BaseSparkCommitActionExecutor` path as `UPSERT`, with two differences:
+
+- `getPartitioner` dispatches on `WriteOperationType.isChangingRecords(operationType)`; for
+ non-changing operations it calls `getInsertPartitioner`, which in the base class simply returns
+ `getUpsertPartitioner` (`BaseSparkCommitActionExecutor.java:312-321`, `:485-487`). So INSERT
+ still gets small-file packing.
+- Index tagging is skipped when `table.getIndex().requiresTagging(operationType)` is false
+ (`BaseWriteHelper.java:71-75`). Dedupe is still controlled by `hoodie.combine.before.insert`.
+
+Parallelism comes from `hoodie.insert.shuffle.parallelism` (default `0`,
+`HoodieWriteConfig.java:399-401`).
+
+### BULK_INSERT
+
+```
+ [1] transitionRequestedToInflight (driver)
+ |
+ [2] optional dedupe (hoodie.combine.before.insert) reduceByKey
+ |
+ [3] partitioner.repartitionRecords(records, targetParallelism) sort / repartition / nothing
+ |
+ [4] mapPartitionsWithIndex(BulkInsertMapFunction) -> flatMap write handles only
+ |
+ [5] updateIndexAndMaybeRunPreCommitValidations (no tagging)
+ |
+ [6] commit
+```
+
+`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkBulkInsertHelper.java:78-93`,
+`:121-140`.
+
+**Why bulk_insert skips index tagging:** there is simply no `tagLocation` call on this path.
+`SparkBulkInsertHelper.bulkInsert` goes dedupe → repartition → write, and then calls
+`updateIndexAndMaybeRunPreCommitValidations` (`:92`), which only calls `index.updateLocation` —
+and for bloom and RLI that is a no-op returning the input unchanged
+(`HoodieBloomIndex.java:329-333`, `SparkMetadataTableGlobalRecordLevelIndex.java:155-159`).
+Bulk_insert therefore **cannot detect that an incoming key already exists**; it writes new file
+groups unconditionally. That is the operation's defining trade-off, not a bug.
+
+### Sort modes
+
+| Mode | Partitioner | Spark op | Notes |
+|---|---|---|---|
+| `NONE` | `NonSortPartitioner` | `coalesce(n)` if `enforceNumOutputPartitions`, else the RDD untouched (`NonSortPartitioner.java:60-66`) | Fastest; file sizing is whatever the input partitioning gives you. |
+| `GLOBAL_SORT` | `GlobalSortPartitioner` | `records.sortBy(partitionPath + "+" + recordKey, true, outputSparkPartitions)` (`GlobalSortPartitioner.java:54-59`) | Full global shuffle+sort — the expensive one. **Throws** if meta fields are disabled (`:49-51`). Best file sizes. |
+| `PARTITION_SORT` | `RDDPartitionSortPartitioner` | `mapToPair` then sort within Spark partitions (`RDDPartitionSortPartitioner.java:60`) | Sorts only within each Spark partition; lower memory than global sort, worse sizing. |
+| `PARTITION_PATH_REPARTITION` | `PartitionPathRepartitionPartitioner` | repartition by partition path | One executor per table partition. |
+| `PARTITION_PATH_REPARTITION_AND_SORT` | `PartitionPathRepartitionAndSortPartitioner` | `mapToPair(partitionPath, record)` → `repartitionAndSortWithinPartitions(partitioner)` (`PartitionPathRepartitionAndSortPartitioner.java:70-71`) | As above plus an intra-partition sort by partition path. |
+
+Mode descriptions, including the skew warning on the two `PARTITION_PATH_REPARTITION*` modes, are
+in the enum itself: `hudi-client/hudi-client-common/src/main/java/org/apache/hudi/execution/bulkinsert/BulkInsertSortMode.java:28-51`.
+
+Selection happens in
+`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/BulkInsertInternalPartitionerFactory.java:66-85`,
+with two earlier overrides: a `BUCKET` index routes to `RDDConsistentBucketBulkInsertPartitioner`
+or `RDDSimpleBucketBulkInsertPartitioner` regardless of sort mode (`:41-47`), and an LSM-layout
+table *rejects* `NONE` and the `PARTITION_PATH_REPARTITION` mode outright because they do not
+guarantee record ordering (`:48-62`).
+
+**Common failures:** `GLOBAL_SORT` on a large import is a single full shuffle and is the usual
+cause of "bulk_insert spilled to disk / lost executor". Using `NONE` with few input partitions
+produces few, enormous files. The two `PARTITION_PATH_REPARTITION*` modes on skewed partition
+paths put almost all data on one executor — the enum documentation says so explicitly
+(`BulkInsertSortMode.java:40-51`).
+
+### Row-writer path vs record path
+
+`hoodie.datasource.write.row.writer.enable` defaults to `true`
+(`hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:546-548`).
+When on, bulk_insert operates on `Dataset[Row]` / `InternalRow` and never materialises
+`HoodieRecord` objects:
+
+- `prepareForBulkInsert` prepends the five meta fields as a schema change and fills record key /
+ partition path with a `mapPartitions` over `queryExecution.toRdd`
+ (`hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieDatasetBulkInsertHelper.scala:81-126`).
+- When meta fields are *not* populated, it avoids dereferencing the DataFrame into an RDD at all
+ and just projects null stubs (`:135-149`) — so the DAG looks completely different.
+- Dedupe, if enabled, is `dedupeRows` on the `InternalRow` RDD (`:128-132`).
+- Then `partitioner.repartitionRecords(updatedDF, targetParallelism)` (`:151`).
+
+The record path (`SparkBulkInsertHelper`) is what runs when the row writer is off. Practically:
+if the Spark UI shows `WholeStageCodegen` and `Exchange` nodes with a SQL query plan, you are on
+the row-writer path; if you see plain RDD stages with `MapPartitionsRDD`, you are on the record
+path **[inferred from the two code paths' use of Dataset vs JavaRDD — the UI rendering itself is
+standard Spark behaviour]**.
+
+---
+
+## 4. DELETE and DELETE_PARTITION
+
+### DELETE
+
+```
+ dedupe/repartition keys -> create delete records -> tagLocation
+ -> filter(isCurrentLocationKnown) -> normal commit executor
+```
+
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/commit/HoodieDeleteHelper.java:86-107`.
+
+- Parallelism: `hoodie.delete.shuffle.parallelism`, default `0` (`HoodieWriteConfig.java:469-471`),
+ deduced as usual (`HoodieDeleteHelper.java:86`).
+- Dedupe is `distinctWithKey(HoodieKey::getRecordKey, parallelism)` for a global index, plain
+ `distinct(parallelism)` otherwise (`:67-73`). If dedupe is off, the keys are still
+ `repartition(targetParallelism)`-ed (`:92`) — so there is always a shuffle.
+- **Tagging always runs** for DELETE (`:99`), unconditionally, unlike the
+ `requiresTagging(operationType)` gate on the upsert path.
+- Keys that tag as new are dropped: `filter(HoodieRecord::isCurrentLocationKnown)` (`:103`).
+
+**Common failure:** deleting keys that do not exist costs a full index lookup and then writes
+nothing. A DELETE whose Spark job looks identical in cost to an UPSERT but produces zero write
+stats is this.
+
+### DELETE_PARTITION
+
+Not a data-writing operation at all. `SparkDeletePartitionCommitActionExecutor.execute()`:
+
+- `context.parallelize(partitions).distinct().mapToPair(p -> (p, getAllExistingFileIds(p))).collectAsMap()`
+ — one task per distinct partition, result collected to the driver
+ (`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/SparkDeletePartitionCommitActionExecutor.java:63-67`).
+- Write statuses are **empty** (`:71`); the whole effect is `partitionToReplaceFileIds` on a
+ `replacecommit`.
+- Pre-flight check: `DeletePartitionUtils.checkForPendingTableServiceActions` (`:58`) — this is
+ what raises when a pending compaction or clustering touches a partition you are dropping.
+- If the requested instant file is missing it throws
+ `HoodieDeletePartitionException("Commit instant was not created for instant …")` (`:76-80`).
+
+---
+
+## 5. COMPACTION (MOR)
+
+```
+ SCHEDULING (driver)
+ needCompact(trigger strategy) ?
+ | yes
+ v
+ plan generator -> strategy.orderAndFilter -> HoodieCompactionPlan
+ |
+ write .compaction.requested to timeline
+ |
+ EXECUTION
+ v
+ context.parallelize(operations) <- ONE TASK PER FILE GROUP
+ .map(compact) .flatMap
+ |
+ commit as a `commit` action on the MOR timeline
+```
+
+### Scheduling
+
+`ScheduleCompactionActionExecutor.needCompact` (`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java:197-250`)
+dispatches on `hoodie.compact.inline.trigger.strategy` (`HoodieCompactionConfig.java:105`):
+
+| Strategy | Meaning | Source |
+|---|---|---|
+| `NUM_COMMITS` | ≥ N delta commits since last **completed** compaction | `CompactionTriggerStrategy.java:28-30` |
+| `NUM_COMMITS_AFTER_LAST_REQUEST` | ≥ N delta commits since last completed **or requested** compaction | `:32-35` |
+| `TIME_ELAPSED` | ≥ N seconds since last compaction | `:37-39` |
+| `NUM_AND_TIME` | both conditions | `:41-44` |
+| `NUM_OR_TIME` | either condition | `:46-49` |
+
+N and the seconds come from `hoodie.compact.inline.max.delta.commits`
+(`HoodieCompactionConfig.java:90`) and the delta-seconds config, read at
+`ScheduleCompactionActionExecutor.java:208-209`.
+
+### Plan strategies
+
+`hoodie.compaction.strategy` (`HoodieCompactionConfig.java:152`) selects a `CompactionStrategy`,
+whose `orderAndFilter` picks which file groups make the plan
+(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/strategy/CompactionStrategy.java:101`).
+
+- **`BoundedIOCompactionStrategy`** walks operations in order and accepts them while a running
+ budget lasts: `long targetIORemaining = writeConfig.getTargetIOPerCompactionInMB();` then
+ `targetIORemaining -= opIo` per operation (`BoundedIOCompactionStrategy.java:47-52`). When the
+ budget is exhausted it stops (`:55-57`).
+- **`LogFileSizeBasedCompactionStrategy`** (the default family) first *filters out* file groups
+ whose total log size is below `hoodie.compaction.logfile.size.threshold`
+ (`HoodieCompactionConfig.java:137`), sorts the rest by descending log size, then delegates to
+ the bounded-IO filter (`LogFileSizeBasedCompactionStrategy.java:47-66`).
+
+**`hoodie.compaction.target.io` is the lever.** Default `500 * 1024` MB = 512 GB
+(`HoodieCompactionConfig.java:130-134`) — large enough that most users never hit it, which is
+exactly why it is the right knob when compaction runs are too long. Lowering it caps the per-run
+I/O (and therefore the runtime), at the cost of more runs. Note the budget check is `> 0` *before*
+subtracting, so one oversized operation can still exceed the budget by its own size
+(`BoundedIOCompactionStrategy.java:50-52`).
+
+### Execution
+
+```java
+return context.parallelize(operations).map(
+ operation -> compact(config, operation, compactionInstantTime, ...))
+ .flatMap(List::iterator);
+```
+
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/HoodieCompactor.java:154-156`
+
+and `parallelize(List)` with no explicit parallelism uses `data.size()`
+(`hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieEngineContext.java:75-80`).
+
+**So compaction task count = number of file groups in the plan. There is no compaction
+parallelism config.** This is the single most useful fact in this section: a compaction stage
+with 37 tasks has 37 file groups in its plan, and if 36 finish in a minute and one runs for an
+hour, that one file group has a pathological log-to-base ratio.
+
+Other execution facts:
+- `maxInstantTime` bounds which log blocks are merged (`HoodieCompactor.java:127`, `:189-194`).
+- `LOG_COMPACT` is a separate branch using a different reader context (`:134-145`).
+- On the metadata table, the reader context is forced to Avro with HFile caching props
+ (`:148-152`).
+
+**Common failures:** executor OOM on a single task (one file group with an enormous log),
+`HoodieCompactionException` wrapping a log-block read failure, and compaction scheduled but never
+executed (async compaction configured but no async service running — the plan accumulates as
+`.compaction.requested` instants and blocks archival, see §8).
+
+---
+
+## 6. CLUSTERING
+
+```
+ PLAN (driver)
+ commits since last clustering >= hoodie.clustering.inline.max.commits ?
+ |
+ plan strategy -> clustering groups (each group = N input slices -> M output file groups)
+ |
+ write requested clustering / replacecommit instant
+ |
+ EXECUTION
+ thread pool of min(#groups, hoodie.clustering.max.parallelism) threads
+ each thread launches an INDEPENDENT Spark job:
+ read group -> bulk-insert-style repartition -> write
+ |
+ union all groups' WriteStatus
+ |
+ COMMIT as replacecommit: new file groups REPLACE the input file groups
+```
+
+### Plan
+
+`ClusteringPlanActionExecutor.createClusteringPlan` gates on
+`config.getInlineClusterMaxCommits() > commitsSinceLastClustering`
+(`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ClusteringPlanActionExecutor.java:57-69`),
+i.e. `hoodie.clustering.inline.max.commits` (`HoodieClusteringConfig.java:157`) or
+`hoodie.clustering.async.max.commits` (`:164`).
+
+The instant action is `CLUSTERING_ACTION` on timeline layout v2 and `REPLACE_COMMIT_ACTION`
+otherwise (`ClusteringPlanActionExecutor.java:93-94`) — a version-dependent detail worth knowing
+when reading a timeline by hand.
+
+`PartitionAwareClusteringPlanStrategy.buildClusteringGroupsForPartition` fills groups up to
+`hoodie.clustering.plan.strategy.max.bytes.per.group` and computes the number of output file
+groups from `hoodie.clustering.plan.strategy.target.file.max.bytes`
+(`PartitionAwareClusteringPlanStrategy.java:83-86`, `:106-108`), stopping at
+`hoodie.clustering.plan.strategy.max.num.groups` (`:92-93`). Config keys are built from the prefix
+`hoodie.clustering.plan.strategy.` (`HoodieClusteringConfig.java:51`) plus `small.file.limit`
+(`:103-104`), `max.bytes.per.group` (`:201-202`), `max.num.groups` (`:210-211`),
+`target.file.max.bytes` (`:217-218`), `sort.columns` (`:241-242`).
+
+### Execution
+
+`MultipleSparkJobExecutionStrategy.performClustering` creates a fixed thread pool of
+`min(#inputGroups, hoodie.clustering.max.parallelism)` (default 15,
+`HoodieClusteringConfig.java:171-172`) and submits one async job per group
+(`MultipleSparkJobExecutionStrategy.java:108-132`), then unions the results
+(`:134`).
+
+**This is why clustering looks like many small Spark jobs rather than one big one.** Each group's
+job is effectively a bulk_insert: `SparkSortAndSizeExecutionStrategy` builds a *new* write config
+with `withBulkInsertParallelism(numOutputGroups)` and
+`PARQUET_MAX_FILE_SIZE = hoodie.clustering.plan.strategy.target.file.max.bytes`, then calls the
+bulk-insert helper (`SparkSortAndSizeExecutionStrategy.java:66-76` for the row path, `:90-97` for
+the RDD path). So each group's write stage has exactly `numOutputGroups` tasks.
+
+Row writer vs RDD is chosen by `hoodie.datasource.write.row.writer.enable`, read with a
+string literal rather than the config constant (`MultipleSparkJobExecutionStrategy.java:112`).
+
+Group reads use `jsc.parallelize(clusteringOps, readParallelism)` where the parallelism comes from
+`hoodie.clustering.group.read.parallelism` (default 20, `HoodieClusteringConfig.java:181-182`;
+used at `MultipleSparkJobExecutionStrategy.java:293`).
+
+### Replacecommit semantics
+
+The commit is a `replacecommit` carrying `partitionToReplaceFileIds`. The old file groups remain
+on storage until cleaning removes them (§7) — so a clustering commit does **not** free space
+immediately.
+
+**Concurrency hazard worth calling out:** updates landing on a file group that has pending
+clustering are handled by `hoodie.clustering.updates.strategy`
+(`BaseSparkCommitActionExecutor.clusteringHandleUpdate`, `:118-169`). With
+`SparkAllowUpdateStrategy` and `hoodie.clustering.rollback.pending.replacecommit.on.conflict`
+false, Hudi returns early and defers to conflict resolution (`:135-137`). Otherwise it *rolls back
+the pending clustering instant* mid-write (`:148-167`) — and the code itself flags the race:
+"there could be race condition, for example, if the clustering completes after instants are
+fetched but before rollback completed" (`:147`). A write that mysteriously rolled back someone
+else's clustering is this path.
+
+---
+
+## 7. CLEANING
+
+```
+ PLAN
+ getPartitionPathsToClean(earliestRetainedInstant) <- incremental or full, per policy
+ |
+ per partition (parallelism = min(#partitions, hoodie.clean.parallelism)):
+ getFilesToCleanKeepingLatest{Versions,Commits,ByHours}
+ |
+ write .clean.requested with the file list
+ |
+ EXECUTE
+ mapPartitionsToPairAndReduceByKey(deleteFilesFunc, PartitionCleanStat::merge, parallelism)
+ |
+ write .clean completed
+```
+
+### Policies
+
+`HoodieCleaningPolicy` (`hudi-common/src/main/java/org/apache/hudi/common/model/HoodieCleaningPolicy.java:38-46`),
+selected by `hoodie.clean.policy` (`HoodieCleanConfig.java:78`):
+
+| Policy | Keeps | Config | Deletes |
+|---|---|---|---|
+| `KEEP_LATEST_COMMITS` (default) | file slices needed to serve the last N commits | `hoodie.clean.commits.retained`, default 10, alt key `hoodie.cleaner.commits.retained` (`HoodieCleanConfig.java:52-53`, `:105-108`) | older file slices, and replaced file groups eligible to clean |
+| `KEEP_LATEST_FILE_VERSIONS` | last N versions of each file group | `hoodie.clean.fileversions.retained`, default 3 (`:56-57`, `:122-124`) | all older versions |
+| `KEEP_LATEST_BY_HOURS` | file slices within the last N hours | `hoodie.clean.hours.retained`, default 24 (`:54-55`, `:113-115`) | older slices |
+
+`getPartitionPathsToClean` treats `KEEP_LATEST_COMMITS` and `KEEP_LATEST_BY_HOURS` identically
+(both can clean incrementally from the previous clean's `earliestCommitToRetain`) and
+`KEEP_LATEST_FILE_VERSIONS` separately (`CleanPlanner.java:156-162`). Incremental planning is
+gated by `hoodie.clean.incremental.enabled` (`HoodieCleanConfig.java:144`).
+
+`getEarliestCommitToRetain` seeds from the previous clean's metadata and is capped by
+`hoodie.clean.max.commits.to.clean` (`CleanPlanner.java:621-650`, config at
+`HoodieCleanConfig.java:248`) — so a table that has not been cleaned in a long time does **not**
+try to clean everything in one run.
+
+### Savepoint interaction
+
+Savepointed data files are never deleted:
+
+- `KEEP_LATEST_FILE_VERSIONS` collects every savepoint's data files and skips any slice present in
+ that set — the comment is explicit: "do not clean up a savepoint data file"
+ (`CleanPlanner.java:364-371`, `:394-395`).
+- `isFileSliceExistInSavepointedFiles` checks both the base file **and** every log file of the
+ slice (`:338-348`) — an MOR slice is protected if *any* of its files is savepointed.
+- Removing a savepoint invalidates incremental cleaning: "Since savepoints have been removed
+ compared to previous clean, triggering clean planning for all partitions"
+ (`:232`, logic at `:250-258`). So the first clean after deleting a savepoint is a **full**
+ planning pass over every partition — a sudden slow clean right after a savepoint was dropped is
+ this.
+
+### Parallelism and failures
+
+Plan: `int cleanerParallelism = Math.min(partitionsToClean.size(), config.getCleanerParallelism());`
+(`CleanPlanActionExecutor.java:145`). Execute:
+`context.mapPartitionsToPairAndReduceByKey(..., deleteFilesFunc, PartitionCleanStat::merge, cleanerParallelism)`
+(`CleanActionExecutor.java:160-161`), with the same `min` applied at `:148-149`. Config is
+`hoodie.clean.parallelism` (`HoodieCleanConfig.java:168`).
+
+**Common failures:** object-store delete throttling (S3 503s) in the execute stage; a very long
+plan stage on a table with huge partition counts when incremental cleaning is off or was
+invalidated; and the quiet failure mode where cleaning never runs at all
+(`hoodie.clean.automatic` false, no async cleaner) so storage grows without bound.
+
+---
+
+## 8. ARCHIVAL
+
+Archival moves completed instants out of the active timeline (`.hoodie/`) into the archived /
+LSM timeline, then deletes the active-timeline files.
+
+```
+ getCommitInstantsToArchive()
+ |- bail out if completedCommits <= maxInstantsToKeep
+ |- compute earliestInstantToRetain = MIN of several candidates
+ |- filter out everything at/after the first savepoint
+ |- limit to (completedCommits - minInstantsToKeep)
+ |
+ getCleanAndRollbackInstantsToArchive(latestCommitInstantToArchive)
+ |
+ timelineWriter.write(...) -> archived timeline
+ deleteArchivedActions(...) -> remove from active timeline
+ timelineWriter.compactAndClean(...)
+```
+
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/TimelineArchiverV2.java:96-146`,
+`:180-312`, `:314-328`.
+
+### Min / max commits
+
+`hoodie.keep.min.commits` (`HoodieArchivalConfig.java:79`) and `hoodie.keep.max.commits` (`:59`).
+Nothing is archived until the completed-commits count exceeds `maxInstantsToKeep`
+(`TimelineArchiverV2.java:183-185`); when it does, archival trims down to `minInstantsToKeep`
+(`:310`).
+
+**These get silently overridden.** `ArchivalUtils.getMinAndMaxInstantsToKeep` compares the
+configured minimum against what cleaning needs. If the cleaner's `earliestCommitToRetain` implies
+more commits must be kept than `hoodie.keep.min.commits` allows, Hudi raises both values and logs
+a warning (`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/ArchivalUtils.java:72-105`):
+
+```java
+int minInstantsToKeepBasedOnCleaning =
+ completedCommitsTimeline.findInstantsAfter(earliestCommitToRetain.get().requestedTime())
+ .countInstants() + 2;
+```
+(`:73-75`)
+
+So "I set `hoodie.keep.min.commits=20` but my timeline has 400 instants" is usually this
+adjustment, and the fix is the *cleaning* config, not the archival config. The warning text names
+the relevant cleaner key per policy (`:86-101`).
+
+### What blocks archival
+
+`earliestInstantToRetain` is the minimum over several candidates (`TimelineArchiverV2.java:188-290`):
+
+1. the greatest completed commit before the earliest **pending** instant (`:190-213`) — so a single
+ stuck inflight instant pins the whole timeline;
+2. the earliest instant needed for pending **compaction** (`:219-226`);
+3. the earliest instant needed for pending **clustering** (`:232-235`);
+4. when the metadata table is enabled, the first commit modified after the MDT's latest
+ compaction — and if the MDT has never compacted, **nothing is archived at all**: "Not archiving
+ as there is no compaction yet on the metadata table" (`:240-251`);
+5. for the MDT's own timeline, the earliest instant still live in the data table (`:257-281`).
+
+### Savepoints stop archival
+
+```java
+Option<HoodieInstant> firstSavepoint = table.getCompletedSavepointTimeline().firstInstant();
+...
+return !firstSavepoint.isPresent()
+ || compareTimestamps(s.requestedTime(), LESSER_THAN, firstSavepoint.get().requestedTime());
+```
+`TimelineArchiverV2.java:294-306`
+
+Archival stops at the **earliest** savepoint — not the latest. An old forgotten savepoint pins the
+entire timeline after it. `hoodie.archive.beyond.savepoint` (`HoodieArchivalConfig.java:107`)
+changes this to merely skipping the savepointed commits themselves (`TimelineArchiverV2.java:299-301`),
+but that is a deliberate trade: the savepoint is no longer restorable once the surrounding
+timeline is archived. Described here because it explains the archiver's behaviour; it is **not**
+something to recommend. The feature is not fully supported, and `restore` is refused outright while
+it is on (`BaseHoodieWriteClient.java:883`).
+
+Clean and rollback instants are archived separately, bounded by the latest commit instant being
+archived (`:159-178`) — the ASCII diagram in that comment (`:167-173`) is the clearest statement
+of the rule.
+
+Archival takes the state-change lock for its whole run (`:98-101`, `:142-144`) and increments
+dedicated OOM / failure metrics (`:134-140`).
+
+---
+
+## 9. METADATA TABLE writes
+
+### MDT is a MOR table
+
+```java
+HoodieTableMetaClient.newTableBuilder()
+ .setTableType(HoodieTableType.MERGE_ON_READ)
+ ...
+ .setBaseFileFormat(HoodieFileFormat.HFILE.toString())
+ .setRecordKeyFields(RECORD_KEY_FIELD_NAME)
+```
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:559-572`
+
+HFile base files, Avro-payload log blocks, keyed by a single record-key field.
+
+### One deltacommit per data-table commit
+
+Every MDT write — bulk insert during initialisation, upsert during steady state, and partition
+deletes — commits with `DELTA_COMMIT_ACTION`:
+
+`hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:169-170`
+(bulk insert), `:181-182` and `:190-191` (upsert), `:206-210` (delete partitions, which commits as
+`REPLACE_COMMIT_ACTION`).
+
+The MDT deltacommit is written from `writeTableMetadata` inside the data table's commit, before
+`saveAsComplete` (`BaseCommitActionExecutor.java:230`, `:233`) — so the MDT is updated first and
+the data commit becomes visible second.
+
+### MDT compaction
+
+`compactIfNecessary` picks a compaction instant *below* the earliest instant that is pending on
+the data table but completed on the MDT, minus 1 millisecond, so that instant can still be rolled
+back (`HoodieBackedTableMetadataWriter.java:1643-1654`). It skips if a completed instant already
+exists at that time (`:1660`), then schedules and runs synchronously (`:1664-1670`), optionally
+followed by log compaction (`:1681-1691`). Both paths bump failure metrics and rethrow
+(`:1672-1676`, `:1693-1697`).
+
+### Partitions
+
+`hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java`:
+
+| Partition | Enum line | Written when |
+|---|---|---|
+| `files` | `:93` | always, when MDT is enabled — the file listing index |
+| `column_stats` | `:109` | `hoodie.metadata.index.column.stats.enable`; per column, per file |
+| `bloom_filters` | `:135` | `hoodie.metadata.index.bloom.filter.enable`; consumed by bloom tagging (§1) |
+| `record_index` | `:166` | `hoodie.metadata.record.index.enable`; consumed by RLI tagging (§1) |
+| `expression_index` | `:191` | per explicitly created expression index |
+| `secondary_index` | `:213` | per explicitly created secondary index |
+| `partition_stats` | `:247` | `hoodie.metadata.index.partition.stats.enable`; partition-level column ranges |
+
+`ALL_PARTITIONS` (`:274`) shares the `files` partition path and is not a separate physical
+partition.
+
+**Common failures:** the MDT is the single most common cause of "my commit failed for no visible
+reason" — a failure in `writeTableMetadata` aborts the data commit. RLI file-group count
+under-provisioning shows up as slow tagging (§1). An MDT that has never compacted blocks the data
+table's archival entirely (§8, candidate 4). MDT compaction failures are logged and rethrown, so
+they surface as the data write's exception.
+
+---
+
+## 10. ROLLBACK
+
+### Which strategy runs
+
+```java
+this.shouldRollbackUsingMarkers = shouldRollbackUsingMarkers && !instantToRollback.isCompleted();
+```
+`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:65`
+
+then
+
+```java
+if (shouldRollbackUsingMarkers) {
+ return new MarkerBasedRollbackStrategy(table, context, config, instantTime);
+} else {
+ return new ListingBasedRollbackStrategy(table, context, config, instantTime, isRestore);
+}
+```
+`:87-93`
+
+**This is the fact most people get wrong.** `hoodie.rollback.using.markers`
+(`ROLLBACK_USING_MARKERS_ENABLE`, `HoodieWriteConfig.java:320`) is necessary but not sufficient:
+rolling back a **completed** instant — which is what a restore or a savepoint rollback does —
+always falls through to the listing-based strategy regardless of the config, because the markers
+for a completed commit have already been cleaned up.
+
+### Marker-based
+
+`MarkerBasedRollbackStrategy.getRollbackRequests` reads the marker paths and maps over them with
+`parallelism = max(min(markerPaths.size(), hoodie.rollback.parallelism), 1)`
+(`MarkerBasedRollbackStrategy.java:76-79`). Cost is proportional to the number of files the failed
+write created — which is the whole point.
+
+### Listing-based
+
+```java
+List<String> partitionPaths = FSUtils.getAllPartitionPaths(context, table.getMetaClient(), false);
+int numPartitions = Math.max(Math.min(partitionPaths.size(), config.getRollbackParallelism()), 1);
+...
+return context.flatMap(partitionPaths, partitionPath -> { ... });
+```
+`ListingBasedRollbackStrategy.java:101-120`
+
+**Why it is expensive on wide tables:** it lists **every partition of the table**, not just the
+ones the failed write touched, and for each one lists files and matches them against the instant
+being rolled back (`fetchFilesFromInstant`, `:123-130`). On a table with 100 000 partitions this
+is 100 000 storage listings capped at `hoodie.rollback.parallelism` (default 100,
+`HoodieWriteConfig.java:481-483`) — ~1 000 sequential listing batches. On an object store this is
+minutes to hours, and it is a listing-bound workload, so adding executors past the parallelism cap
+does nothing.
+
+The MOR branch additionally reloads the active timeline per partition
+(`ListingBasedRollbackStrategy.java:135`) and rewrites the action type when the instant was a
+compaction or log-compaction (`:137-141`) — compaction instants carry the `commit` action, so this
+override is needed to find the right files.
+
+**Symptom to recognise:** a stage with exactly 100 tasks (or exactly
+`hoodie.rollback.parallelism`) whose description is `…:Creating Listing Rollback Plan: <table>`
+(`:105`), running far longer than the write it is rolling back.
+
+---
+
+## 11. The streaming ingestion loop (HoodieStreamer)
+
+```
+ syncOnce() StreamSync.java:529
+ |
+ [1] initializeMetaClientAndRefreshTimeline :535
+ |
+ [2] readFromSource :630
+ resolveCheckpointToResumeFrom :632
+ fetchNextBatchFromSource (retry loop) :638-653
+ | -> source read -> transformer -> schema reconciliation
+ | -> if checkpoint unchanged and !allowCommitOnNoCheckpointChange: RETURN NULL (no commit)
+ | :673-679
+ |
+ [3] initializeWriteClientAndRetryTableServices :556
+ new schema seen -> reInitWriteClient :564-579
+ |
+ [4] writeToSinkAndDoMetaSync :875
+ startCommit :1065
+ writeToSink :1090
+ pre-commit validators -> rollback on failure :955-962
+ count records :966-971
+ write-error gate (commitOnErrors) :984-997
+ writeClient.commit(...) :1000-1001
+ |
+ [5] source.onCommit(checkpointKey) :1010-1011
+ |
+ [6] scheduleCompaction if async :1013-1015
+ runMetaSync if records > 0 :1017-1021
+ |
+ [7] schemaProvider.refresh() :546-548
+ |
+ loop
+```
+
+| Stage | Where it stalls | Signature |
+|---|---|---|
+| Source read | `fetchFromSourceAndPrepareRecords` sets job description `…:Fetching next batch: <table>` (`:658`). Source timeouts are retried `cfg.maxRetryCount` times with `cfg.retryIntervalSecs` sleeps (`:638-652`) | repeated `HoodieSourceTimeoutException` in the log, each followed by a sleep; the job appears hung |
+| No-op cycle | If the checkpoint did not move and `allowCommitOnNoCheckpointChange` is false, the whole cycle returns `null` and nothing commits (`:672-679`) | "No new data, source checkpoint has not changed" in the log; empty-data metrics bumped (`:677`); a streamer that looks alive but never commits |
+| Schema change | Any new source or target schema triggers `reInitWriteClient` (`:564-579`) | a pause between batches with "Seeing new schema" logged (`:570`); repeated re-init every batch means the schema is oscillating |
+| Row writer eligibility | Only `BULK_INSERT` + a `ROW`-typed source + `hoodie.streamer.write.row.writer.enable` (default **false**) (`:694-697`) | users expecting the row-writer fast path on upsert never get it |
+| Spark record type guard | Spark record type + MOR + non-bulk-insert + non-parquet log block ⇒ `UnsupportedOperationException("Spark record only support parquet log.")` (`:660-665`) | immediate hard failure at batch start |
+| Write | the full §1/§2 pipeline | see those sections |
+| Validators | failure rolls back the instant and throws `HoodieStreamerWriteException` (`:956-962`) | write succeeded, then a rollback immediately after |
+| Write errors | `commitOnErrors=false` (default) ⇒ log top N errors, **roll back**, throw (`:988-996`); `true` ⇒ warn and commit (`:985-987`) | repeated write-then-rollback cycles with no forward progress |
+| Checkpoint commit | `source.onCommit(checkpointKey)` runs **after** the Hudi commit (`:1010-1011`) | a crash between the two re-reads the batch; Hudi's own checkpoint in commit metadata is the source of truth |
+| Table services | async compaction scheduled only after a successful commit (`:1013-1015`) | compaction plans accumulate if the async executor is not running |
+| Meta sync | skipped entirely when `totalSuccessfulRecords == 0` unless `forceEmptyMetaSync` (`:1017-1021`) | catalog drifts behind on low-volume tables |
+
+---
+
+## Read paths (lower detail)
+
+Enough to orient a failed *query*. This section is deliberately shallower than the write sections.
+
+`hoodie.datasource.query.type` (`DataSourceOptions.scala:59-67`) takes `snapshot` (default),
+`read_optimized`, or `incremental` (`:56-58`).
+
+| Query type | Reads | Typical failure |
+|---|---|---|
+| **snapshot** | CoW: latest base files. MOR: latest base file **merged with** its log files at read time | On MOR this is where un-compacted logs hurt: read latency grows with log size per file slice. Executor OOM in a scan stage on an MOR table usually means a file slice with a very large log set — fix compaction (§5), not the reader. |
+| **read_optimized** | base files only; log files ignored | Cheap and stable, but returns stale data on MOR between compactions. A user reporting "my upsert didn't show up" on an MOR table is very often on this query type. |
+| **incremental** | records between a begin and end instant | Fails when the begin instant has been **archived** (§8) — the timeline no longer has the commit metadata needed to resolve the range. The fix is archival/cleaning retention, not the reader. |
+| **time travel** | `as.of.instant` (`TIME_TRAVEL_AS_OF_INSTANT`, `DataSourceOptions.scala:184`, aliasing `HoodieCommonConfig.TIMESTAMP_AS_OF`) | Fails when the requested instant's file slices have been **cleaned** (§7). Savepoints are the mechanism for guaranteeing a time-travel point survives. |
+
+Data skipping (`hoodie.enable.data.skipping`, `DataSourceOptions.scala:187`) prunes files using
+the metadata table's `column_stats` / `partition_stats` partitions (§9). If it is on and those
+partitions do not exist, pruning silently does nothing — a query that is slower than expected with
+data skipping "enabled" may simply have no index to skip with.
+
+---
+
+## Cross-cutting lookup: Spark stage symptom → likely Hudi phase
+
+| Spark symptom | Likely phase | Confirm by | Typical cause |
+|---|---|---|---|
+| Stage input record count is 10–1000× the batch size | **Bloom index fan-out** (`explodeRecordsWithFileComparisons`) | `HoodieBloomIndex.java:300-310`; job description contains `Tagging:` | Random/UUID keys defeat range pruning; every record compared against every file group |
+| Executor OOM during tagging, many small tasks | **Bloom index probe** (`HoodieSparkBloomIndexCheckFunction`) | `SparkHoodieBloomIndexHelper.java:173` | Too many keys per bucket; raise `hoodie.bloom.index.parallelism` or lower `hoodie.bloom.index.keys.per.bucket` |
+| **Driver** OOM right after tagging finishes | **`buildProfile` `countByKey`** | `BaseSparkCommitActionExecutor.java:284-288`; description `Building workload profile:` | One map entry per `(partitionPath, fileId)` touched; very many file groups receiving updates |
+| Task count unrelated to any config, write stage, long tail | **`UpsertPartitioner` buckets** | `UpsertPartitioner.java:336-338`; description `Doing partition and writing data:` | Skewed bucket; one file group receiving most updates |
+| Executor OOM in the write stage, `HoodieUpsertException: Error upserting bucketType UPDATE for partition :N` | **Merge handle** | `BaseSparkCommitActionExecutor.java:413-417` | Too many records into one file group; reduce `hoodie.parquet.max.file.size` or fix key skew |
+| Stage reads thousands of base files for a tiny batch | **SIMPLE / GLOBAL_SIMPLE index** | `HoodieSimpleIndex.java:146-149`; task count = number of base files | Wrong index for the workload; move to bloom, bucket, or RLI |
+| Task count is an exact round number with no matching config (e.g. 64, 128) | **RLI lookup** — one task per MDT record-index file group | `SparkMetadataTableGlobalRecordLevelIndex.java:127-139` | RLI under-provisioned; each task looks up too many keys |
+| RLI table suddenly scanning base files | **RLI fallback to SIMPLE / GLOBAL_SIMPLE** | `SparkMetadataTableGlobalRecordLevelIndex.java:81-94` ("Record index not initialized") | `record_index` partition missing or not yet built |
+| Tagging stage has no shuffle at all | **Bucket index** | `HoodieBucketIndex.java:81-91` | Expected — look elsewhere for the slowdown |
+| Stage with exactly 200 tasks deleting files | **`finalizeWrite` marker reconciliation** | `HoodieWriteConfig.java:561-563`; description `Delete all partially written files:` | Speculative execution / task retries produced duplicate files |
+| `HoodieDuplicateDataFileDetectedException` | **marker reconciliation with fail-on-duplicate** | `HoodieTable.java:828-830` | `hoodie.fail.job.on.duplicate.data.file.detection` is on and retries produced extras |
+| Stage with exactly 100 tasks, description `Creating Listing Rollback Plan` | **Listing-based rollback** | `ListingBasedRollbackStrategy.java:101-105`; `HoodieWriteConfig.java:481-483` | Rolling back a **completed** instant, or markers disabled; cost ∝ total partitions |
+| One compaction task runs for hours while the rest finish fast | **Compaction, one pathological file group** | `HoodieCompactor.java:154-156` — one task per file group | A file group with a huge log-to-base ratio; cap with `hoodie.compaction.target.io` |
+| Many small concurrent Spark jobs during a replacecommit | **Clustering groups** | `MultipleSparkJobExecutionStrategy.java:108-134` | Expected; concurrency is `hoodie.clustering.max.parallelism` (default 15) |
+| Single very long shuffle during an initial load | **`GLOBAL_SORT` bulk insert** | `GlobalSortPartitioner.java:54-59` | Switch to `PARTITION_SORT` if global file sizing is not required |
+| Bulk insert produces huge or tiny files | **`NONE` sort mode** — input partitioning passed through | `NonSortPartitioner.java:60-66` | Set `hoodie.bulkinsert.shuffle.parallelism` or change sort mode |
+| Most data on one executor during bulk insert | **`PARTITION_PATH_REPARTITION[_AND_SORT]` with skewed partitions** | `BulkInsertSortMode.java:40-51` | Documented limitation of those modes |
+| Task count = number of table partitions, before the write stage | **`UpsertPartitioner` small-file probe** | `UpsertPartitioner.java:289`; description `Getting small files from partitions:` | High partition count; disable with `hoodie.parquet.small.file.limit <= 0` if acceptable |
+| Clean stage slow and listing every partition right after a savepoint was deleted | **Incremental cleaning invalidated** | `CleanPlanner.java:232`, `:250-258` | Expected one-time full pass |
+| Timeline grows without bound despite `hoodie.keep.min.commits` | **Archival blocked** | `TimelineArchiverV2.java:188-290`; `ArchivalUtils.java:72-105` | A stuck inflight instant, a pending compaction/clustering, an old savepoint, an MDT that never compacted, or cleaning retention overriding the archival config |
+| Nothing archives, log says "no compaction yet on the metadata table" | **MDT never compacted** | `TimelineArchiverV2.java:244-246` | MDT compaction not running |
+| Commit fails with no visible data-write error | **Metadata table write** | `BaseCommitActionExecutor.java:230` runs before `:233` | MDT deltacommit or MDT compaction failed |
+| `HoodieWriteConflictException` at commit time | **Concurrency conflict resolution** | `BaseCommitActionExecutor.java:204-205` | Overlapping concurrent writers; expected under OCC |
+| A write mysteriously rolled back someone else's clustering | **`clusteringHandleUpdate` rollback branch** | `BaseSparkCommitActionExecutor.java:148-167` (race documented at `:147`) | `hoodie.clustering.rollback.pending.replacecommit.on.conflict` is on |
+| Streamer alive but never commits | **Checkpoint unchanged** | `StreamSync.java:672-679` | No new source data, or `allowCommitOnNoCheckpointChange` off |
+| Streamer commits then immediately rolls back, repeatedly | **Write-error gate or pre-commit validator** | `StreamSync.java:956-962`, `:988-996` | Records failing to write; `commitOnErrors=false` |
+| MOR query slow / OOM in a scan stage | **Snapshot read merging large logs** | §5, §"Read paths" | Compaction not keeping up |
+| Incremental query fails to resolve its begin instant | **Begin instant archived** | §8 | Retention too aggressive for the consumer's lag |
+| Time-travel query fails on a valid-looking instant | **File slices cleaned** | §7 | Use a savepoint to pin the point in time |
+| Stage has no job description at all | Not instrumented, or lazily evaluated outside a `setJobStatus` block | `HoodieSparkEngineContext.java:237-239` | Fall back to task count and shuffle shape |
+
+---
+
+## Coverage notes
+
+Written against this checkout. Operations 1, 2, 3, 5, 7, 8, 10 and 11 are covered at full depth
+with per-phase source citations. Operations 4 (DELETE / DELETE_PARTITION), 6 (CLUSTERING) and 9
+(METADATA TABLE) are covered at phase-and-config depth — their plan-strategy internals and, for
+the MDT, the per-partition record-generation code, go deeper than this catalog does. The read-path
+section is deliberately orientation-only.
+
+Claims marked **[inferred]** are: the exact rendering of job descriptions in the Spark UI Stages
+tab; the absence of `RDD.setName` meaning generic RDD names in the DAG; and the visual distinction
+between the row-writer and record bulk-insert paths in the UI. Everything else is read from the
+files cited.
diff --git a/hudi-ai-operator/skills/hudi-rca/validate_config_keys.py b/hudi-ai-operator/skills/hudi-rca/validate_config_keys.py
new file mode 100644
index 000000000000..3ba2d7e5149c
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/validate_config_keys.py
@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Validate hoodie.* config keys referenced by the hudi-rca skill.
+
+Extracts every `hoodie.*` key mentioned in the skill's markdown (SKILL.md +
+references/*.md) and checks that each one is a real config key defined in the
+Hudi source tree — either a primary key (`.key("...")`) or a declared
+alternative (`withAlternatives("...")`). Keys built from prefix constants
+(e.g. METADATA_PREFIX + ".record.index.growth.factor") are resolved by
+scanning String constants and folding one level of `CONST + "literal"`
+concatenation to a fixpoint.
+
+Exit codes: 0 = all keys valid (or allowlisted), 1 = unknown keys found.
+
+Intentional exceptions (e.g. keys targeting a future Hudi version that the
+skill discusses but does not emit) live in validate_config_keys_allowlist.txt,
+one key per line, '#' comments allowed.
+
+Run from anywhere: python3 hudi-ai-operator/skills/hudi-rca/validate_config_keys.py
+"""
+
+import re
+import sys
+from pathlib import Path
+
+SKILL_DIR = Path(__file__).resolve().parent
+
+
+def _find_repo_root(start: Path) -> Path:
+ """Walk up from the skill directory to the enclosing Hudi checkout.
+
+ Located by marker rather than by counting parents, so relocating the skill
+ within the tree cannot silently break source-tree resolution.
+ """
+ for candidate in (start, *start.parents):
+ if (candidate / "hudi-common").is_dir() and (candidate / "pom.xml").is_file():
+ return candidate
+ return start.parents[-1]
+
+
+REPO_ROOT = _find_repo_root(SKILL_DIR)
+ALLOWLIST_FILE = SKILL_DIR / "validate_config_keys_allowlist.txt"
+
+# Source modules that define ConfigProperty keys.
+SOURCE_GLOBS = [
+ "hudi-common/src/main/**/*.java",
+ "hudi-client/**/src/main/**/*.java",
+ "hudi-utilities/src/main/**/*.java",
+ "hudi-spark-datasource/**/src/main/**/*.scala",
+ "hudi-spark-datasource/**/src/main/**/*.java",
+ "hudi-flink-datasource/**/src/main/**/*.java",
+ "hudi-sync/**/src/main/**/*.java",
+ # Cloud lock providers live in their own modules; the multi-writer references
+ # emit hoodie.write.lock.dynamodb.* keys defined only here.
+ "hudi-aws/src/main/**/*.java",
+ "hudi-gcp/src/main/**/*.java",
+]
+
+# test-corpus.md quotes stacktraces from real GitHub issues verbatim. Those are
+# evidence of what users actually hit, including keys from older releases, so
+# they are deliberately not validated against today's source tree.
+CORPUS_FILES = {"test-corpus.md"}
+MARKDOWN_FILES = [SKILL_DIR / "SKILL.md"] + [
+ f for f in sorted((SKILL_DIR / "references").glob("*.md"))
+ if f.name not in CORPUS_FILES
+]
+
+# --- extraction from markdown -------------------------------------------------
+
+KEY_TOKEN = re.compile(r"\bhoodie\.[a-z0-9_]+(?:\.[a-z0-9_]+|\.\{[a-z0-9_,]+\})+")
+
+
+def expand_braces(token: str):
+ """hoodie.a.{min,max}.b -> [hoodie.a.min.b, hoodie.a.max.b]"""
+ m = re.search(r"\{([a-z0-9_,]+)\}", token)
+ if not m:
+ return [token]
+ out = []
+ for alt in m.group(1).split(","):
+ out.extend(expand_braces(token[: m.start()] + alt + token[m.end():]))
+ return out
+
+
+def extract_markdown_keys():
+ found = {} # key -> list of "file:line"
+ for md in MARKDOWN_FILES:
+ for lineno, line in enumerate(md.read_text().splitlines(), 1):
+ for tok in KEY_TOKEN.findall(line):
+ for key in expand_braces(tok):
+ found.setdefault(key, []).append(f"{md.name}:{lineno}")
+ return found
+
+
+# --- valid-key set from the source tree ----------------------------------------
+
+CONST_DEF = re.compile(r'String\s+(\w+)\s*=\s*(.+?);')
+STRING_LIT = re.compile(r'"((?:hoodie|_hoodie)[^"]*)"')
+KEY_CALL = re.compile(r'(?:\.key|withAlternatives)\(([^;]*?)\)')
+
+
+def build_valid_keys():
+ constants = {} # NAME -> string value (only fully-resolved ones kept)
+ key_exprs = [] # raw argument expressions of .key(...) / withAlternatives(...)
+ literals = set()
+
+ files = []
+ for pattern in SOURCE_GLOBS:
+ files.extend(REPO_ROOT.glob(pattern))
+
+ pending_defs = []
+ for f in files:
+ try:
+ text = f.read_text(errors="ignore")
+ except OSError:
+ continue
+ # No content gate: files like KafkaSourceConfig build every key from
+ # imported prefix constants and may contain no literal "hoodie." at all.
+ for m in CONST_DEF.finditer(text):
+ pending_defs.append((m.group(1), m.group(2)))
+ for m in KEY_CALL.finditer(text):
+ key_exprs.append(m.group(1))
+ # any quoted hoodie.* literal counts as evidence the key exists somewhere
+ # in config-definition context; keep them for prefix resolution only.
+
+ def eval_expr(expr):
+ """Resolve '"lit"', 'CONST + "lit"', etc. to the SET of possible strings.
+
+ Constant names like PREFIX repeat across files with different values, so
+ constants map to value-sets and concatenation is cartesian. A key is
+ considered valid if any combination resolves to it — still strong
+ evidence, since every combination pairs a real prefix with a real
+ .key()/.withAlternatives() suffix.
+ """
+ parts = [p.strip() for p in expr.split("+")]
+ out = {""}
+ for p in parts:
+ if len(p) >= 2 and p.startswith('"') and p.endswith('"'):
+ out = {o + p[1:-1] for o in out}
+ elif p in constants:
+ out = {o + v for o in out for v in constants[p]}
+ if len(out) > 512: # runaway cartesian guard
+ return set()
+ else:
+ return set()
+ return out
+
+ # resolve constants to a fixpoint (handles PREFIX = OTHER_PREFIX + "x.")
+ changed = True
+ while changed:
+ changed = False
+ for name, expr in pending_defs:
+ for val in eval_expr(expr):
+ if "hoodie." in val and val not in constants.setdefault(name, set()):
+ constants[name].add(val)
+ changed = True
+
+ valid = set()
+ for expr in key_exprs:
+ for val in eval_expr(expr):
+ if val.startswith("hoodie."):
+ valid.add(val)
+ # plain constants that are themselves complete keys
+ for vals in constants.values():
+ for val in vals:
+ if val.startswith("hoodie.") and not val.endswith("."):
+ valid.add(val)
+ return valid
+
+
+def load_allowlist():
+ if not ALLOWLIST_FILE.exists():
+ return set()
+ out = set()
+ for line in ALLOWLIST_FILE.read_text().splitlines():
+ line = line.split("#", 1)[0].strip()
+ if line:
+ out.add(line)
+ return out
+
+
+def main():
+ md_keys = extract_markdown_keys()
+ valid = build_valid_keys()
+ allow = load_allowlist()
+
+ if not valid:
+ print("ERROR: no config keys resolved from the source tree.")
+ print(f" Looked under: {REPO_ROOT}")
+ print(" This script must run from a copy of the skill INSIDE a Hudi checkout")
+ print(" (hudi-ai-operator/skills/hudi-rca/), not from an installed copy under")
+ print(" ~/.claude/skills/. Run the repo copy instead.")
+ return 2
+
+ unknown = {}
+ for key, locations in sorted(md_keys.items()):
+ if key in valid or key in allow:
+ continue
+ unknown[key] = locations
+
+ print(f"Checked {len(md_keys)} distinct hoodie.* keys from {len(MARKDOWN_FILES)} markdown files")
+ print(f"against {len(valid)} keys resolved from the Hudi source tree "
+ f"({len(allow)} allowlisted).")
+ if not unknown:
+ print("OK — every referenced key exists in source (or is allowlisted).")
+ return 0
+
+ print(f"\nUNKNOWN KEYS ({len(unknown)}) — not defined in source, not allowlisted:")
+ for key, locations in unknown.items():
+ locs = ", ".join(locations[:3]) + (" …" if len(locations) > 3 else "")
+ print(f" {key}\n at {locs}")
+ print("\nFix the reference file, or add to validate_config_keys_allowlist.txt "
+ "with a comment explaining why.")
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/hudi-ai-operator/skills/hudi-rca/validate_config_keys_allowlist.txt b/hudi-ai-operator/skills/hudi-rca/validate_config_keys_allowlist.txt
new file mode 100644
index 000000000000..f5313c330944
--- /dev/null
+++ b/hudi-ai-operator/skills/hudi-rca/validate_config_keys_allowlist.txt
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Keys the validator cannot resolve from ConfigProperty declarations, each
+# verified by hand against the source tree. One key per line; '#' comments.
+#
+# Wildcard prefixes used in prose, not emitted as keys. The reference text says
+# "the keys under hoodie.streamer.kafka.*", and the token extractor cannot tell
+# that from a real key.
+hoodie.streamer.kafka
+hoodie.streamer.source
+hoodie.write.concurrency
+hoodie.write.lock.dynamodb
+hoodie.write.lock.zookeeper
+
+# Prefix constants that configs are built from, referenced as prefixes.
+# HoodieClusteringConfig.CLUSTERING_STRATEGY_PARAM_PREFIX, line 51.
+hoodie.clustering.plan.strategy
+
+# Built from a prefix constant the resolver does not fold:
+# KafkaSourceConfig.java:114 declares .key(PREFIX + "minPartitions").
+hoodie.streamer.source.kafka.min
+
+# Read as a raw string literal rather than declared as a ConfigProperty, so it
+# is invisible to a declaration-based scan. StreamSync.java:696:
+# this.props.getBoolean("hoodie.streamer.write.row.writer.enable", false)
+hoodie.streamer.write.row.writer.enable