You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/04 04:33:46 UTC
[PR] feat(agent): add Hudi optimizer catalogs and skill [hudi]
nsivabalan opened a new pull request, #20197:
URL: https://github.com/apache/hudi/pull/20197
### Describe the issue this Pull Request addresses
Part of #19263 (specialized sub-agents for optimization and analysis), under the Agentic Lakehouse
umbrella #19256. Builds on the event-log analyzer in #20196.
Hudi exposes a lot of tuning surface — 939 distinct `hoodie.*` keys, of which roughly 237 are
performance or sizing levers. Knowing which of them matters for a particular table, at its current
size, with its current access pattern, is expertise most users reasonably do not have. Worse, about
93 of those levers only take effect under some other config's value, so a change can be made in good
faith and do nothing at all.
This adds the knowledge layer for an optimization advisor: what to look for in a *working* pipeline,
what it costs, what to change, and — equally important — when to leave it alone.
### Summary and Changelog
A skill plus two catalogs under `hudi-agent-gateway/skills/hudi-optimizer/`. No code; this is
knowledge, structured so an agent can apply it against evidence.
**Three required inputs.** Writer properties, a Spark event log, and read access to the table.
The central discipline is that **an observation is only trustworthy when at least two of the three
agree**. An event log alone says "the write stage is slow". With the table it says "and the median
base file is 8 MB". With the properties it says "and your small-file limit sits below your batch
size, so small-file handling never engages" — which is the first version a user can act on. Every
catalog row records which inputs corroborate it, and the skill states its confidence accordingly:
with one input it describes, with two it recommends, with three it recommends with verification.
**`references/observation-catalog.md`** — 30 rows of "this works, but could be better". Each row
names the measurable signals as concrete fields in the analyzer, health-checker or layout-analyzer
JSON; which inputs corroborate it; the change with exact config keys and direction; a risk tier;
expected payoff; and how to verify it worked.
The backbone is derived expectations — comparing what the configuration *implies* against what was
measured. "Your target file size is 120 MB and your median base file is 8 MB" needs no baseline and
no history. Several rows are counter-intuitive and say so: reducing index parallelism, lowering
Spark memory fractions, and one row whose correct outcome is to change nothing at all, because a
cleaner that appears idle is usually correct.
**`references/capability-catalog.md`** — 7 rows of "adopt this feature", which is where an agent
beats documentation. The docs can say a record-level index gives faster lookups; they cannot say
whether it suits *this* table at *this* size with *this* update pattern. Each row carries
preconditions, measured signals both for and against, estimated payoff, adoption cost, and
verification.
The record-level index row is the worked example. Its premise is mechanical:
`HoodieSimpleIndex.fetchRecordLocations` parallelises over base files, so simple-index tagging cost
scales with table size rather than batch size — which is visible in an event log as a tagging stage
whose task count tracks the file count instead of the record count. It carries eight signals for and
seven against, including the most important counter-signal: with time-ordered keys and updates
concentrated in recent partitions, bloom key-range pruning may already be near-optimal and the index
would add metadata-table overhead for no gain. The agent must be able to say "don't".
It also sizes the index before recommending it, using the same `estimateFileGroupCount` the writer
uses at initialisation, so the recommendation carries its own cost estimate. And it notes that
`hoodie.index.type` defaults to `SIMPLE` on Spark via a per-engine builder override — so the user is
on that index by default rather than by choice, and the wording reflects that.
**Risk tiers, and a refusal.** `safe` (reversible config), `operational` (affects table services,
reversible but needs a cycle to observe), `durable` (changes what is on storage). **The agent does
not recommend durable changes as optimizations** — switching index type, table type, partitioning or
key generator belongs to a design conversation with an architecture decision record, not to a
tuning pass. The refusal is structural rather than advisory: the observation catalog contains no
durable rows by construction, the three durable capability rows carry an explicit refusal block
where the recommendation would otherwise be, and the skill has a worked example of saying the
refusal out loud so it is not quietly skipped.
**`SKILL.md`** — collects the inputs, invokes `HoodieTableHealthChecker` and
`HoodieTableLayoutAnalyzer` rather than asking the user to run them separately, and reasons from
their underlying measurements rather than their summary verdicts. It carries the event-log
analyzer's caveats into its own reasoning: a phase share names where work was *forced*, not where it
logically belongs, and with streaming writes to the metadata table enabled a large share of the DAG
is unattributable, so the skill says so rather than reasoning over a partial picture. It recommends
only, and never applies a change.
Recipes come from Spark deployments, since that is where the evidence is. The core patterns — index
choice, file sizing, table-service cadence, metadata-table cost — apply to Flink as well, and Flink
coverage will deepen over time.
Every `hoodie.*` key cited was verified against the source tree; several keys that exist only in
folklore were dropped rather than guessed at. The JSON field paths were verified the same way,
against the actual tool sources.
### Impact
Purely additive documentation. No Java, no configs, no behaviour change. Nothing imports or executes
any of it. The skill is read-only by construction: it recommends, and never mutates a table.
### Risk Level
none
### Documentation Update
Self-contained. Thresholds in the catalogs are starting points derived from a small set of measured
runs; they are documented as such rather than presented as tuned constants, and are the most likely
thing to want revision once more workloads are measured.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(agent): add Hudi optimizer catalogs and skill [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20197:
URL: https://github.com/apache/hudi/pull/20197#issuecomment-5976717779
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17442",
"triggerID" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c94d4e72a7797a57edcdf5bc2b346e05328a6b07 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17442)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(agent): add Hudi optimizer catalogs and skill [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20197:
URL: https://github.com/apache/hudi/pull/20197#issuecomment-5977406973
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17442",
"triggerID" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c94d4e72a7797a57edcdf5bc2b346e05328a6b07 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17442)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(agent): add Hudi optimizer catalogs and skill [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20197:
URL: https://github.com/apache/hudi/pull/20197#issuecomment-5976683004
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c94d4e72a7797a57edcdf5bc2b346e05328a6b07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c94d4e72a7797a57edcdf5bc2b346e05328a6b07 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(agent): add Hudi optimizer catalogs and skill [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #20197:
URL: https://github.com/apache/hudi/pull/20197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org