You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "shangxinli (via GitHub)" <gi...@apache.org> on 2026/05/15 19:41:34 UTC
[I] [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework [hudi]
shangxinli opened a new issue, #18750:
URL: https://github.com/apache/hudi/issues/18750
## Background
Follow-up to #18405 ([Danny's review comment](https://github.com/apache/hudi/pull/18405#discussion_r3214248837)).
#18405 introduced a Spark/HoodieStreamer pre-commit validator (`SparkKafkaOffsetValidator`) built on the framework from #18068. We now have two parallel write-error / write-status validation paths in HoodieStreamer:
1. The new `BasePreCommitValidator` framework (#18068, #18405) — runs *before* `writeClient.commit()`, parameterized by `TypedProperties`, state exposed via `ValidationContext`.
2. The existing `HoodieStreamerWriteStatusValidator` (`StreamSync.java:1403`) — runs *inside* `writeClient.commit()` via the `WriteStatusValidator` callback hook.
Danny suggested consolidating these by migrating (2) into the (1) framework. This issue tracks that work.
## Why it isn't a trivial migration
`HoodieStreamerWriteStatusValidator` does substantially more than pure validation:
- Counts records/errors and writes to a shared `AtomicLong totalSuccessfulRecords` consumed *after* commit (drives the `runMetaSync()` decision).
- **Commits the error table** as a side effect, with `ROLLBACK_COMMIT` / `LOG_ERROR` strategies.
- Rolls back the instant on validation failure; logs the top-100 errors.
- Runs *inside* `writeClient.commit()`, not before it.
The current `BasePreCommitValidator` / `ValidationContext` interface has no path to:
- Surface the raw `JavaRDD<WriteStatus>` (needed for error-table commit).
- Inject the error-table writer.
- Expose a post-commit counter back to `StreamSync`.
## Proposed approach (to discuss)
Option A — **Extend `ValidationContext`**: add accessors for `JavaRDD<WriteStatus>` and an error-table writer handle; keep side effects inside the validator.
Option B — **Split side-effects out**: keep validation pure (just succeed/fail with reasons) and move error-table commit + counter wiring into a separate post-validation step in `StreamSync`.
Leaning toward Option B because it keeps the validator framework clean and matches the "validate, don't mutate" contract of `BasePreCommitValidator`. Open to feedback.
## Acceptance criteria
- [ ] Decision on Option A vs. B (or alternative).
- [ ] `HoodieStreamerWriteStatusValidator` removed; equivalent behavior implemented as a pre-commit validator.
- [ ] Error-table commit + `totalSuccessfulRecords` wiring preserved (no regression in `runMetaSync` decision).
- [ ] Tests covering `ROLLBACK_COMMIT` and `LOG_ERROR` error policies.
- [ ] No behavior change for users who don't configure `hoodie.precommit.validators`.
cc @danny0405 @codope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework [hudi]
Posted by "puneetdixit200 (via GitHub)" <gi...@apache.org>.
puneetdixit200 commented on issue #18750:
URL: https://github.com/apache/hudi/issues/18750#issuecomment-4468031275
Hello maintainers, I would like to work on this refactor. I will compare the existing HoodieStreamerWriteStatusValidator flow with the pre-commit validator framework, migrate the validation path, and add tests to confirm equivalent write-error behavior. Please assign this to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #18750:
URL: https://github.com/apache/hudi/issues/18750#issuecomment-4469193873
@shangxinli might already be working on it. Please confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #18750: [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework
URL: https://github.com/apache/hudi/issues/18750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [RFC] Migrate HoodieStreamerWriteStatusValidator into the pre-commit validator framework [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on issue #18750:
URL: https://github.com/apache/hudi/issues/18750#issuecomment-4472442459
@puneetdixit200 I already started working on it. I will let you review the code and there is pending work, I can work with you for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org