You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:33:06 UTC
[I] Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer [hudi]
hudi-bot opened a new issue, #17287:
URL: https://github.com/apache/hudi/issues/17287
Currently, S3Event/GcsEvent source uses `QueryInfo` to get time range for incremental queries.
`QueryContext` uses completion time to get time range for incremental queries and is more ideal. But QueryContext misses some logic that are required by S3/GCSEventSource. These logic can be found under IncrSourceHelper#generateQueryInfo: https://github.com/apache/hudi/blob/622c8a37cafea903b59c66fd4d40c80380df8e94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L158
The goal is to:
1. migrate these logic to QueryContext
2. Have S3/GCSEvent source uses QueryContext instead
3. Remove QueryInfo
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-8354
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9105
- Fix version(s):
- 1.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #17287:
URL: https://github.com/apache/hudi/issues/17287#issuecomment-3630138987
### Linked PR(s)
- https://github.com/apache/hudi/pull/12131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17287:
URL: https://github.com/apache/hudi/issues/17287#issuecomment-5365814717
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8354).
**Findings: not done, and the linked PR is closed without merging.**
PR #12131 (`[HUDI-8354] WIP: Have cloud event source use QueryContext`) is **closed and unmerged**, and its title marks it WIP - so it was never a candidate to land. The `status:pr-available` label on this issue is misleading as a result: it suggests a patch is awaiting review when there is none. Worth removing.
The two-step plan in the description is the useful part and still stands:
1. migrate the logic under `IncrSourceHelper#generateQueryInfo` into `QueryContext`
2. have `S3EventsHoodieIncrSource` / `GcsEventsHoodieIncrSource` use `QueryContext` instead of `QueryInfo`
Since the reference in the description is pinned to an old commit, whoever picks this up should re-derive which pieces of `generateQueryInfo` are still missing from `QueryContext` against current `master` rather than trusting that link.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org