You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:52:51 UTC
[I] Refactor code that is common to all ingestions from cloud sources [hudi]
hudi-bot opened a new issue, #15444:
URL: https://github.com/apache/hudi/issues/15444
Currently, there are features to ingest incrementally from S3 (HUDI-1897) and GCS (HUDI-4850). Refactor common logic used across both. This will help in easier implementation of future cloud based ingestion sources.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4929
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Refactor code that is common to all ingestions from cloud sources [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15444: Refactor code that is common to all ingestions from cloud sources
URL: https://github.com/apache/hudi/issues/15444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Refactor code that is common to all ingestions from cloud sources [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15444:
URL: https://github.com/apache/hudi/issues/15444#issuecomment-4795425987
This refactoring task appears to have been substantially completed through several incremental PRs. Common logic between S3 and GCS incremental sources is now centralized in `hudi-utilities/.../sources/helpers/`:
- `CloudObjectsSelectorCommon` (with a `Type` enum distinguishing S3 vs GCS)
- `CloudDataFetcher`
- `CloudObjectIncrCheckpoint`, `CloudObjectMetadata`, `CloudObjectsSelector`
- `IncrSourceCloudStorageHelper`
- `CloudSourceConfig`
Both `S3EventsHoodieIncrSource` (~143 lines) and `GcsEventsHoodieIncrSource` (~197 lines) are now thin wrappers delegating to these shared helpers. Notable refactoring PRs that addressed this:
- [HUDI-7416] #10701 — Remove duplicate `getFileFormat` and refactor filter methods for S3/GCS
- [HUDI-7418] #10724 — Common method for filtering in S3 and GCS sources
- [HUDI-7501] #10861 — Use source profile for S3 and GCS sources
- [HUDI-7452] #10777 — Repartition row dataset in S3/GCS based on task size
- [HUDI-7535] #10918 — Metrics for sourceParallelism and Refresh profile in S3/GCS
If there's any remaining duplication you have in mind that should be folded into the shared helpers, please call it out specifically — otherwise this issue can likely be closed as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org