You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "voon (Jira)" <ji...@apache.org> on 2025/05/02 02:51:01 UTC

[jira] [Updated] (HUDI-8354) Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer

     [ https://issues.apache.org/jira/browse/HUDI-8354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

voon updated HUDI-8354:
-----------------------
    Fix Version/s: 1.1.0
                       (was: 1.0.2)

> Support completion time as checkpoint in S3Event/GcsEvent incremental source in Hudi Streamer
> ---------------------------------------------------------------------------------------------
>
>                 Key: HUDI-8354
>                 URL: https://issues.apache.org/jira/browse/HUDI-8354
>             Project: Apache Hudi
>          Issue Type: Sub-task
>            Reporter: Shawn Chang
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 1.1.0
>
>   Original Estimate: 20h
>  Remaining Estimate: 20h
>
> Currently, S3Event/GcsEvent source uses `QueryInfo` to get time range for incremental queries. 
> `QueryContext` uses completion time to get time range for incremental queries and is more ideal. But QueryContext misses some logic that are required by S3/GCSEventSource. These logic can be found under IncrSourceHelper#generateQueryInfo: https://github.com/apache/hudi/blob/622c8a37cafea903b59c66fd4d40c80380df8e94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L158
> The goal is to: 
> 1. migrate these logic to QueryContext
> 2. Have S3/GCSEvent source uses QueryContext instead
> 3. Remove QueryInfo



--
This message was sent by Atlassian Jira
(v8.20.10#820010)