You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Shawn Chang (Jira)" <ji...@apache.org> on 2024/10/13 06:28:00 UTC

[jira] [Created] (HUDI-8354) Have S3Event/GcsEvent source switch to use QueryContext

Shawn Chang created HUDI-8354:
---------------------------------

             Summary: Have S3Event/GcsEvent source switch to use QueryContext
                 Key: HUDI-8354
                 URL: https://issues.apache.org/jira/browse/HUDI-8354
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Shawn Chang


Currently, S3Event/GcsEvent source uses `QueryInfo` to get time range for incremental queries. 

`QueryContext` uses completion time to get time range for incremental queries and is more ideal. But QueryContext misses some logic that are required by S3/GCSEventSource. These logic can be found under IncrSourceHelper#generateQueryInfo: https://github.com/apache/hudi/blob/622c8a37cafea903b59c66fd4d40c80380df8e94/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L158

The goal is to: 
1. migrate these logic to QueryContext
2. Have S3/GCSEvent source uses QueryContext instead
3. Remove QueryInfo




--
This message was sent by Atlassian Jira
(v8.20.10#820010)