You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2025/12/09 22:23:14 UTC

[I] Support new Flink Hudi Source [hudi]

HuangZhenQiu opened a new issue, #17538:
URL: https://github.com/apache/hudi/issues/17538

   ### Feature Description
   
   **What the feature achieves:**
   Hudi currently supports reading data via Flink Source Function APIs. However, it lacks a native Flink Source API implementation for consuming Hudi tables as a first-class Flink source. The proposal aims to fill that gap by implementing a Flink Source that adheres to Flink's Source API (introduced in Flink 1.11 and enhanced in 1.13+) to enable efficient, scalable and consistent reading of Hudi dataset in both streaming and batch modes.
   
   
   **Why this feature is needed:**
   
   Flink FLIP-27 solve several problems/shortcomings in the streaming source interface (SourceFunction) and simultaneously to unify the source interfaces between the batch and streaming APIs. By adopting the new source interface, Flink can read hudi data in batch mode for better processing efficiency of bounded data. It also unblocks Flink Hudi user to use Flink hybrid source to seamless switch reading data from hudi to kafka for back-fill use cases.
   
   
   
   ### User Experience
   
   **How users will use this feature:**
   - Configuration changes needed
   - API changes
   - Usage examples
   
   
   ### Hudi RFC Requirements
   
   **RFC PR link:** (if applicable)
   https://github.com/apache/hudi/tree/master/rfc/rfc-95
   
   **Why RFC is/isn't needed:**
   - Does this change public interfaces/APIs? Yes
   - Does this change storage format? Yes
   - Justification:
   
   The new design will use new Flink Source API for Hudi connectors. It will align the implementation with the Flink 
   APIs for batch and streaming unification. With the feature, Flink users may use Flink hybrid source to read from Hudi table first then switch to read from Kafka.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support new Flink Hudi Source [hudi]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #17538:
URL: https://github.com/apache/hudi/issues/17538#issuecomment-3634561294

   @HuangZhenQiu merged into https://github.com/apache/hudi/issues/17038


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Support new Flink Hudi Source [hudi]

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan closed issue #17538: Support new Flink Hudi Source
URL: https://github.com/apache/hudi/issues/17538


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org