You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Poorvank Bhatia <pu...@gmail.com> on 2025/06/10 03:30:00 UTC

[DISCUSS] FLIP Suggestion: Externalize and Modernize Apache Pinot Connector

Hi all,


I’d like to start a discussion about externalizing and evolving the Flink-Pinot
connector
<https://github.com/apache/bahir-flink/tree/master/flink-connector-pinot>,
which currently lives in the Apache Bahir repo
<https://attic.apache.org/projects/bahir.html> and supports Pinot 0.6 and
Flink 1.17.

The proposal outlines a 2-phase plan:

     Externalize the connector: Into a separate git repo.

   -

   Upgrade to Pinot 1.2+ and Flink 2.0, and clean up deprecated APIs.
   -

   Add Table API & SQL support, enabling users to write to Pinot using
   Flink SQL.

     Support Pinot upserts:

   -

   Generating primary-key-aware segments and pushing them to the Pinot
   controller.

We build on the Bahir connector's foundations—streaming mode support,
checkpoint integration, and segment management—aiming to modernize it for
long-term compatibility.

Looking forward to your feedback!

For more details, please refer to this document
<https://docs.google.com/document/d/1XgTH-vummF-JZz5VUibKxRRtXMhDQE3T7tJbU18Cb7c/edit?tab=t.0>

Thanks,

Poorvank