You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Hongshun Wang <lo...@gmail.com> on 2025/04/03 05:54:51 UTC
Re: [DISCUSS] FLIP-515: Dynamic Kafka Sink
Hi Matyas,
Sorry for the late reply. Dynamic Kafka Source has been introduced for a
long time, and it is a little confusing for developers like me
that KafkaDynamicSource and DynamicKafkaSource have the same kafka
connector repo for different things.
Now DynamicKafkaSink will be introduced again. I wonder:
1. Is there any plan to merge this code into one class?
2. Is there any plan to expose them to SQL?
Best
Hongshun
On Wed, Mar 26, 2025 at 2:48 PM Arvid Heise <ah...@confluent.io.invalid>
wrote:
> Hi Matyas,
>
> thanks for clarifying. The design LGTM.
>
> Based on your description, I'd phrase the state management a bit
> differently:
> * The source of truth is and always will be the MDS.
> * The response is cached/replicated in Flink state to bridge MDS downtime.
> * Since MDS is queried periodically, state is reconsolidated async to
> reflect the latest changes.
>
> My main confusion came from the fact that there are seemingly two
> sources of truth in the document. I take that the writer will never
> update the MDS but the MDS will update the writer.
>
> Best,
>
> Arvid
>
> On Mon, Mar 24, 2025 at 9:22 PM Őrhidi Mátyás <ma...@gmail.com>
> wrote:
> >
> > Hi Arvin,
> >
> > Thank you for taking time to look at this proposal.
> >
> > The concept of KafkaMetadataService is not new, it has been introduced
> with
> > Dynamic Kafka Source already:
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamic-kafka/#kafka-metadata-service
> >
> > This is an external component often hosted by Kafka infra teams (often
> as a
> > REST API). In essence, the Kafka Metadata Service (MDS) removes the
> burden
> > of managing Kafka connection details and infrastructure changes from the
> > Flink developer.
> >
> > - *No Hardcoding:* Instead of hardcoding Kafka broker addresses, topic
> > names, and other details directly into your Flink code, you can fetch
> them
> > dynamically from the MDS.
> > - *Dynamic Changes:* If the Kafka cluster changes (e.g., a broker is
> added
> > or removed, a topic is moved), the MDS reflects these changes. Your Flink
> > application, using the MDS, can automatically adapt without manual
> > intervention or redeployment.
> > - *Environment Agnostic:* You can easily switch between development,
> > testing, and production environments because the MDS provides the correct
> > Kafka details for each environment. Your code doesn't need
> > environment-specific configuration.
> > - *Easier Topic Management:* The MDS helps you discover available topics
> > and streams. You can find the right topic for your data without needing
> to
> > know the exact Kafka cluster details. You use a logical "stream ID," and
> > the MDS tells you the corresponding Kafka topic(s).
> >
> > Flink state becomes the "source of truth" after fetching this metadata
> from
> > the MDS, the initial source of truth is the MDS. The planned
> > DynamicKafkaWriter can cache and use Flink's state management to store
> the
> > mapping between logical streams and physical Kafka clusters/topics. The
> > writer uses a background thread and a reconciliation process to
> dynamically
> > update this state based on information from the MDS. This state is
> > checkpointed for fault tolerance and becomes the "source of truth." when
> > recovering its internal state.
> >
> > I hope this clarifies. Will fix the copy paste errors soon.
> >
> > Cheers,
> > Matyas
> >
> >
> > On Wed, Mar 19, 2025 at 8:13 AM Arvid Heise <ar...@apache.org> wrote:
> >
> > > Hi Matyas,
> > >
> > > could you please provide more details on the KafkaMetadataService?
> > > Where does it live? How does it work? How does "Metadata state will be
> > > stored in Flink state as the source of truth for the Flink job" work?
> > >
> > > Also nit: the test plan contains copy&paste errors.
> > >
> > > Best,
> > >
> > > Arvid
> > >
> > > On Thu, Mar 13, 2025 at 10:04 PM Őrhidi Mátyás <
> matyas.orhidi@gmail.com>
> > > wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion on FLIP-515: Dynamic Kafka Sink [1].
> This
> > > is
> > > > an addition to the existing Dynamic Kafka Source [2] to make the
> > > > functionality complete.
> > > >
> > > > Feel free to share your thoughts and suggestions to make this feature
> > > > better.
> > > >
> > > > + Mason Chen
> > > >
> > > > Thanks,
> > > > Matyas
> > > >
> > > > [1]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-515%3A+Dynamic+Kafka+Sink
> > > >
> > > > [2]
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320
> > >
>
Re: [DISCUSS] FLIP-515: Dynamic Kafka Sink
Posted by Hongshun Wang <lo...@gmail.com>.
Hi Matyas,
I also have another question.
This FLIP-515 seems based on FLIP-246 and then I read FLIP-246. It seems
that KafkaMetadataService is a key concept in the FLIP.
In this FLIP-515 and FLIP-246, KafkaMetadataServiceImpl is mentioned ,but
I didn't found it in flink-connector-kafka repo while I just saw a
SingleClusterTopicMetadataService which does nothing different than
before.Thus, it is hard for me to understand the overall flow.
*What's the plan for KafkaMetadataServiceImpl? now this code is just a toy
which cannot be used in production?*
What more, the source is updated with MetadataUpdateEvent. How is it going
in Sink? I really hope you can draw a diagram like what FLIP-246 does.
Best
hongshun
On Thu, Apr 3, 2025 at 1:54 PM Hongshun Wang <lo...@gmail.com>
wrote:
> Hi Matyas,
>
> Sorry for the late reply. Dynamic Kafka Source has been introduced for a
> long time, and it is a little confusing for developers like me
> that KafkaDynamicSource and DynamicKafkaSource have the same kafka
> connector repo for different things.
> Now DynamicKafkaSink will be introduced again. I wonder:
> 1. Is there any plan to merge this code into one class?
> 2. Is there any plan to expose them to SQL?
>
> Best
> Hongshun
>
> On Wed, Mar 26, 2025 at 2:48 PM Arvid Heise <ah...@confluent.io.invalid>
> wrote:
>
>> Hi Matyas,
>>
>> thanks for clarifying. The design LGTM.
>>
>> Based on your description, I'd phrase the state management a bit
>> differently:
>> * The source of truth is and always will be the MDS.
>> * The response is cached/replicated in Flink state to bridge MDS downtime.
>> * Since MDS is queried periodically, state is reconsolidated async to
>> reflect the latest changes.
>>
>> My main confusion came from the fact that there are seemingly two
>> sources of truth in the document. I take that the writer will never
>> update the MDS but the MDS will update the writer.
>>
>> Best,
>>
>> Arvid
>>
>> On Mon, Mar 24, 2025 at 9:22 PM Őrhidi Mátyás <ma...@gmail.com>
>> wrote:
>> >
>> > Hi Arvin,
>> >
>> > Thank you for taking time to look at this proposal.
>> >
>> > The concept of KafkaMetadataService is not new, it has been introduced
>> with
>> > Dynamic Kafka Source already:
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamic-kafka/#kafka-metadata-service
>> >
>> > This is an external component often hosted by Kafka infra teams (often
>> as a
>> > REST API). In essence, the Kafka Metadata Service (MDS) removes the
>> burden
>> > of managing Kafka connection details and infrastructure changes from the
>> > Flink developer.
>> >
>> > - *No Hardcoding:* Instead of hardcoding Kafka broker addresses, topic
>> > names, and other details directly into your Flink code, you can fetch
>> them
>> > dynamically from the MDS.
>> > - *Dynamic Changes:* If the Kafka cluster changes (e.g., a broker is
>> added
>> > or removed, a topic is moved), the MDS reflects these changes. Your
>> Flink
>> > application, using the MDS, can automatically adapt without manual
>> > intervention or redeployment.
>> > - *Environment Agnostic:* You can easily switch between development,
>> > testing, and production environments because the MDS provides the
>> correct
>> > Kafka details for each environment. Your code doesn't need
>> > environment-specific configuration.
>> > - *Easier Topic Management:* The MDS helps you discover available topics
>> > and streams. You can find the right topic for your data without
>> needing to
>> > know the exact Kafka cluster details. You use a logical "stream ID,"
>> and
>> > the MDS tells you the corresponding Kafka topic(s).
>> >
>> > Flink state becomes the "source of truth" after fetching this metadata
>> from
>> > the MDS, the initial source of truth is the MDS. The planned
>> > DynamicKafkaWriter can cache and use Flink's state management to store
>> the
>> > mapping between logical streams and physical Kafka clusters/topics. The
>> > writer uses a background thread and a reconciliation process to
>> dynamically
>> > update this state based on information from the MDS. This state is
>> > checkpointed for fault tolerance and becomes the "source of truth." when
>> > recovering its internal state.
>> >
>> > I hope this clarifies. Will fix the copy paste errors soon.
>> >
>> > Cheers,
>> > Matyas
>> >
>> >
>> > On Wed, Mar 19, 2025 at 8:13 AM Arvid Heise <ar...@apache.org> wrote:
>> >
>> > > Hi Matyas,
>> > >
>> > > could you please provide more details on the KafkaMetadataService?
>> > > Where does it live? How does it work? How does "Metadata state will be
>> > > stored in Flink state as the source of truth for the Flink job" work?
>> > >
>> > > Also nit: the test plan contains copy&paste errors.
>> > >
>> > > Best,
>> > >
>> > > Arvid
>> > >
>> > > On Thu, Mar 13, 2025 at 10:04 PM Őrhidi Mátyás <
>> matyas.orhidi@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi devs,
>> > > >
>> > > > I'd like to start a discussion on FLIP-515: Dynamic Kafka Sink [1].
>> This
>> > > is
>> > > > an addition to the existing Dynamic Kafka Source [2] to make the
>> > > > functionality complete.
>> > > >
>> > > > Feel free to share your thoughts and suggestions to make this
>> feature
>> > > > better.
>> > > >
>> > > > + Mason Chen
>> > > >
>> > > > Thanks,
>> > > > Matyas
>> > > >
>> > > > [1]
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-515%3A+Dynamic+Kafka+Sink
>> > > >
>> > > > [2]
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320
>> > >
>>
>