You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Prasanna kumar <pr...@gmail.com> on 2020/12/02 16:28:04 UTC

Routing events to different kafka topics dynamically

Hi,

Events need to be routed to different kafka topics dynamically based upon
some info in the message.

We have implemented using KeyedSerializationSchema similar to
https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even.
But its deprecated and we cannot use it for production.

I looked at the alternative KafkaSerializationSchema but there i do not see
an option there.

Then i stumbled upon this
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html.
which asks us to use KafkaContextAware.

Is there a more intuitive/easier way to do the same ?

Thanks,
Prasanna.

Re: Routing events to different kafka topics dynamically

Posted by Till Rohrmann <tr...@apache.org>.

Great to hear :-)

Cheers,
Till

On Thu, Dec 3, 2020 at 10:15 AM Prasanna kumar <
prasannakumarramani@gmail.com> wrote:

> Thanks Till,
>
> Able to deduce topics by extending the KafkaSerializarion Schema class.
>
> Prasanna.
>
> On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Prasanna,
>>
>> I believe that what Aljoscha suggestd in the linked discussion is still
>> the best way to go forward. Given your description of the problem this
>> should actually be pretty straightforward as you can deduce the topic from
>> the message. Hence, you just need to create the ProducerRecord with the
>> right target topic you extracted from the record/message.
>>
>> Cheers,
>> Till
>>
>> On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <
>> prasannakumarramani@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Events need to be routed to different kafka topics dynamically based
>>> upon some info in the message.
>>>
>>> We have implemented using KeyedSerializationSchema similar to
>>> https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even.
>>> But its deprecated and we cannot use it for production.
>>>
>>> I looked at the alternative KafkaSerializationSchema but there i do not
>>> see an option there.
>>>
>>> Then i stumbled upon this
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html.
>>> which asks us to use KafkaContextAware.
>>>
>>> Is there a more intuitive/easier way to do the same ?
>>>
>>> Thanks,
>>> Prasanna.
>>>
>>>
>>>

Re: Routing events to different kafka topics dynamically

Posted by Prasanna kumar <pr...@gmail.com>.

Thanks Till,

Able to deduce topics by extending the KafkaSerializarion Schema class.

Prasanna.

On Wed, Dec 2, 2020 at 11:18 PM Till Rohrmann <tr...@apache.org> wrote:

> Hi Prasanna,
>
> I believe that what Aljoscha suggestd in the linked discussion is still
> the best way to go forward. Given your description of the problem this
> should actually be pretty straightforward as you can deduce the topic from
> the message. Hence, you just need to create the ProducerRecord with the
> right target topic you extracted from the record/message.
>
> Cheers,
> Till
>
> On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <
> prasannakumarramani@gmail.com> wrote:
>
>> Hi,
>>
>> Events need to be routed to different kafka topics dynamically based upon
>> some info in the message.
>>
>> We have implemented using KeyedSerializationSchema similar to
>> https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even.
>> But its deprecated and we cannot use it for production.
>>
>> I looked at the alternative KafkaSerializationSchema but there i do not
>> see an option there.
>>
>> Then i stumbled upon this
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html.
>> which asks us to use KafkaContextAware.
>>
>> Is there a more intuitive/easier way to do the same ?
>>
>> Thanks,
>> Prasanna.
>>
>>
>>

Re: Routing events to different kafka topics dynamically

Posted by Till Rohrmann <tr...@apache.org>.

Hi Prasanna,

I believe that what Aljoscha suggestd in the linked discussion is still the
best way to go forward. Given your description of the problem this should
actually be pretty straightforward as you can deduce the topic from the
message. Hence, you just need to create the ProducerRecord with the right
target topic you extracted from the record/message.

Cheers,
Till

On Wed, Dec 2, 2020 at 5:28 PM Prasanna kumar <pr...@gmail.com>
wrote:

> Hi,
>
> Events need to be routed to different kafka topics dynamically based upon
> some info in the message.
>
> We have implemented using KeyedSerializationSchema similar to
> https://stackoverflow.com/questions/49508508/apache-flink-how-to-sink-events-to-different-kafka-topics-depending-on-the-even.
> But its deprecated and we cannot use it for production.
>
> I looked at the alternative KafkaSerializationSchema but there i do not
> see an option there.
>
> Then i stumbled upon this
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Usage-of-KafkaDeserializationSchema-and-KafkaSerializationSchema-td32347.html.
> which asks us to use KafkaContextAware.
>
> Is there a more intuitive/easier way to do the same ?
>
> Thanks,
> Prasanna.
>
>
>