You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Sucheth S <su...@gmail.com> on 2022/06/12 04:53:13 UTC

Apache flink doesn't work with avro kafka topic with multiple event types

Hi,

Apache Flink doesn't work with Avro serialized kafka topic with multiple
event types (
TopicRecordNameStrategy for subject )

Is there a way to read a generic record from avro serialized kafka topic
which can have messages with different schemas, basically
TopicRecordNameStrategy
for the subject in the schema registry. ?


Regards,
Sucheth Shivakumar
website : https://sucheths.com
mobile : +1(650)-576-8050
San Mateo, United States

Re: Apache flink doesn't work with avro kafka topic with multiple event types

Posted by Sucheth S <su...@gmail.com>.

Thanks, I'll check it out.

On Mon, Jun 13, 2022 at 2:40 AM Qingsheng Ren <re...@gmail.com> wrote:

> Hi Sucheth,
>
> If you are referring to Table / SQL API, I'm afraid it doesn't support
> schema evolution or different types from one Kafka table. An
> alternative way is to consume the topic with raw format [1] and do
> deserialization with a UDTF. If you are using the DataStream API, you
> can implement the KafkaRecordDeserializationSchema interface to
> deserialize the message consumed from Kafka.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/raw/
>
> Best regards,
>
> Qingsheng
>
> On Sun, Jun 12, 2022 at 12:53 PM Sucheth S <su...@gmail.com> wrote:
> >
> > Hi,
> >
> > Apache Flink doesn't work with Avro serialized kafka topic with multiple
> event types (
> > TopicRecordNameStrategy for subject )
> >
> > Is there a way to read a generic record from avro serialized kafka topic
> which can have messages with different schemas, basically
> TopicRecordNameStrategy for the subject in the schema registry. ?
> >
> >
> > Regards,
> > Sucheth Shivakumar
> > website : https://sucheths.com
> > mobile : +1(650)-576-8050
> > San Mateo, United States
>
-- 
Regards,
Sucheth Shivakumar
website: https://sucheths.com
Mobile : +1(650)-576-8050
San Mateo, United States

Fwd: Apache flink doesn't work with avro kafka topic with multiple event types

Posted by Qingsheng Ren <re...@gmail.com>.

Hi Sucheth,

If you are referring to Table / SQL API, I'm afraid it doesn't support
schema evolution or different types from one Kafka table. An
alternative way is to consume the topic with raw format [1] and do
deserialization with a UDTF. If you are using the DataStream API, you
can implement the KafkaRecordDeserializationSchema interface to
deserialize the message consumed from Kafka.

[1] https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/raw/

Best regards,

Qingsheng

On Sun, Jun 12, 2022 at 12:53 PM Sucheth S <su...@gmail.com> wrote:
>
> Hi,
>
> Apache Flink doesn't work with Avro serialized kafka topic with multiple event types (
> TopicRecordNameStrategy for subject )
>
> Is there a way to read a generic record from avro serialized kafka topic which can have messages with different schemas, basically TopicRecordNameStrategy for the subject in the schema registry. ?
>
>
> Regards,
> Sucheth Shivakumar
> website : https://sucheths.com
> mobile : +1(650)-576-8050
> San Mateo, United States