You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Wouter Zorgdrager <W....@tudelft.nl> on 2019/02/07 10:04:22 UTC

Avro serialization and deserialization to Kafka in Scala

Hello all,


I saw the recent updates in Flink related to supporting Avro schema evolution in state. I'm curious how Flink handles this internally for Scala case classes. I'm working on custom (de-)serialization schema's to write and read from Kafka. However, I'm currently stuck because of the fact that Avro doesn't natively support Scala. This means that in order to support case class serialization using Scala specific types (like Option, Either, etc.) I need a library like Avro4s [1] or AvroHugger [2] which on compile-time generates schemas using macros. These macro-extensions are extremely slow for complex case classes (compile-time of 15 minutes for a few nested types). I'm looking for an approach without the use of these libraries and therefore curious how Flink handles this.


Does anyone has some good leads for this?


Thanks in advance!


Kind regards,
Wouter Zorgdrager


[1] https://github.com/sksamuel/avro4s
[2] https://github.com/julianpeeters/avrohugger

Re: Avro serialization and deserialization to Kafka in Scala

Posted by Kostas Kloudas <k....@da-platform.com>.

Hi Wouter,

I think Gordon or Igal are the best to answer this question.

Cheers,
Kostas

On Thu, Feb 7, 2019 at 11:04 AM Wouter Zorgdrager <W....@tudelft.nl>
wrote:

> Hello all,
>
>
> I saw the recent updates in Flink related to supporting Avro schema
> evolution in state. I'm curious how Flink handles this internally for Scala
> case classes. I'm working on custom (de-)serialization schema's to write
> and read from Kafka. However, I'm currently stuck because of the fact that
> Avro doesn't natively support Scala. This means that in order to support
> case class serialization using Scala specific types (like Option, Either,
> etc.) I need a library like Avro4s [1] or AvroHugger [2] which on
> compile-time generates schemas using macros. These macro-extensions are
> extremely slow for complex case classes (compile-time of 15 minutes for a
> few nested types). I'm looking for an approach without the use of these
> libraries and therefore curious how Flink handles this.
>
>
> Does anyone has some good leads for this?
>
>
> Thanks in advance!
>
>
> Kind regards,
> Wouter Zorgdrager
>
>
> [1] https://github.com/sksamuel/avro4s
> [2] https://github.com/julianpeeters/avrohugger
>
>