You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Alex Cruise <al...@cluonflux.com> on 2021/09/14 18:32:29 UTC

Fast serialization for Kotlin data classes

Hi there,

I appreciate the fact that Flink has built-in support for making POJO and
Scala `case class` serialization faster, but in my project we use immutable
Kotlin `data class`es (analogous to Scala `case class`es) extensively, and
we'd really prefer not to make them POJOs, mostly for style/taste reasons
(e.g. need a default constructor and setters, both are anathema!)

Does anyone know of a good way for us to keep using idiomatic, immutable
Kotlin data classes, but to get much faster serialization performance in
Flink?

Thanks!

-0xe1a

Re: Fast serialization for Kotlin data classes

Posted by Matthias Pohl <ma...@ververica.com>.
True, that's a valid concern you raised here, Alexis. Thanks for pointing
that out.

On Thu, Sep 16, 2021 at 1:58 PM Alexis Sarda-Espinosa <
alexis.sarda-espinosa@microfocus.com> wrote:

> Someone please correct me if I’m wrong but, until FLINK-16686 [1] is
> fixed, a class must be a POJO to be used in managed state with RocksDB,
> right? That’s not to say that the approach with TypeInfoFactory won’t work,
> just that even then it will mean none of the data classes can be used for
> managed state.
>
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-16686
>
>
>
> Regards,
>
> Alexis.
>
>
>
> *From:* Matthias Pohl <ma...@ververica.com>
> *Sent:* Donnerstag, 16. September 2021 13:12
> *To:* Alex Cruise <al...@cluonflux.com>
> *Cc:* Flink ML <us...@flink.apache.org>
> *Subject:* Re: Fast serialization for Kotlin data classes
>
>
>
> Hi Alex,
>
> have you had a look at TypeInfoFactory? That might be the best way to come
> up with a custom serialization mechanism. See the docs [1] for further
> details.
>
>
>
> Best,
> Matthias
>
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#defining-type-information-using-a-factory
>
>
>
> On Tue, Sep 14, 2021 at 8:33 PM Alex Cruise <al...@cluonflux.com> wrote:
>
> Hi there,
>
>
>
> I appreciate the fact that Flink has built-in support for making POJO and
> Scala `case class` serialization faster, but in my project we use immutable
> Kotlin `data class`es (analogous to Scala `case class`es) extensively, and
> we'd really prefer not to make them POJOs, mostly for style/taste reasons
> (e.g. need a default constructor and setters, both are anathema!)
>
>
>
> Does anyone know of a good way for us to keep using idiomatic, immutable
> Kotlin data classes, but to get much faster serialization performance in
> Flink?
>
>
>
> Thanks!
>
>
>
> -0xe1a
>
>

RE: Fast serialization for Kotlin data classes

Posted by Alexis Sarda-Espinosa <al...@microfocus.com>.
Someone please correct me if I’m wrong but, until FLINK-16686 [1] is fixed, a class must be a POJO to be used in managed state with RocksDB, right? That’s not to say that the approach with TypeInfoFactory won’t work, just that even then it will mean none of the data classes can be used for managed state.

[1] https://issues.apache.org/jira/browse/FLINK-16686

Regards,
Alexis.

From: Matthias Pohl <ma...@ververica.com>
Sent: Donnerstag, 16. September 2021 13:12
To: Alex Cruise <al...@cluonflux.com>
Cc: Flink ML <us...@flink.apache.org>
Subject: Re: Fast serialization for Kotlin data classes

Hi Alex,
have you had a look at TypeInfoFactory? That might be the best way to come up with a custom serialization mechanism. See the docs [1] for further details.

Best,
Matthias

[1] https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#defining-type-information-using-a-factory

On Tue, Sep 14, 2021 at 8:33 PM Alex Cruise <al...@cluonflux.com>> wrote:
Hi there,

I appreciate the fact that Flink has built-in support for making POJO and Scala `case class` serialization faster, but in my project we use immutable Kotlin `data class`es (analogous to Scala `case class`es) extensively, and we'd really prefer not to make them POJOs, mostly for style/taste reasons (e.g. need a default constructor and setters, both are anathema!)

Does anyone know of a good way for us to keep using idiomatic, immutable Kotlin data classes, but to get much faster serialization performance in Flink?

Thanks!

-0xe1a

Re: Fast serialization for Kotlin data classes

Posted by Matthias Pohl <ma...@ververica.com>.
Hi Alex,
have you had a look at TypeInfoFactory? That might be the best way to come
up with a custom serialization mechanism. See the docs [1] for further
details.

Best,
Matthias

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/types_serialization/#defining-type-information-using-a-factory

On Tue, Sep 14, 2021 at 8:33 PM Alex Cruise <al...@cluonflux.com> wrote:

> Hi there,
>
> I appreciate the fact that Flink has built-in support for making POJO and
> Scala `case class` serialization faster, but in my project we use immutable
> Kotlin `data class`es (analogous to Scala `case class`es) extensively, and
> we'd really prefer not to make them POJOs, mostly for style/taste reasons
> (e.g. need a default constructor and setters, both are anathema!)
>
> Does anyone know of a good way for us to keep using idiomatic, immutable
> Kotlin data classes, but to get much faster serialization performance in
> Flink?
>
> Thanks!
>
> -0xe1a
>