You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Stephan Ewen <se...@apache.org> on 2017/07/03 15:18:15 UTC

Re: Kafka and Flink integration

Hi Urs!

Inside Flink (between Flink operators)
  - Kryo is not a problem, but types must be registered up front for good
performance
  - Tuples and POJOs are often faster than the types that fall back to Kryo

Persistent-storage (HDFS, Kafka, ...)
  - Kryo is not recommended, because its binary data format is not stable.
It changes between major Kryo versions and between Kryo setups with
different type registrations.
  - A stable format with schema evolution support (Avro, Thrift, ...) is
recommended here.



On Thu, Jun 22, 2017 at 9:28 AM, Urs Schoenenberger <
urs.schoenenberger@tngtech.com> wrote:

> Hi Greg,
>
> do you have a link where I could read up on the rationale behind
> avoiding Kryo? I'm currently facing a similar decision and would like to
> get some more background on this.
>
> Thank you very much,
> Urs
>
> On 21.06.2017 12:10, Greg Hogan wrote:
> > The recommendation has been to avoid Kryo where possible.
> >
> > General data exchange: avro or thrift.
> >
> > Flink internal data exchange: POJO (or Tuple, which are slightly faster
> though less readable, and there is an outstanding PR to narrow or close the
> performance gap).
> >
> > Kryo is useful for types which cannot be modified to be a POJO. There
> are also cases where Kryo must be used because Flink has insufficient
> TypeInformation, such as when returning an interface or abstract type when
> the actual concrete type can be known.
> >
> >
> >
> >> On Jun 21, 2017, at 3:19 AM, nragon <nuno.goncalves@
> wedotechnologies.com> wrote:
> >>
> >> So, serialization between producer application -> kafka -> flink kafka
> >> consumer will use avro, thrift or kryo right? From there, the remaining
> >> pipeline can just use standard pojo serialization, which would be
> better?
> >>
> >>
> >>
> >> --
> >> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-
> integration-tp13792p13885.html
> >> Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
> >
>
> --
> Urs Schönenberger - urs.schoenenberger@tngtech.com
>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>

Re: Kafka and Flink integration

Posted by Konstantin Knauf <ko...@tngtech.com>.
Hi Jürgen,

one easy way is to disable the Kryo fallback with

env.getConfig().disableGenericTypes();

If it was using Kryo you should see an exception, which also states the
class, for which it needs to fallback to Kryo. This fails on the first
non-Kryo class though. So depending on the other classes, you might need
a minimal job using the Class to test it this way.

Best,

Konstantin


On 05.07.2017 10:35, Jürgen Thomann wrote:
> Hi Stephan,
> 
> do you know an easy way to find out if Kryo or POJO is used? I have an
> Object that would be a POJO, but it has one field that uses an object
> without a public no argument constructor. As I understood the
> documentation, this should result in Kryo being used.
> 
> Thanks,
> Jürgen
> 
> On 03.07.2017 17:18, Stephan Ewen wrote:
>> Hi Urs!
>>
>> Inside Flink (between Flink operators)
>>   - Kryo is not a problem, but types must be registered up front for
>> good performance
>>   - Tuples and POJOs are often faster than the types that fall back to
>> Kryo
>>
>> Persistent-storage (HDFS, Kafka, ...)
>>   - Kryo is not recommended, because its binary data format is not
>> stable. It changes between major Kryo versions and between Kryo setups
>> with different type registrations.
>>   - A stable format with schema evolution support (Avro, Thrift, ...)
>> is recommended here.
>>
>>
>>
>> On Thu, Jun 22, 2017 at 9:28 AM, Urs Schoenenberger
>> <urs.schoenenberger@tngtech.com
>> <ma...@tngtech.com>> wrote:
>>
>>     Hi Greg,
>>
>>     do you have a link where I could read up on the rationale behind
>>     avoiding Kryo? I'm currently facing a similar decision and would
>>     like to
>>     get some more background on this.
>>
>>     Thank you very much,
>>     Urs
>>
>>     On 21.06.2017 12:10, Greg Hogan wrote:
>>     > The recommendation has been to avoid Kryo where possible.
>>     >
>>     > General data exchange: avro or thrift.
>>     >
>>     > Flink internal data exchange: POJO (or Tuple, which are slightly
>>     faster though less readable, and there is an outstanding PR to
>>     narrow or close the performance gap).
>>     >
>>     > Kryo is useful for types which cannot be modified to be a POJO.
>>     There are also cases where Kryo must be used because Flink has
>>     insufficient TypeInformation, such as when returning an interface
>>     or abstract type when the actual concrete type can be known.
>>     >
>>     >
>>     >
>>     >> On Jun 21, 2017, at 3:19 AM, nragon
>>     <nuno.goncalves@wedotechnologies.com
>>     <ma...@wedotechnologies.com>> wrote:
>>     >>
>>     >> So, serialization between producer application -> kafka ->
>>     flink kafka
>>     >> consumer will use avro, thrift or kryo right? From there, the
>>     remaining
>>     >> pipeline can just use standard pojo serialization, which would
>>     be better?
>>     >>
>>     >>
>>     >>
>>     >> --
>>     >> View this message in context:
>>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html
>>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html>
>>     >> Sent from the Apache Flink User Mailing List archive. mailing
>>     list archive at Nabble.com.
>>     >
>>
>>     --
>>     Urs Schönenberger - urs.schoenenberger@tngtech.com
>>     <ma...@tngtech.com>
>>
>>     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>>     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>>     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>>
>>
> 
> -- 
> Jürgen Thomann
> System Administrator
> 
> 
> InnoGames GmbH
> Friesenstraße 13 - 20097 Hamburg - Germany
> Tel +49 40 7889335-0
> 
> Managing Directors: Hendrik Klindworth, Michael Zillmer
> VAT-ID: DE264068907 Amtsgericht Hamburg, HRB 108973
> 
> http://www.innogames.com – juergen.thomann@innogames.com
> 

-- 
Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082


Re: Kafka and Flink integration

Posted by Jürgen Thomann <ju...@innogames.com>.
Hi Stephan,

do you know an easy way to find out if Kryo or POJO is used? I have an 
Object that would be a POJO, but it has one field that uses an object 
without a public no argument constructor. As I understood the 
documentation, this should result in Kryo being used.

Thanks,
Jürgen

On 03.07.2017 17:18, Stephan Ewen wrote:
> Hi Urs!
>
> Inside Flink (between Flink operators)
>   - Kryo is not a problem, but types must be registered up front for 
> good performance
>   - Tuples and POJOs are often faster than the types that fall back to 
> Kryo
>
> Persistent-storage (HDFS, Kafka, ...)
>   - Kryo is not recommended, because its binary data format is not 
> stable. It changes between major Kryo versions and between Kryo setups 
> with different type registrations.
>   - A stable format with schema evolution support (Avro, Thrift, ...) 
> is recommended here.
>
>
>
> On Thu, Jun 22, 2017 at 9:28 AM, Urs Schoenenberger 
> <urs.schoenenberger@tngtech.com 
> <ma...@tngtech.com>> wrote:
>
>     Hi Greg,
>
>     do you have a link where I could read up on the rationale behind
>     avoiding Kryo? I'm currently facing a similar decision and would
>     like to
>     get some more background on this.
>
>     Thank you very much,
>     Urs
>
>     On 21.06.2017 12:10, Greg Hogan wrote:
>     > The recommendation has been to avoid Kryo where possible.
>     >
>     > General data exchange: avro or thrift.
>     >
>     > Flink internal data exchange: POJO (or Tuple, which are slightly
>     faster though less readable, and there is an outstanding PR to
>     narrow or close the performance gap).
>     >
>     > Kryo is useful for types which cannot be modified to be a POJO.
>     There are also cases where Kryo must be used because Flink has
>     insufficient TypeInformation, such as when returning an interface
>     or abstract type when the actual concrete type can be known.
>     >
>     >
>     >
>     >> On Jun 21, 2017, at 3:19 AM, nragon
>     <nuno.goncalves@wedotechnologies.com
>     <ma...@wedotechnologies.com>> wrote:
>     >>
>     >> So, serialization between producer application -> kafka ->
>     flink kafka
>     >> consumer will use avro, thrift or kryo right? From there, the
>     remaining
>     >> pipeline can just use standard pojo serialization, which would
>     be better?
>     >>
>     >>
>     >>
>     >> --
>     >> View this message in context:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html>
>     >> Sent from the Apache Flink User Mailing List archive. mailing
>     list archive at Nabble.com.
>     >
>
>     --
>     Urs Schönenberger - urs.schoenenberger@tngtech.com
>     <ma...@tngtech.com>
>
>     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

-- 
Jürgen Thomann
System Administrator


InnoGames GmbH
Friesenstraße 13 - 20097 Hamburg - Germany
Tel +49 40 7889335-0

Managing Directors: Hendrik Klindworth, Michael Zillmer
VAT-ID: DE264068907 Amtsgericht Hamburg, HRB 108973

http://www.innogames.com – juergen.thomann@innogames.com