You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2014/07/17 19:15:06 UTC
[jira] [Commented] (FLINK-610) Replace Avro serialization by Kryo
[ https://issues.apache.org/jira/browse/FLINK-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14065153#comment-14065153 ]
ASF GitHub Bot commented on FLINK-610:
--------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/incubator-flink/pull/74
[FLINK-610] Added KryoSerializer
I added the KryoSerializer and replaced the AvroSerializer as the standard generic type serializer. Due to the way Flink's serialization works, we cannot exploit Kryo's Inputs which buffer data for serialization. Instead, we have to read byte after byte if we do not know the length of the underlying data. This has consequences for variable length encodings. Kryo will fall back to a slow deserialization of these types, since it will read only one byte at a time. I do not know the implications of this on Kryo's performance. Therefore, I did not remove Avro completely yet. Furthermore, Kryo should be tested on the cluster and if possible its performance should be compared to Avro.
We could mitigate the problem if we don't use variable length encodings for types such as ```int```s and ```long```s. This, however, comes at the price of larger serialized objects. For UTF8 strings, the problem would persist.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/incubator-flink FLINK-610
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/74.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #74
----
commit 0c618c20bc7be64de82e0e7d3f7734e8c68593bb
Author: Till Rohrmann <ti...@gmail.com>
Date: 2014-07-17T15:21:59Z
Added KryoSerializer
----
> Replace Avro serialization by Kryo
> -----------------------------------
>
> Key: FLINK-610
> URL: https://issues.apache.org/jira/browse/FLINK-610
> Project: Flink
> Issue Type: Bug
> Components: Java API
> Reporter: GitHub Import
> Priority: Critical
> Labels: github-import
> Fix For: pre-apache
>
>
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/stratosphere/stratosphere/issues/610
> Created by: [rmetzger|https://github.com/rmetzger]
> Labels: java api,
> Milestone: Release 0.6 (unplanned)
> Created at: Tue Mar 18 17:29:28 CET 2014
> State: open
--
This message was sent by Atlassian JIRA
(v6.2#6252)