[jira] [Created] (BEAM-848) Make post-read (unbounded) shuffle use coders instead of Kryo.

["Amit Sela (JIRA)" <ji...@apache.org>]

Amit Sela created BEAM-848:
------------------------------

             Summary: Make post-read (unbounded) shuffle use coders instead of Kryo.
                 Key: BEAM-848
                 URL: https://issues.apache.org/jira/browse/BEAM-848
             Project: Beam
          Issue Type: Bug
          Components: runner-spark
            Reporter: Amit Sela
            Assignee: Amit Sela


The SparkRunner uses {{mapWithState}} to read and manage CheckpointMarks, and this stateful operation will be followed by a shuffle: 
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/MapWithStateDStream.scala#L159

It would be best to use coders here: https://github.com/apache/incubator-beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java#L71 in order to get an Iterator<byte[]> and decode afterwards.

This method is preferred for two reasons:
1. Known coding should be faster then Kryo.
2. Users are already required to explicitly use coders when authoring a Pipeline and the runner should avoid adding complexity of "registering" additional serializers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)