You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Mark Liu (Jira)" <ji...@apache.org> on 2019/09/09 19:04:00 UTC
[jira] [Comment Edited] (BEAM-5775) Make the spark runner not
serialize data unless spark is spilling to disk
[ https://issues.apache.org/jira/browse/BEAM-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926005#comment-16926005 ]
Mark Liu edited comment on BEAM-5775 at 9/9/19 7:03 PM:
--------------------------------------------------------
Is this issue on track for 2.16 release? (branch will be cut in 3 days)
was (Author: markflyhigh):
Is this issue on trackf or 2.16 release? (branch will be cut in 3 days)
> Make the spark runner not serialize data unless spark is spilling to disk
> -------------------------------------------------------------------------
>
> Key: BEAM-5775
> URL: https://issues.apache.org/jira/browse/BEAM-5775
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Mike Kaplinskiy
> Assignee: Mike Kaplinskiy
> Priority: Minor
> Fix For: 2.16.0
>
> Time Spent: 11h 50m
> Remaining Estimate: 0h
>
> Currently for storage level MEMORY_ONLY, Beam does not coder-ify the data. This lets Spark keep the data in memory avoiding the serialization round trip. Unfortunately the logic is fairly coarse - as soon as you switch to MEMORY_AND_DISK, Beam coder-ifys the data even though Spark might have chosen to keep the data in memory, incurring the serialization overhead.
>
> Ideally Beam would serialize the data lazily - as Spark chooses to spill to disk. This would be a change in behavior when using beam, but luckily Spark has a solution for folks that want data serialized in memory - MEMORY_AND_DISK_SER will keep the data serialized.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)