You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kyle Weaver (Jira)" <ji...@apache.org> on 2019/09/20 21:47:00 UTC
[jira] [Resolved] (BEAM-7600) Spark portable runner: reuse SDK
harness
[ https://issues.apache.org/jira/browse/BEAM-7600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kyle Weaver resolved BEAM-7600.
-------------------------------
Fix Version/s: 2.16.0
Resolution: Fixed
> Spark portable runner: reuse SDK harness
> ----------------------------------------
>
> Key: BEAM-7600
> URL: https://issues.apache.org/jira/browse/BEAM-7600
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-spark
> Fix For: 2.16.0
>
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> Right now, we're creating a new SDK harness every time an executable stage is run [1], which is expensive. We should be able to re-use code from the Flink runner to re-use the SDK harness [2].
>
> [1] [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageFunction.java#L135]
> [2] [https://github.com/apache/beam/blob/c9fb261bc7666788402840bb6ce1b0ce2fd445d1/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)