You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kyle Weaver (Jira)" <ji...@apache.org> on 2019/10/09 17:53:00 UTC
[jira] [Resolved] (BEAM-8312) Flink portable pipeline jars do not
need to stage artifacts remotely
[ https://issues.apache.org/jira/browse/BEAM-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kyle Weaver resolved BEAM-8312.
-------------------------------
Fix Version/s: 2.17.0
Resolution: Fixed
> Flink portable pipeline jars do not need to stage artifacts remotely
> --------------------------------------------------------------------
>
> Key: BEAM-8312
> URL: https://issues.apache.org/jira/browse/BEAM-8312
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-flink
> Fix For: 2.17.0
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Currently, Flink job jars re-stage all artifacts at runtime (on the JobManager) by using the usual BeamFileSystemArtifactRetrievalService [1]. However, since the manifest and all the artifacts live on the classpath of the jar, and everything from the classpath is copied to the Flink workers anyway [2], it should not be necessary to do additional artifact staging. We could replace BeamFileSystemArtifactRetrievalService in this case with a simple ArtifactRetrievalService that just pulls the artifacts from the classpath.
>
> [1] [https://github.com/apache/beam/blob/340c3202b1e5824b959f5f9f626e4c7c7842a3cb/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java]
> [2] [https://github.com/apache/beam/blob/2f1b56ccc506054e40afe4793a8b556e872e1865/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L93]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)