You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Axel Magnuson (JIRA)" <ji...@apache.org> on 2018/03/05 22:09:00 UTC
[jira] [Commented] (BEAM-3672) FlinkRunner: Implement an Artifact
service using the Flink DistributedCache
[ https://issues.apache.org/jira/browse/BEAM-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386828#comment-16386828 ]
Axel Magnuson commented on BEAM-3672:
-------------------------------------
Although this is mostly implemented, the current artifact service handles strings naively and would likely fail when receiving strings that represent illegal input for file paths. We plan on replacing this path resolution with a filepath-safe encoding such as base64, but this is a low priority item that has not been tackled yet. Known affected files are:
# /runners/local-artifact-service-java/src/main/java/*
# /runners/flink/src/main/java/org/apache/beam/runners/flink/execution/FlinkArtifactSource.java
# /runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkCachedArtifactPaths.java
> FlinkRunner: Implement an Artifact service using the Flink DistributedCache
> ---------------------------------------------------------------------------
>
> Key: BEAM-3672
> URL: https://issues.apache.org/jira/browse/BEAM-3672
> Project: Beam
> Issue Type: Sub-task
> Components: runner-flink
> Reporter: Ben Sidhom
> Assignee: Aljoscha Krettek
> Priority: Major
>
> We need to have a DistributedCache-based artifact service to ship with the portable Flink runner. The DistributedCache is a perfect fit for Flink because it comes for free and is the mechanism that Flink already uses to distribute its own artifacts.
>
> The final artifact service implementation should be pluggable, but using the DistributedCache allows the Flink runner to work without additional external dependencies (beyond perhaps Docker).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)