You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Steve Niemitz (Jira)" <ji...@apache.org> on 2020/07/01 01:32:00 UTC

[jira] [Created] (BEAM-10395) Dataflow runner should deduplicate files to stage by destination

Steve Niemitz created BEAM-10395:
------------------------------------

             Summary: Dataflow runner should deduplicate files to stage by destination 
                 Key: BEAM-10395
                 URL: https://issues.apache.org/jira/browse/BEAM-10395
             Project: Beam
          Issue Type: Improvement
          Components: runner-dataflow
            Reporter: Steve Niemitz
            Assignee: Steve Niemitz


If a pipeline contains multiple files with the same destination path, the dataflow runner will try to stage them both in parallel, resulting in the upload usually failing (due to conflicting uploads).

The runner should only upload one file per destination, and ideally check that the sources are the same as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)