[jira] [Created] (BEAM-3408) PackageUtil should retry check for whether file is already staged

["Henning Rohde (JIRA)" <ji...@apache.org>]

Henning Rohde created BEAM-3408:
-----------------------------------

             Summary: PackageUtil should retry check for whether file is already staged
                 Key: BEAM-3408
                 URL: https://issues.apache.org/jira/browse/BEAM-3408
             Project: Beam
          Issue Type: Bug
          Components: runner-dataflow
            Reporter: Henning Rohde
            Assignee: Thomas Groh


The code to stage files retries the upload, but the existence (and size) check is attempted only once:

https://github.com/apache/beam/blob/dc3a664e3b4b4f63500954aae25dce0cb5f8b314/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L193

This causes post-commit tests to fail sometimes:

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4632/testReport/junit/org.apache.beam.sdk.io/BoundedReadFromUnboundedSourceTest/testDedup/

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/org.apache.beam$beam-runners-google-cloud-dataflow-java/4636/testReport/junit/org.apache.beam.sdk.transforms/ViewTest/testEmptyMultimapSideInput/

We should retry this check as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)