You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by ke...@apache.org on 2017/10/18 20:09:19 UTC

[7/7] beam git commit: This closes #3977: [BEAM-2963] Stage pipeline in DataflowRunner

This closes #3977: [BEAM-2963] Stage pipeline in DataflowRunner

  Remove duplicate mocking in DataflowRunnerTest
  Add assertion that valid jobs must have staged pipeline
  Stage the pipeline without using a temp file
  Stage the portable pipeline in Dataflow
  Add ability to stage explicit file list
  Improve GcsFileSystem errors messages slightly


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1974b920
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1974b920
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1974b920

Branch: refs/heads/master
Commit: 1974b920e4b3bbe8549e25fe789f9dada13c1769
Parents: 2acdc74 128f3a6
Author: Kenneth Knowles <kl...@google.com>
Authored: Wed Oct 18 13:08:50 2017 -0700
Committer: Kenneth Knowles <kl...@google.com>
Committed: Wed Oct 18 13:08:50 2017 -0700

----------------------------------------------------------------------
 .../beam/runners/dataflow/DataflowRunner.java   |  15 ++-
 .../beam/runners/dataflow/util/GcsStager.java   |  53 +++++++--
 .../beam/runners/dataflow/util/PackageUtil.java | 116 ++++++++++++++-----
 .../beam/runners/dataflow/util/Stager.java      |  32 ++++-
 .../runners/dataflow/DataflowRunnerTest.java    |  87 +++++++++++---
 .../extensions/gcp/storage/GcsFileSystem.java   |   5 +-
 6 files changed, 239 insertions(+), 69 deletions(-)
----------------------------------------------------------------------