You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by he...@apache.org on 2018/07/30 17:26:02 UTC

[beam] branch master updated (f67541e -> dc1111b)

This is an automated email from the ASF dual-hosted git repository.

herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


    from f67541e  Merge pull request #6077: [BEAM-4862] Fixes bug in Spanner's MutationGroupEncoder by converting timestamps into Long and not Int.
     add 89fe68a  Add utilities for manipulating proto pipelines
     add 4224bcc  [BEAM-4636] Refactor Go dataflow runner for more flexible uses
     add 585fdce  [BEAM-4813] Switch to translation from model protos to Dataflow v1beta3
     add 1abd375  Add more Go integration tests and fix CoGBK bugs
     add dc9be5e  [BEAM-4636] Ensure unique filename for compiled worker binaries
     add 6228ebd  Handle explicit flatten
     add b6433a3  Fix Dataflow argument bug
     add bdaefa6  CR: tweak code and fix comments
     new dc1111b   [BEAM-4813] Refactor Go Dataflow runner and translation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/go/pkg/beam/core/runtime/exec/plan.go         |  10 +-
 sdks/go/pkg/beam/core/runtime/exec/translate.go    |   3 +
 sdks/go/pkg/beam/core/runtime/graphx/dataflow.go   | 301 +++++++++++++
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  | 266 ------------
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  | 184 ++++----
 sdks/go/pkg/beam/core/runtime/graphx/tree.go       |  31 --
 sdks/go/pkg/beam/core/runtime/pipelinex/clone.go   |  56 +++
 .../runtime/pipelinex/clone_test.go}               |  32 +-
 sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
 .../beam/core/runtime/pipelinex/replace_test.go    | 119 +++++
 sdks/go/pkg/beam/core/runtime/pipelinex/util.go    |  89 ++++
 sdks/go/pkg/beam/core/util/reflectx/util.go        |  92 ++++
 sdks/go/pkg/beam/core/util/reflectx/util_test.go   | 106 +++++
 .../beam/core/util/stringx/{bytes.go => map.go}    |  40 +-
 sdks/go/pkg/beam/runners/dataflow/dataflow.go      | 286 ++----------
 .../beam/runners/dataflow/dataflowlib/execute.go   | 110 +++++
 .../pkg/beam/runners/dataflow/dataflowlib/job.go   | 220 ++++++++++
 .../runners/dataflow/{ => dataflowlib}/messages.go |   2 +-
 .../pkg/beam/runners/dataflow/dataflowlib/stage.go |  56 +++
 .../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
 sdks/go/pkg/beam/runners/dataflow/translate.go     | 479 ---------------------
 .../beam/runners/universal/runnerlib/compile.go    |   7 +-
 .../go/pkg/beam/runners/universal/runnerlib/job.go |   2 +
 sdks/go/pkg/beam/runners/universal/universal.go    |   6 +-
 .../pkg/beam/testing/passert/{hash.go => sum.go}   |  44 +-
 sdks/go/pkg/beam/util/gcsx/gcs.go                  |  12 +
 sdks/go/test/integration/driver.go                 |  11 +-
 sdks/go/test/integration/primitives/cogbk.go       |  84 ++++
 .../integration/primitives/cogbk_test.go}          |  19 +-
 .../integration/primitives/flatten.go}             |  40 +-
 .../primitives/flatten_test.go}                    |  10 +-
 sdks/go/test/integration/primitives/pardo.go       |  82 ++++
 .../primitives}/pardo_test.go                      |  16 +-
 sdks/go/test/regression/pardo.go                   |  80 +++-
 sdks/go/test/regression/pardo_test.go              |  24 ++
 35 files changed, 2240 insertions(+), 1232 deletions(-)
 create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/clone.go
 copy sdks/go/pkg/beam/{coder_test.go => core/runtime/pipelinex/clone_test.go} (62%)
 create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
 create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/util.go
 create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util.go
 create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util_test.go
 copy sdks/go/pkg/beam/core/util/stringx/{bytes.go => map.go} (50%)
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/execute.go
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
 rename sdks/go/pkg/beam/runners/dataflow/{ => dataflowlib}/messages.go (99%)
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/stage.go
 create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
 delete mode 100644 sdks/go/pkg/beam/runners/dataflow/translate.go
 copy sdks/go/pkg/beam/testing/passert/{hash.go => sum.go} (50%)
 create mode 100644 sdks/go/test/integration/primitives/cogbk.go
 copy sdks/go/{pkg/beam/core/runtime/exec/status.go => test/integration/primitives/cogbk_test.go} (80%)
 copy sdks/go/{pkg/beam/transforms/filter/distinct.go => test/integration/primitives/flatten.go} (57%)
 copy sdks/go/test/{regression/pardo_test.go => integration/primitives/flatten_test.go} (80%)
 create mode 100644 sdks/go/test/integration/primitives/pardo.go
 copy sdks/go/test/{regression => integration/primitives}/pardo_test.go (73%)


[beam] 01/01: [BEAM-4813] Refactor Go Dataflow runner and translation

Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

herohde pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit dc1111ba14608b864edafb9e16716601450e2dad
Merge: f67541e bdaefa6
Author: Henning Rohde <he...@seekerror.org>
AuthorDate: Mon Jul 30 10:25:59 2018 -0700

     [BEAM-4813] Refactor Go Dataflow runner and translation

 sdks/go/pkg/beam/core/runtime/exec/plan.go         |  10 +-
 sdks/go/pkg/beam/core/runtime/exec/translate.go    |   3 +
 sdks/go/pkg/beam/core/runtime/graphx/dataflow.go   | 301 +++++++++++++
 sdks/go/pkg/beam/core/runtime/graphx/serialize.go  | 266 ------------
 sdks/go/pkg/beam/core/runtime/graphx/translate.go  | 184 ++++----
 sdks/go/pkg/beam/core/runtime/graphx/tree.go       |  31 --
 sdks/go/pkg/beam/core/runtime/pipelinex/clone.go   |  56 +++
 .../beam/core/runtime/pipelinex/clone_test.go}     |  25 +-
 sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
 .../beam/core/runtime/pipelinex/replace_test.go    | 119 +++++
 sdks/go/pkg/beam/core/runtime/pipelinex/util.go    |  89 ++++
 sdks/go/pkg/beam/core/util/reflectx/util.go        |  92 ++++
 sdks/go/pkg/beam/core/util/reflectx/util_test.go   | 106 +++++
 .../beam/core/util/stringx/map.go}                 |  40 +-
 sdks/go/pkg/beam/runners/dataflow/dataflow.go      | 286 ++----------
 .../beam/runners/dataflow/dataflowlib/execute.go   | 110 +++++
 .../pkg/beam/runners/dataflow/dataflowlib/job.go   | 220 ++++++++++
 .../runners/dataflow/{ => dataflowlib}/messages.go |   2 +-
 .../pkg/beam/runners/dataflow/dataflowlib/stage.go |  56 +++
 .../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
 sdks/go/pkg/beam/runners/dataflow/translate.go     | 479 ---------------------
 .../beam/runners/universal/runnerlib/compile.go    |   7 +-
 .../go/pkg/beam/runners/universal/runnerlib/job.go |   2 +
 sdks/go/pkg/beam/runners/universal/universal.go    |   6 +-
 sdks/go/pkg/beam/testing/passert/sum.go            |  57 +++
 sdks/go/pkg/beam/util/gcsx/gcs.go                  |  12 +
 sdks/go/test/integration/driver.go                 |  11 +-
 sdks/go/test/integration/primitives/cogbk.go       |  84 ++++
 .../primitives/cogbk_test.go}                      |  12 +-
 .../primitives/flatten.go}                         |  38 +-
 .../primitives/flatten_test.go}                    |  10 +-
 sdks/go/test/integration/primitives/pardo.go       |  82 ++++
 .../primitives}/pardo_test.go                      |  16 +-
 sdks/go/test/regression/pardo.go                   |  80 +++-
 sdks/go/test/regression/pardo_test.go              |  24 ++
 35 files changed, 2276 insertions(+), 1193 deletions(-)