You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by he...@apache.org on 2018/07/30 17:26:02 UTC
[beam] branch master updated (f67541e -> dc1111b)
This is an automated email from the ASF dual-hosted git repository.
herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from f67541e Merge pull request #6077: [BEAM-4862] Fixes bug in Spanner's MutationGroupEncoder by converting timestamps into Long and not Int.
add 89fe68a Add utilities for manipulating proto pipelines
add 4224bcc [BEAM-4636] Refactor Go dataflow runner for more flexible uses
add 585fdce [BEAM-4813] Switch to translation from model protos to Dataflow v1beta3
add 1abd375 Add more Go integration tests and fix CoGBK bugs
add dc9be5e [BEAM-4636] Ensure unique filename for compiled worker binaries
add 6228ebd Handle explicit flatten
add b6433a3 Fix Dataflow argument bug
add bdaefa6 CR: tweak code and fix comments
new dc1111b [BEAM-4813] Refactor Go Dataflow runner and translation
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
sdks/go/pkg/beam/core/runtime/exec/plan.go | 10 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 301 +++++++++++++
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 266 ------------
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 184 ++++----
sdks/go/pkg/beam/core/runtime/graphx/tree.go | 31 --
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 56 +++
.../runtime/pipelinex/clone_test.go} | 32 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
.../beam/core/runtime/pipelinex/replace_test.go | 119 +++++
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 89 ++++
sdks/go/pkg/beam/core/util/reflectx/util.go | 92 ++++
sdks/go/pkg/beam/core/util/reflectx/util_test.go | 106 +++++
.../beam/core/util/stringx/{bytes.go => map.go} | 40 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 286 ++----------
.../beam/runners/dataflow/dataflowlib/execute.go | 110 +++++
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 220 ++++++++++
.../runners/dataflow/{ => dataflowlib}/messages.go | 2 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 56 +++
.../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
sdks/go/pkg/beam/runners/dataflow/translate.go | 479 ---------------------
.../beam/runners/universal/runnerlib/compile.go | 7 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 2 +
sdks/go/pkg/beam/runners/universal/universal.go | 6 +-
.../pkg/beam/testing/passert/{hash.go => sum.go} | 44 +-
sdks/go/pkg/beam/util/gcsx/gcs.go | 12 +
sdks/go/test/integration/driver.go | 11 +-
sdks/go/test/integration/primitives/cogbk.go | 84 ++++
.../integration/primitives/cogbk_test.go} | 19 +-
.../integration/primitives/flatten.go} | 40 +-
.../primitives/flatten_test.go} | 10 +-
sdks/go/test/integration/primitives/pardo.go | 82 ++++
.../primitives}/pardo_test.go | 16 +-
sdks/go/test/regression/pardo.go | 80 +++-
sdks/go/test/regression/pardo_test.go | 24 ++
35 files changed, 2240 insertions(+), 1232 deletions(-)
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/clone.go
copy sdks/go/pkg/beam/{coder_test.go => core/runtime/pipelinex/clone_test.go} (62%)
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/util.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util_test.go
copy sdks/go/pkg/beam/core/util/stringx/{bytes.go => map.go} (50%)
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/execute.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
rename sdks/go/pkg/beam/runners/dataflow/{ => dataflowlib}/messages.go (99%)
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/stage.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
delete mode 100644 sdks/go/pkg/beam/runners/dataflow/translate.go
copy sdks/go/pkg/beam/testing/passert/{hash.go => sum.go} (50%)
create mode 100644 sdks/go/test/integration/primitives/cogbk.go
copy sdks/go/{pkg/beam/core/runtime/exec/status.go => test/integration/primitives/cogbk_test.go} (80%)
copy sdks/go/{pkg/beam/transforms/filter/distinct.go => test/integration/primitives/flatten.go} (57%)
copy sdks/go/test/{regression/pardo_test.go => integration/primitives/flatten_test.go} (80%)
create mode 100644 sdks/go/test/integration/primitives/pardo.go
copy sdks/go/test/{regression => integration/primitives}/pardo_test.go (73%)
[beam] 01/01: [BEAM-4813] Refactor Go Dataflow runner and
translation
Posted by he...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
herohde pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit dc1111ba14608b864edafb9e16716601450e2dad
Merge: f67541e bdaefa6
Author: Henning Rohde <he...@seekerror.org>
AuthorDate: Mon Jul 30 10:25:59 2018 -0700
[BEAM-4813] Refactor Go Dataflow runner and translation
sdks/go/pkg/beam/core/runtime/exec/plan.go | 10 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 301 +++++++++++++
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 266 ------------
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 184 ++++----
sdks/go/pkg/beam/core/runtime/graphx/tree.go | 31 --
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 56 +++
.../beam/core/runtime/pipelinex/clone_test.go} | 25 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
.../beam/core/runtime/pipelinex/replace_test.go | 119 +++++
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 89 ++++
sdks/go/pkg/beam/core/util/reflectx/util.go | 92 ++++
sdks/go/pkg/beam/core/util/reflectx/util_test.go | 106 +++++
.../beam/core/util/stringx/map.go} | 40 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 286 ++----------
.../beam/runners/dataflow/dataflowlib/execute.go | 110 +++++
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 220 ++++++++++
.../runners/dataflow/{ => dataflowlib}/messages.go | 2 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 56 +++
.../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
sdks/go/pkg/beam/runners/dataflow/translate.go | 479 ---------------------
.../beam/runners/universal/runnerlib/compile.go | 7 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 2 +
sdks/go/pkg/beam/runners/universal/universal.go | 6 +-
sdks/go/pkg/beam/testing/passert/sum.go | 57 +++
sdks/go/pkg/beam/util/gcsx/gcs.go | 12 +
sdks/go/test/integration/driver.go | 11 +-
sdks/go/test/integration/primitives/cogbk.go | 84 ++++
.../primitives/cogbk_test.go} | 12 +-
.../primitives/flatten.go} | 38 +-
.../primitives/flatten_test.go} | 10 +-
sdks/go/test/integration/primitives/pardo.go | 82 ++++
.../primitives}/pardo_test.go | 16 +-
sdks/go/test/regression/pardo.go | 80 +++-
sdks/go/test/regression/pardo_test.go | 24 ++
35 files changed, 2276 insertions(+), 1193 deletions(-)