You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by he...@apache.org on 2018/07/30 17:26:03 UTC
[beam] 01/01: [BEAM-4813] Refactor Go Dataflow runner and
translation
This is an automated email from the ASF dual-hosted git repository.
herohde pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit dc1111ba14608b864edafb9e16716601450e2dad
Merge: f67541e bdaefa6
Author: Henning Rohde <he...@seekerror.org>
AuthorDate: Mon Jul 30 10:25:59 2018 -0700
[BEAM-4813] Refactor Go Dataflow runner and translation
sdks/go/pkg/beam/core/runtime/exec/plan.go | 10 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 301 +++++++++++++
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 266 ------------
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 184 ++++----
sdks/go/pkg/beam/core/runtime/graphx/tree.go | 31 --
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 56 +++
.../beam/core/runtime/pipelinex/clone_test.go} | 25 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
.../beam/core/runtime/pipelinex/replace_test.go | 119 +++++
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 89 ++++
sdks/go/pkg/beam/core/util/reflectx/util.go | 92 ++++
sdks/go/pkg/beam/core/util/reflectx/util_test.go | 106 +++++
.../beam/core/util/stringx/map.go} | 40 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 286 ++----------
.../beam/runners/dataflow/dataflowlib/execute.go | 110 +++++
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 220 ++++++++++
.../runners/dataflow/{ => dataflowlib}/messages.go | 2 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 56 +++
.../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
sdks/go/pkg/beam/runners/dataflow/translate.go | 479 ---------------------
.../beam/runners/universal/runnerlib/compile.go | 7 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 2 +
sdks/go/pkg/beam/runners/universal/universal.go | 6 +-
sdks/go/pkg/beam/testing/passert/sum.go | 57 +++
sdks/go/pkg/beam/util/gcsx/gcs.go | 12 +
sdks/go/test/integration/driver.go | 11 +-
sdks/go/test/integration/primitives/cogbk.go | 84 ++++
.../primitives/cogbk_test.go} | 12 +-
.../primitives/flatten.go} | 38 +-
.../primitives/flatten_test.go} | 10 +-
sdks/go/test/integration/primitives/pardo.go | 82 ++++
.../primitives}/pardo_test.go | 16 +-
sdks/go/test/regression/pardo.go | 80 +++-
sdks/go/test/regression/pardo_test.go | 24 ++
35 files changed, 2276 insertions(+), 1193 deletions(-)