You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ke...@apache.org on 2017/12/04 20:36:38 UTC
[beam] branch go-sdk updated (ead5d43 -> 473d7a0)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git.
from ead5d43 This closes #4177: [BEAM-2870] Strips partition decorators when creating/patching tables in batch
add e2aa9b3 Initial sketches of a Go SDK
add 7197d5e Initial version of the direct style w/ direct runner. Incomplete.
add e92e919 Add Data as UserFn context w/ immediate value.
add 995b2ec Added no-I/O wordcount for profiling.
add 900da2e Fleshed out possible approach to generic transformations.
add 7bc0498 Add “dag” example that use multiplexing and side input.
add 915416a Added a more complex DAG example.
add fa8723a Add yatzy example with more complex construction-time setup
add 60d17c1 Add proto for Fn API
add b1aaac5 Add beam.Composite helper for the most common pattern to align with java
add d08fdaa Move pipeline-construction time errors into an accumulator
add ee71e83 Add Dataflow job and Fn API clients. Incomplete.
add 61db2fb Add binary cross-compile and upload to Dataflow runner. Incomplete.
add 5ce4d1c Add tentative runner indirection (default: local).
add 01111a9 Made data flow runner detect user main for cross-compilation.
add 08ba947 Remove error accumulation in favor of panic.
add f66bd66 Improve Dataflow translation of coders, side input and composite names.
add c579e85 Fix name for AsView.
add 26a99d5 Use 2 grpc endpoints in harness
add d5cf63d Add gRPC harness logging
add 0b2fdbc Flesh out harness and serialization further.
add 6f4af7f Made the dataflow runner wait for job termination by default
add 3c8f986 beam:
add 687cbf8 beam:
add c47a546 combinefn.go: fix compilation issues
add b9d03b2 Improve dataflow serialization and execution. Incomplete.
add af65f9d Sleep 30 sec in wordcap to allow logs to propagate to Cloud Logging.
add 38ddbbb Move the 30s sleep for logging to the harness instead of in WordCap.
add e185f48 Post-review updates.
add 8dcd0c4 Doc updates.
add d64868c Flesh out coders. Incomplete.
add ca49202 Added prototype implementation of more coders and the runner source. Incomplete.
add 0444a45 dofn: illustrates how dofns are written.
add 9e2a7cc beam: add viewfn and windowfn to side inputs match support Beam 1.0
add b767a95 dofn: timers
add be7e061 Complete revamp: coders, graph and execution use element-wise processing.
add 055c859 Fix coder encoding for Dataflow side input. Otherwise, the job is rejected.
add 22536e3 Added more godoc comments to graph types.
add 747930f Added more comments plus made local GBK use coder equality.
add ce8c4df Added Flatten support and “forest” example that uses it.
add 61d8932 Move bigqueryio to defunct
add ca59228 Make forest example print less
add dbafc8b Add external struct registry and serialization.
add 174a2b0 Updated comments in node.go.
add 72e3bbe Replace real type with 'full type' since that's the current term.
add 58dda52 Refactor Fn API dependency.
add bf6ff10 Added more comments to the runner/dataflow and runner/beamexec packages
add bcc5001 Fix most go vet issues
add 24e2078 Make core operations panic to cut down on the error propagation boilerplate
add 2662fc8 Add more comments to the graph package.
add 4eab543 Add DoFn wrapper to handle either function or (ptr to) struct
add eca6fec Fix remaining go vet warnings.
add f1953d3 Code review for beam/graph/coder package.
add 5951d28 Code review of the runtime/graphx package.
add 978b60e Remove Data options in favor of using a Fn struct
add d56db10 Code review of the beam/graph/userfn package.
add 9a74200 Code review for beam/graph package.
add 15489ee godoc for runtime/graphx
add af323c8 Add support for []T and Combine functions
add b1d8df2 Add adapted documentation from the Java SDK to the beam package
add 4f5126c Update snapshot of Fn API.
add bf3cac1 Add experiments flag to the Dataflow runner
add 3c848b6 Remove context arg from beamexec.Init
add ca38035 Migration to Runner API.
add 7b6ca59 Add support for creating DOT graphs.
add 0dc6e2d Make pretty printing of types and coders more concise
add a854fbc Add flexible Signature to aid type checking
add c87e176 Adding unit testability to harness translation.
add b3f65ea Fix crash due to initialization order
add ea6c7c4 Add CreateValues and Impulse
add 1637568 Add Runner API support for WindowingStrategy.
add 65d0195 Run goimports on baseline.
add dc23901 Fix encoding of global window strategy.
add adc7869 Ensure the windowed value is atomically encoded.
add ce880b7 Limit gRPC messages to max size.
add 841520e Developer conveniences for running jobs.
add 35ff36e Fix sends to not close the network channel.
add 461c315 Add re-iterable side input
add ea78889 Add per-key Combine
add 121acd8 Add Min
add cbd622e Reorganize non-user-facing code into core
add c0962fb Make type register reject unnamed or predeclared types
add 6b51242 Add type specialization tool
add 0d439e9 Don't run grpc plugin in generate phase.
add 40d8111 Fix import reference path for runner API proto.
add 34d9647 Revamp runner registration as _ imports
add a9136b3 Add stats.Max and Mean
add 214c076 Add global pipeline options
add 11f9c03 Unify global and per-key combiners
add c0658c5 Add beam convenience wrapper for imports and runner selection
add dca064e Add session recording and CPU profiling to harness.
add d17c601 Add ptest and passert for testing pipelines
add 9e575d6 Add GCS and glob support to textio
add 4fa8db0 Add BigQuery IO and examples
add f01e74a Adds a session runner for testing.
add 1627288 Add Partition and dynamic functions
add 9bf757f Adding example that returns 10 words that contain provided search string.
add e95172b Remove duplicate LOG line
add 5322ed6 Enable Combine Fns in Dataflow runner by modifying translation.
add 5e26b7a Fixing type bug by dropping T and using underlying type of value in addInput(). Changing Count() to use Sum().
add dbbe214 Adding graph validation at build time.
add fc5ab4f Import the Fn API changes.
add f945954 Simple changes to support new Fn API coder changes.
add bf6bc20 Update translator to work with new Fn API changes.
add 5c9cba6 Use appropriate equality tests.
add b897ea4 Fix test to not use path of package.
add 330d037 Renaming directory to match package name.
add 895b735 Fixing random nits in comments.
add 308fee0 Modify build command to avoid bash.
add c762062 Fixing selected golint issues.
add f109f5e Addressing import review comments.
add 0c89de4 Add coder specialization for bytes/strings.
add 7ee4c51 Adding unit tests to stats.
add 00fefa5 Fixing typo.
add b7b8c31 Add beam.External
add da1c077 Fix grpc.Dial calls to block properly.
add 3246480 Creates a symtab verifier by running Sym2Addr and Addr2Sym in a binary.
add c75f747 Add spec field to help interpretation of payload.
add 7ac34c7 Use beam.T alias for typex.T etc with Go 1.9
add a1ed07c Move shared GCP options to a separate package
add fb2c2a3 Update portability protos
add 3fdf71e Remove old source/sink from beam package
add e6e05f5 Add context-aware logging for both pipeline-submission time and runtime
add 52fdf4b Fix coder inference for strings.
add 5f38b6d Improve tornadoes example
add 6187549 Fix beam.External to map arguments correctly.
add e99cb8e Added comments to yatzy and forest
add e0a5b71 Add comments to tornadoes from the java counterpart
add 738415f Rename Pipeline Composite to Scope
add b8616e8 Add 3 progressive wordcount examples
add 10f9c28 Clarify comments in wordcount pipelines
add 55ea735 Add apache 2.0 license to files
add c665314 Updates to examples.
add afc6097 Adding more godoc for the main beam package.
add 00a3bbc Update to new proto structure
add bc58989 Split Combine and fields in to global and per-key variants
add 1899718 Refactor Flatten of a single []T into Explode
add bd962f6 Rename local runner to direct runner
add 663046e Fix argument index error in ParDo execution
add 670b13c Add Apache copyright header to files that need it.
add 2d5a991 Made debug.Head not just work per bundle
add 47b2e31 Impose a total ordering on Fn parameters.
add 0b0aeac Rename Dedup to Distinct for consistency with other SDKs
add baa38c2 Add coder to model coder translation
add a088674 Simplify harness coder translation
add 663fad3 Split Pipeline into Pipeline and Scope
add 5743a37 Relocate Go SDK code
add 999985c Fix Go SDK maven build
add 38df1b7 Move Go SKD to latest version of bigquery
new 473d7a0 Merge pull request #4200 from herohde/go-sdk2
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
pom.xml | 5 +-
sdks/go/cmd/specialize/main.go | 175 +
sdks/go/cmd/symtab/main.go | 101 +
sdks/go/data/haiku/old_pond.txt | 3 +
sdks/go/data/shakespeare/hamlet.txt | 6045 ++++++++++++++++++++
sdks/go/data/shakespeare/kinglear-hashtag.txt | 5525 ++++++++++++++++++
sdks/go/data/shakespeare/kinglear.txt | 5525 ++++++++++++++++++
sdks/go/data/shakespeare/romeoandjuliet.txt | 4766 +++++++++++++++
.../examples/complete/autocomplete/autocomplete.go | 71 +
sdks/go/examples/contains/contains.go | 99 +
sdks/go/examples/cookbook/combine/combine.go | 106 +
sdks/go/examples/cookbook/filter/filter.go | 106 +
sdks/go/examples/cookbook/max/max.go | 88 +
sdks/go/examples/cookbook/tornadoes/tornadoes.go | 119 +
.../debugging_wordcount/debugging_wordcount.go | 163 +
sdks/go/examples/forest/forest.go | 81 +
sdks/go/examples/grades/grades.go | 105 +
.../minimal_wordcount/minimal_wordcount.go | 101 +
sdks/go/examples/pingpong/pingpong.go | 131 +
sdks/go/examples/wordcap/wordcap.go | 75 +
sdks/go/examples/wordcount/wordcount.go | 168 +
sdks/go/examples/yatzy/yatzy.go | 139 +
sdks/go/pkg/beam/coder.go | 165 +
sdks/go/pkg/beam/coder_test.go | 42 +
sdks/go/pkg/beam/combine.go | 81 +
sdks/go/pkg/beam/core/funcx/doc.go | 22 +
sdks/go/pkg/beam/core/funcx/fn.go | 436 ++
sdks/go/pkg/beam/core/funcx/fn_test.go | 223 +
sdks/go/pkg/beam/core/funcx/output.go | 69 +
sdks/go/pkg/beam/core/funcx/output_test.go | 48 +
sdks/go/pkg/beam/core/funcx/sideinput.go | 100 +
sdks/go/pkg/beam/core/funcx/sideinput_test.go | 70 +
sdks/go/pkg/beam/core/funcx/signature.go | 212 +
sdks/go/pkg/beam/core/funcx/signature_test.go | 212 +
sdks/go/pkg/beam/core/graph/bind.go | 324 ++
sdks/go/pkg/beam/core/graph/bind_test.go | 193 +
sdks/go/pkg/beam/core/graph/coder/coder.go | 279 +
sdks/go/pkg/beam/core/graph/coder/int.go | 71 +
sdks/go/pkg/beam/core/graph/coder/int_test.go | 88 +
sdks/go/pkg/beam/core/graph/coder/time.go | 44 +
sdks/go/pkg/beam/core/graph/coder/varint.go | 98 +
sdks/go/pkg/beam/core/graph/coder/varint_test.go | 94 +
sdks/go/pkg/beam/core/graph/doc.go | 22 +
sdks/go/pkg/beam/core/graph/edge.go | 382 ++
sdks/go/pkg/beam/core/graph/fn.go | 306 +
sdks/go/pkg/beam/core/graph/graph.go | 126 +
sdks/go/pkg/beam/core/graph/graph_test.go | 68 +
sdks/go/pkg/beam/core/graph/node.go | 70 +
sdks/go/pkg/beam/core/graph/scope.go | 39 +
sdks/go/pkg/beam/core/graph/window/window.go | 70 +
sdks/go/pkg/beam/core/runtime/exec/coder.go | 213 +
sdks/go/pkg/beam/core/runtime/exec/exec.go | 50 +
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 135 +
sdks/go/pkg/beam/core/runtime/exec/nodes.go | 881 +++
sdks/go/pkg/beam/core/runtime/exec/unit.go | 93 +
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 353 ++
sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 111 +
sdks/go/pkg/beam/core/runtime/graphx/doc.go | 29 +
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 905 +++
sdks/go/pkg/beam/core/runtime/graphx/user.go | 109 +
sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go | 18 +
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 763 +++
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto | 217 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 309 +
sdks/go/pkg/beam/core/runtime/harness/gen.go | 18 +
sdks/go/pkg/beam/core/runtime/harness/harness.go | 240 +
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 113 +
sdks/go/pkg/beam/core/runtime/harness/logging.go | 151 +
sdks/go/pkg/beam/core/runtime/harness/session.go | 212 +
.../go/pkg/beam/core/runtime/harness/session.proto | 69 +
.../core/runtime/harness/session/session.pb.go | 444 ++
sdks/go/pkg/beam/core/runtime/harness/translate.go | 343 ++
.../beam/core/runtime/harness/translate_test.go | 298 +
sdks/go/pkg/beam/core/runtime/init.go | 41 +
sdks/go/pkg/beam/core/runtime/options.go | 91 +
sdks/go/pkg/beam/core/runtime/options_test.go | 49 +
sdks/go/pkg/beam/core/runtime/symbols.go | 59 +
sdks/go/pkg/beam/core/runtime/types.go | 62 +
sdks/go/pkg/beam/core/runtime/types_test.go | 74 +
sdks/go/pkg/beam/core/typex/class.go | 174 +
sdks/go/pkg/beam/core/typex/class_test.go | 105 +
sdks/go/pkg/beam/core/typex/fulltype.go | 393 ++
sdks/go/pkg/beam/core/typex/fulltype_test.go | 125 +
sdks/go/pkg/beam/core/typex/special.go | 80 +
sdks/go/pkg/beam/core/util/dot/dot.go | 130 +
sdks/go/pkg/beam/core/util/ioutilx/read.go | 42 +
sdks/go/pkg/beam/core/util/protox/any.go | 101 +
sdks/go/pkg/beam/core/util/protox/any_test.go | 83 +
sdks/go/pkg/beam/core/util/protox/base64.go | 41 +
sdks/go/pkg/beam/core/util/protox/query.go | 45 +
sdks/go/pkg/beam/core/util/reflectx/functions.go | 45 +
sdks/go/pkg/beam/core/util/reflectx/json.go | 32 +
sdks/go/pkg/beam/core/util/reflectx/tags.go | 74 +
sdks/go/pkg/beam/core/util/reflectx/types.go | 111 +
sdks/go/pkg/beam/core/util/symtab/symtab.go | 121 +
sdks/go/pkg/beam/create.go | 95 +
sdks/go/pkg/beam/doc.go | 67 +
sdks/go/pkg/beam/encoding.go | 147 +
sdks/go/pkg/beam/external.go | 111 +
sdks/go/pkg/beam/flatten.go | 69 +
sdks/go/pkg/beam/forward.go | 84 +
sdks/go/pkg/beam/gbk.go | 88 +
sdks/go/pkg/beam/impulse.go | 41 +
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 234 +
sdks/go/pkg/beam/io/bigqueryio/bigquery_test.go | 38 +
sdks/go/pkg/beam/io/textio/filesystem.go | 49 +
sdks/go/pkg/beam/io/textio/gcs/gcs.go | 130 +
sdks/go/pkg/beam/io/textio/local/local.go | 55 +
sdks/go/pkg/beam/io/textio/textio.go | 204 +
sdks/go/pkg/beam/log/log.go | 183 +
sdks/go/pkg/beam/log/standard.go | 37 +
sdks/go/pkg/beam/option.go | 72 +
sdks/go/pkg/beam/options/gcpopts/options.go | 38 +
sdks/go/pkg/beam/pardo.go | 314 +
sdks/go/pkg/beam/partition.go | 96 +
sdks/go/pkg/beam/partition_test.go | 164 +
sdks/go/pkg/beam/pcollection.go | 90 +
sdks/go/pkg/beam/pipeline.go | 75 +
sdks/go/pkg/beam/runner.go | 51 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 325 ++
sdks/go/pkg/beam/runners/dataflow/messages.go | 174 +
sdks/go/pkg/beam/runners/dataflow/translate.go | 326 ++
sdks/go/pkg/beam/runners/direct/direct.go | 270 +
sdks/go/pkg/beam/runners/direct/nodes.go | 241 +
sdks/go/pkg/beam/runners/dot/dot.go | 55 +
sdks/go/pkg/beam/runners/session/session.go | 337 ++
sdks/go/pkg/beam/testing/passert/passert.go | 218 +
sdks/go/pkg/beam/testing/ptest/ptest.go | 59 +
sdks/go/pkg/beam/testing/ptest/symbols.go | 67 +
sdks/go/pkg/beam/transforms/filter/distinct.go | 39 +
.../go/pkg/beam/transforms/filter/distinct_test.go | 70 +
sdks/go/pkg/beam/transforms/filter/filter.go | 88 +
sdks/go/pkg/beam/transforms/filter/filter_test.go | 90 +
sdks/go/pkg/beam/transforms/stats/count.go | 34 +
sdks/go/pkg/beam/transforms/stats/count_test.go | 69 +
sdks/go/pkg/beam/transforms/stats/max.go | 44 +
sdks/go/pkg/beam/transforms/stats/max_switch.go | 138 +
sdks/go/pkg/beam/transforms/stats/max_switch.tmpl | 40 +
sdks/go/pkg/beam/transforms/stats/max_test.go | 134 +
sdks/go/pkg/beam/transforms/stats/mean.go | 91 +
sdks/go/pkg/beam/transforms/stats/mean_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/min.go | 44 +
sdks/go/pkg/beam/transforms/stats/min_switch.go | 138 +
sdks/go/pkg/beam/transforms/stats/min_switch.tmpl | 40 +
sdks/go/pkg/beam/transforms/stats/min_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/sum.go | 44 +
sdks/go/pkg/beam/transforms/stats/sum_switch.go | 102 +
sdks/go/pkg/beam/transforms/stats/sum_switch.tmpl | 37 +
sdks/go/pkg/beam/transforms/stats/sum_test.go | 117 +
sdks/go/pkg/beam/transforms/stats/util.go | 48 +
sdks/go/pkg/beam/transforms/top/top.go | 173 +
sdks/go/pkg/beam/transforms/top/top_test.go | 125 +
sdks/go/pkg/beam/util.go | 110 +
sdks/go/pkg/beam/util/gcsx/gcs.go | 40 +
sdks/go/pkg/beam/validate.go | 76 +
sdks/go/pkg/beam/x/beamx/run.go | 39 +
sdks/go/pkg/beam/x/debug/head.go | 68 +
sdks/go/pkg/beam/x/debug/print.go | 94 +
sdks/go/pom.xml | 3 +
159 files changed, 42783 insertions(+), 1 deletion(-)
create mode 100644 sdks/go/cmd/specialize/main.go
create mode 100644 sdks/go/cmd/symtab/main.go
create mode 100644 sdks/go/data/haiku/old_pond.txt
create mode 100644 sdks/go/data/shakespeare/hamlet.txt
create mode 100644 sdks/go/data/shakespeare/kinglear-hashtag.txt
create mode 100644 sdks/go/data/shakespeare/kinglear.txt
create mode 100644 sdks/go/data/shakespeare/romeoandjuliet.txt
create mode 100644 sdks/go/examples/complete/autocomplete/autocomplete.go
create mode 100644 sdks/go/examples/contains/contains.go
create mode 100644 sdks/go/examples/cookbook/combine/combine.go
create mode 100644 sdks/go/examples/cookbook/filter/filter.go
create mode 100644 sdks/go/examples/cookbook/max/max.go
create mode 100644 sdks/go/examples/cookbook/tornadoes/tornadoes.go
create mode 100644 sdks/go/examples/debugging_wordcount/debugging_wordcount.go
create mode 100644 sdks/go/examples/forest/forest.go
create mode 100644 sdks/go/examples/grades/grades.go
create mode 100644 sdks/go/examples/minimal_wordcount/minimal_wordcount.go
create mode 100644 sdks/go/examples/pingpong/pingpong.go
create mode 100644 sdks/go/examples/wordcap/wordcap.go
create mode 100644 sdks/go/examples/wordcount/wordcount.go
create mode 100644 sdks/go/examples/yatzy/yatzy.go
create mode 100644 sdks/go/pkg/beam/coder.go
create mode 100644 sdks/go/pkg/beam/coder_test.go
create mode 100644 sdks/go/pkg/beam/combine.go
create mode 100644 sdks/go/pkg/beam/core/funcx/doc.go
create mode 100644 sdks/go/pkg/beam/core/funcx/fn.go
create mode 100644 sdks/go/pkg/beam/core/funcx/fn_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/output.go
create mode 100644 sdks/go/pkg/beam/core/funcx/output_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/sideinput.go
create mode 100644 sdks/go/pkg/beam/core/funcx/sideinput_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/signature.go
create mode 100644 sdks/go/pkg/beam/core/funcx/signature_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/bind.go
create mode 100644 sdks/go/pkg/beam/core/graph/bind_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/coder.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/int.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/int_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/time.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/varint.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/varint_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/doc.go
create mode 100644 sdks/go/pkg/beam/core/graph/edge.go
create mode 100644 sdks/go/pkg/beam/core/graph/fn.go
create mode 100644 sdks/go/pkg/beam/core/graph/graph.go
create mode 100644 sdks/go/pkg/beam/core/graph/graph_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/node.go
create mode 100644 sdks/go/pkg/beam/core/graph/scope.go
create mode 100644 sdks/go/pkg/beam/core/graph/window/window.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/coder.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/exec.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fullvalue.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/nodes.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/unit.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/coder.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/doc.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/serialize.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/user.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/datamgr.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/gen.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/harness.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/init/init.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/logging.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session.proto
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/translate.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/translate_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/init.go
create mode 100644 sdks/go/pkg/beam/core/runtime/options.go
create mode 100644 sdks/go/pkg/beam/core/runtime/options_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/symbols.go
create mode 100644 sdks/go/pkg/beam/core/runtime/types.go
create mode 100644 sdks/go/pkg/beam/core/runtime/types_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/class.go
create mode 100644 sdks/go/pkg/beam/core/typex/class_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/fulltype.go
create mode 100644 sdks/go/pkg/beam/core/typex/fulltype_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/special.go
create mode 100644 sdks/go/pkg/beam/core/util/dot/dot.go
create mode 100644 sdks/go/pkg/beam/core/util/ioutilx/read.go
create mode 100644 sdks/go/pkg/beam/core/util/protox/any.go
create mode 100644 sdks/go/pkg/beam/core/util/protox/any_test.go
create mode 100644 sdks/go/pkg/beam/core/util/protox/base64.go
create mode 100644 sdks/go/pkg/beam/core/util/protox/query.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/functions.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/json.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/tags.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/types.go
create mode 100644 sdks/go/pkg/beam/core/util/symtab/symtab.go
create mode 100644 sdks/go/pkg/beam/create.go
create mode 100644 sdks/go/pkg/beam/doc.go
create mode 100644 sdks/go/pkg/beam/encoding.go
create mode 100644 sdks/go/pkg/beam/external.go
create mode 100644 sdks/go/pkg/beam/flatten.go
create mode 100644 sdks/go/pkg/beam/forward.go
create mode 100644 sdks/go/pkg/beam/gbk.go
create mode 100644 sdks/go/pkg/beam/impulse.go
create mode 100644 sdks/go/pkg/beam/io/bigqueryio/bigquery.go
create mode 100644 sdks/go/pkg/beam/io/bigqueryio/bigquery_test.go
create mode 100644 sdks/go/pkg/beam/io/textio/filesystem.go
create mode 100644 sdks/go/pkg/beam/io/textio/gcs/gcs.go
create mode 100644 sdks/go/pkg/beam/io/textio/local/local.go
create mode 100644 sdks/go/pkg/beam/io/textio/textio.go
create mode 100644 sdks/go/pkg/beam/log/log.go
create mode 100644 sdks/go/pkg/beam/log/standard.go
create mode 100644 sdks/go/pkg/beam/option.go
create mode 100644 sdks/go/pkg/beam/options/gcpopts/options.go
create mode 100644 sdks/go/pkg/beam/pardo.go
create mode 100644 sdks/go/pkg/beam/partition.go
create mode 100644 sdks/go/pkg/beam/partition_test.go
create mode 100644 sdks/go/pkg/beam/pcollection.go
create mode 100644 sdks/go/pkg/beam/pipeline.go
create mode 100644 sdks/go/pkg/beam/runner.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflow.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/messages.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/translate.go
create mode 100644 sdks/go/pkg/beam/runners/direct/direct.go
create mode 100644 sdks/go/pkg/beam/runners/direct/nodes.go
create mode 100644 sdks/go/pkg/beam/runners/dot/dot.go
create mode 100644 sdks/go/pkg/beam/runners/session/session.go
create mode 100644 sdks/go/pkg/beam/testing/passert/passert.go
create mode 100644 sdks/go/pkg/beam/testing/ptest/ptest.go
create mode 100644 sdks/go/pkg/beam/testing/ptest/symbols.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/distinct.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/distinct_test.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/filter.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/filter_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/count.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/count_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/max.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/max_switch.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/max_switch.tmpl
create mode 100644 sdks/go/pkg/beam/transforms/stats/max_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/mean.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/mean_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/min.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/min_switch.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/min_switch.tmpl
create mode 100644 sdks/go/pkg/beam/transforms/stats/min_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum_switch.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum_switch.tmpl
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/util.go
create mode 100644 sdks/go/pkg/beam/transforms/top/top.go
create mode 100644 sdks/go/pkg/beam/transforms/top/top_test.go
create mode 100644 sdks/go/pkg/beam/util.go
create mode 100644 sdks/go/pkg/beam/validate.go
create mode 100644 sdks/go/pkg/beam/x/beamx/run.go
create mode 100644 sdks/go/pkg/beam/x/debug/head.go
create mode 100644 sdks/go/pkg/beam/x/debug/print.go
--
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" <co...@beam.apache.org>'].
[beam] 01/01: Merge pull request #4200 from herohde/go-sdk2
Posted by ke...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch go-sdk
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 473d7a00964d17e876b94d3368280a16b6b057cd
Merge: ead5d43 38df1b7
Author: Kenn Knowles <ke...@kennknowles.com>
AuthorDate: Mon Dec 4 12:36:32 2017 -0800
Merge pull request #4200 from herohde/go-sdk2
[BEAM-2083] Go SDK prototype
pom.xml | 5 +-
sdks/go/cmd/specialize/main.go | 175 +
sdks/go/cmd/symtab/main.go | 101 +
sdks/go/data/haiku/old_pond.txt | 3 +
sdks/go/data/shakespeare/hamlet.txt | 6045 ++++++++++++++++++++
sdks/go/data/shakespeare/kinglear-hashtag.txt | 5525 ++++++++++++++++++
sdks/go/data/shakespeare/kinglear.txt | 5525 ++++++++++++++++++
sdks/go/data/shakespeare/romeoandjuliet.txt | 4766 +++++++++++++++
.../examples/complete/autocomplete/autocomplete.go | 71 +
sdks/go/examples/contains/contains.go | 99 +
sdks/go/examples/cookbook/combine/combine.go | 106 +
sdks/go/examples/cookbook/filter/filter.go | 106 +
sdks/go/examples/cookbook/max/max.go | 88 +
sdks/go/examples/cookbook/tornadoes/tornadoes.go | 119 +
.../debugging_wordcount/debugging_wordcount.go | 163 +
sdks/go/examples/forest/forest.go | 81 +
sdks/go/examples/grades/grades.go | 105 +
.../minimal_wordcount/minimal_wordcount.go | 101 +
sdks/go/examples/pingpong/pingpong.go | 131 +
sdks/go/examples/wordcap/wordcap.go | 75 +
sdks/go/examples/wordcount/wordcount.go | 168 +
sdks/go/examples/yatzy/yatzy.go | 139 +
sdks/go/pkg/beam/coder.go | 165 +
sdks/go/pkg/beam/coder_test.go | 42 +
sdks/go/pkg/beam/combine.go | 81 +
sdks/go/pkg/beam/core/funcx/doc.go | 22 +
sdks/go/pkg/beam/core/funcx/fn.go | 436 ++
sdks/go/pkg/beam/core/funcx/fn_test.go | 223 +
sdks/go/pkg/beam/core/funcx/output.go | 69 +
sdks/go/pkg/beam/core/funcx/output_test.go | 48 +
sdks/go/pkg/beam/core/funcx/sideinput.go | 100 +
sdks/go/pkg/beam/core/funcx/sideinput_test.go | 70 +
sdks/go/pkg/beam/core/funcx/signature.go | 212 +
sdks/go/pkg/beam/core/funcx/signature_test.go | 212 +
sdks/go/pkg/beam/core/graph/bind.go | 324 ++
sdks/go/pkg/beam/core/graph/bind_test.go | 193 +
sdks/go/pkg/beam/core/graph/coder/coder.go | 279 +
sdks/go/pkg/beam/core/graph/coder/int.go | 71 +
sdks/go/pkg/beam/core/graph/coder/int_test.go | 88 +
sdks/go/pkg/beam/core/graph/coder/time.go | 44 +
sdks/go/pkg/beam/core/graph/coder/varint.go | 98 +
sdks/go/pkg/beam/core/graph/coder/varint_test.go | 94 +
sdks/go/pkg/beam/core/graph/doc.go | 22 +
sdks/go/pkg/beam/core/graph/edge.go | 382 ++
sdks/go/pkg/beam/core/graph/fn.go | 306 +
sdks/go/pkg/beam/core/graph/graph.go | 126 +
sdks/go/pkg/beam/core/graph/graph_test.go | 68 +
sdks/go/pkg/beam/core/graph/node.go | 70 +
sdks/go/pkg/beam/core/graph/scope.go | 39 +
sdks/go/pkg/beam/core/graph/window/window.go | 70 +
sdks/go/pkg/beam/core/runtime/exec/coder.go | 213 +
sdks/go/pkg/beam/core/runtime/exec/exec.go | 50 +
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 135 +
sdks/go/pkg/beam/core/runtime/exec/nodes.go | 881 +++
sdks/go/pkg/beam/core/runtime/exec/unit.go | 93 +
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 353 ++
sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 111 +
sdks/go/pkg/beam/core/runtime/graphx/doc.go | 29 +
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 905 +++
sdks/go/pkg/beam/core/runtime/graphx/user.go | 109 +
sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go | 18 +
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 763 +++
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto | 217 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 309 +
sdks/go/pkg/beam/core/runtime/harness/gen.go | 18 +
sdks/go/pkg/beam/core/runtime/harness/harness.go | 240 +
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 113 +
sdks/go/pkg/beam/core/runtime/harness/logging.go | 151 +
sdks/go/pkg/beam/core/runtime/harness/session.go | 212 +
.../go/pkg/beam/core/runtime/harness/session.proto | 69 +
.../core/runtime/harness/session/session.pb.go | 444 ++
sdks/go/pkg/beam/core/runtime/harness/translate.go | 343 ++
.../beam/core/runtime/harness/translate_test.go | 298 +
sdks/go/pkg/beam/core/runtime/init.go | 41 +
sdks/go/pkg/beam/core/runtime/options.go | 91 +
sdks/go/pkg/beam/core/runtime/options_test.go | 49 +
sdks/go/pkg/beam/core/runtime/symbols.go | 59 +
sdks/go/pkg/beam/core/runtime/types.go | 62 +
sdks/go/pkg/beam/core/runtime/types_test.go | 74 +
sdks/go/pkg/beam/core/typex/class.go | 174 +
sdks/go/pkg/beam/core/typex/class_test.go | 105 +
sdks/go/pkg/beam/core/typex/fulltype.go | 393 ++
sdks/go/pkg/beam/core/typex/fulltype_test.go | 125 +
sdks/go/pkg/beam/core/typex/special.go | 80 +
sdks/go/pkg/beam/core/util/dot/dot.go | 130 +
sdks/go/pkg/beam/core/util/ioutilx/read.go | 42 +
sdks/go/pkg/beam/core/util/protox/any.go | 101 +
sdks/go/pkg/beam/core/util/protox/any_test.go | 83 +
sdks/go/pkg/beam/core/util/protox/base64.go | 41 +
sdks/go/pkg/beam/core/util/protox/query.go | 45 +
sdks/go/pkg/beam/core/util/reflectx/functions.go | 45 +
sdks/go/pkg/beam/core/util/reflectx/json.go | 32 +
sdks/go/pkg/beam/core/util/reflectx/tags.go | 74 +
sdks/go/pkg/beam/core/util/reflectx/types.go | 111 +
sdks/go/pkg/beam/core/util/symtab/symtab.go | 121 +
sdks/go/pkg/beam/create.go | 95 +
sdks/go/pkg/beam/doc.go | 67 +
sdks/go/pkg/beam/encoding.go | 147 +
sdks/go/pkg/beam/external.go | 111 +
sdks/go/pkg/beam/flatten.go | 69 +
sdks/go/pkg/beam/forward.go | 84 +
sdks/go/pkg/beam/gbk.go | 88 +
sdks/go/pkg/beam/impulse.go | 41 +
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 234 +
sdks/go/pkg/beam/io/bigqueryio/bigquery_test.go | 38 +
sdks/go/pkg/beam/io/textio/filesystem.go | 49 +
sdks/go/pkg/beam/io/textio/gcs/gcs.go | 130 +
sdks/go/pkg/beam/io/textio/local/local.go | 55 +
sdks/go/pkg/beam/io/textio/textio.go | 204 +
sdks/go/pkg/beam/log/log.go | 183 +
sdks/go/pkg/beam/log/standard.go | 37 +
sdks/go/pkg/beam/option.go | 72 +
sdks/go/pkg/beam/options/gcpopts/options.go | 38 +
sdks/go/pkg/beam/pardo.go | 314 +
sdks/go/pkg/beam/partition.go | 96 +
sdks/go/pkg/beam/partition_test.go | 164 +
sdks/go/pkg/beam/pcollection.go | 90 +
sdks/go/pkg/beam/pipeline.go | 75 +
sdks/go/pkg/beam/runner.go | 51 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 325 ++
sdks/go/pkg/beam/runners/dataflow/messages.go | 174 +
sdks/go/pkg/beam/runners/dataflow/translate.go | 326 ++
sdks/go/pkg/beam/runners/direct/direct.go | 270 +
sdks/go/pkg/beam/runners/direct/nodes.go | 241 +
sdks/go/pkg/beam/runners/dot/dot.go | 55 +
sdks/go/pkg/beam/runners/session/session.go | 337 ++
sdks/go/pkg/beam/testing/passert/passert.go | 218 +
sdks/go/pkg/beam/testing/ptest/ptest.go | 59 +
sdks/go/pkg/beam/testing/ptest/symbols.go | 67 +
sdks/go/pkg/beam/transforms/filter/distinct.go | 39 +
.../go/pkg/beam/transforms/filter/distinct_test.go | 70 +
sdks/go/pkg/beam/transforms/filter/filter.go | 88 +
sdks/go/pkg/beam/transforms/filter/filter_test.go | 90 +
sdks/go/pkg/beam/transforms/stats/count.go | 34 +
sdks/go/pkg/beam/transforms/stats/count_test.go | 69 +
sdks/go/pkg/beam/transforms/stats/max.go | 44 +
sdks/go/pkg/beam/transforms/stats/max_switch.go | 138 +
sdks/go/pkg/beam/transforms/stats/max_switch.tmpl | 40 +
sdks/go/pkg/beam/transforms/stats/max_test.go | 134 +
sdks/go/pkg/beam/transforms/stats/mean.go | 91 +
sdks/go/pkg/beam/transforms/stats/mean_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/min.go | 44 +
sdks/go/pkg/beam/transforms/stats/min_switch.go | 138 +
sdks/go/pkg/beam/transforms/stats/min_switch.tmpl | 40 +
sdks/go/pkg/beam/transforms/stats/min_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/sum.go | 44 +
sdks/go/pkg/beam/transforms/stats/sum_switch.go | 102 +
sdks/go/pkg/beam/transforms/stats/sum_switch.tmpl | 37 +
sdks/go/pkg/beam/transforms/stats/sum_test.go | 117 +
sdks/go/pkg/beam/transforms/stats/util.go | 48 +
sdks/go/pkg/beam/transforms/top/top.go | 173 +
sdks/go/pkg/beam/transforms/top/top_test.go | 125 +
sdks/go/pkg/beam/util.go | 110 +
sdks/go/pkg/beam/util/gcsx/gcs.go | 40 +
sdks/go/pkg/beam/validate.go | 76 +
sdks/go/pkg/beam/x/beamx/run.go | 39 +
sdks/go/pkg/beam/x/debug/head.go | 68 +
sdks/go/pkg/beam/x/debug/print.go | 94 +
sdks/go/pom.xml | 3 +
159 files changed, 42783 insertions(+), 1 deletion(-)
--
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" <co...@beam.apache.org>.