You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2018/05/16 04:08:30 UTC
[beam-site] branch mergebot updated (59bd643 -> b89b181)
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.
from 59bd643 This closes #441
add 9441be0 Prepare repository for deployment.
new b040c92 Add Go SDK quickstart and minimal documentation page
new 668a523 Add Go to the Wordcount example
new 6c16369 Fix mailing list links and docker image
new b89b181 This closes #440
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
content/contribute/index.html | 13 ++
src/_includes/section-menu/get-started.html | 1 +
src/_includes/section-menu/sdks.html | 11 +
src/documentation/index.md | 3 +-
src/documentation/sdks/go.md | 20 ++
src/get-started/beam-overview.md | 3 +-
src/get-started/index.md | 2 +-
src/get-started/mobile-gaming-example.md | 3 +
src/get-started/quickstart-go.md | 67 ++++++
src/get-started/wordcount-example.md | 342 ++++++++++++++++++++++++++++
src/images/logos/sdks/go.png | Bin 0 -> 15475 bytes
src/index.md | 2 +
12 files changed, 464 insertions(+), 3 deletions(-)
create mode 100644 src/documentation/sdks/go.md
create mode 100644 src/get-started/quickstart-go.md
create mode 100644 src/images/logos/sdks/go.png
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 02/04: Add Go to the Wordcount example
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 668a523269aea21b140562b346f9a549001a20da
Author: Henning Rohde <he...@google.com>
AuthorDate: Mon May 14 18:49:52 2018 -0700
Add Go to the Wordcount example
---
src/get-started/mobile-gaming-example.md | 3 +
src/get-started/wordcount-example.md | 342 +++++++++++++++++++++++++++++++
2 files changed, 345 insertions(+)
diff --git a/src/get-started/mobile-gaming-example.md b/src/get-started/mobile-gaming-example.md
index 49d1530..7698da4 100644
--- a/src/get-started/mobile-gaming-example.md
+++ b/src/get-started/mobile-gaming-example.md
@@ -27,6 +27,9 @@ This section provides a walkthrough of a series of example Apache Beam pipelines
{:.language-py}
> **Note**: These examples assume some familiarity with the Beam programming model. If you haven't already, we recommend familiarizing yourself with the programming model documentation and running a basic example pipeline before continuing.
+> **Note**: MobileGaming is not yet available for the Go SDK. There is an open issue for this
+([BEAM-4293](https://issues.apache.org/jira/browse/BEAM-4293)).
+
Every time a user plays an instance of our hypothetical mobile game, they generate a data event. Each data event consists of the following information:
- The unique ID of the user playing the game.
diff --git a/src/get-started/wordcount-example.md b/src/get-started/wordcount-example.md
index a334915..3933c8d 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -16,6 +16,7 @@ redirect_from: /use/wordcount-example/
<ul>
<li data-type="language-java">Java SDK</li>
<li data-type="language-py">Python SDK</li>
+ <li data-type="language-go">Go SDK</li>
</ul>
</nav>
@@ -45,6 +46,7 @@ read from a text file, apply transforms to tokenize and count the words, and
write the data to an output text file.
{:.language-java}
+{:.language-go}
This example hard-codes the locations for its input and output files and doesn't
perform any error checking; it is intended to only show you the "bare bones" of
creating a Beam pipeline. This lack of parameterization makes this particular
@@ -60,6 +62,11 @@ $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.MinimalWordCou
python -m apache_beam.examples.wordcount_minimal --input YOUR_INPUT_FILE --output counts
```
+```go
+$ go install github.com/apache/beam/sdks/go/examples/minimal_wordcount
+$ minimal_wordcount
+```
+
{:.language-java}
To view the full code in Java, see
**[MinimalWordCount](https://github.com/apache/beam/blob/master/examples/java/src/main/java/org/apache/beam/examples/MinimalWordCount.java).**
@@ -68,6 +75,10 @@ To view the full code in Java, see
To view the full code in Python, see
**[wordcount_minimal.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py).**
+{:.language-go}
+To view the full code in Go, see
+**[wordcount_minimal.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
+
**Key Concepts:**
* Creating the Pipeline
@@ -83,12 +94,16 @@ excerpts from the MinimalWordCount pipeline.
### Creating the pipeline
+{:.language-java}
+{:.language-py}
In this example, the code first creates a `PipelineOptions` object. This object
lets us set various options for our pipeline, such as the pipeline runner that
will execute our pipeline and any runner-specific configuration required by the
chosen runner. In this example we set these options programmatically, but more
often, command-line arguments are used to set `PipelineOptions`.
+{:.language-java}
+{:.language-py}
You can specify a runner for executing your pipeline, such as the
`DataflowRunner` or `SparkRunner`. If you omit specifying a runner, as in this
example, your pipeline executes locally using the `DirectRunner`. In the next
@@ -106,10 +121,17 @@ sections, we will specify the pipeline's runner.
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_options
%}```
+{:.language-java}
+{:.language-py}
The next step is to create a `Pipeline` object with the options we've just
constructed. The Pipeline object builds up the graph of transformations to be
executed, associated with that particular pipeline.
+{:.language-go}
+The first step is to create a `Pipeline` object. It builds up the graph of
+transformations to be executed, associated with that particular pipeline.
+The scope allows grouping into composite transforms.
+
```java
Pipeline p = Pipeline.create(options);
```
@@ -118,6 +140,11 @@ Pipeline p = Pipeline.create(options);
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_create
%}```
+```go
+p := beam.NewPipeline
+s := p.Root()
+```
+
### Applying pipeline transforms
The MinimalWordCount pipeline contains several transforms to read data into the
@@ -150,6 +177,10 @@ The MinimalWordCount pipeline contains five transforms:
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_read
%}```
+ ```go
+ lines := textio.Read(s, "gs://apache-beam-samples/shakespeare/*")
+ ```
+
2. This transform splits the lines in `PCollection<String>`, where each element
is an individual word in Shakespeare's collected texts.
As an alternative, it would have been possible to use a
@@ -171,6 +202,14 @@ The MinimalWordCount pipeline contains five transforms:
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_pardo
%}```
+ ```go
+ words := beam.ParDo(s, func(line string, emit func(string)) {
+ for _, word := range wordRE.FindAllString(line, -1) {
+ emit(word)
+ }
+ }, lines)
+ ```
+
3. The SDK-provided `Count` transform is a generic transform that takes a
`PCollection` of any type, and returns a `PCollection` of key/value pairs.
Each key represents a unique element from the input collection, and each
@@ -190,6 +229,10 @@ The MinimalWordCount pipeline contains five transforms:
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_count
%}```
+ ```go
+ counted := stats.Count(s, words)
+ ```
+
4. The next transform formats each of the key/value pairs of unique words and
occurrence counts into a printable string suitable for writing to an output
file.
@@ -208,6 +251,12 @@ The MinimalWordCount pipeline contains five transforms:
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_map
%}```
+ ```go
+ formatted := beam.ParDo(s, func(w string, c int) string {
+ return fmt.Sprintf("%s: %v", w, c)
+ }, counted)
+ ```
+
5. A text file write transform. This transform takes the final `PCollection` of
formatted Strings as input and writes each element to an output text file.
Each element in the input `PCollection` represents one line of text in the
@@ -221,14 +270,28 @@ The MinimalWordCount pipeline contains five transforms:
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_write
%}```
+ ```go
+ textio.Write(s, "wordcounts.txt", formatted)
+ ```
+
+{:.language-java}
+{:.language-py}
Note that the `Write` transform produces a trivial result value of type `PDone`,
which in this case is ignored.
+{:.language-go}
+Note that the `Write` transform returns no PCollections.
+
### Running the pipeline
+{:.language-java}
+{:.language-py}
Run the pipeline by calling the `run` method, which sends your pipeline to be
executed by the pipeline runner that you specified in your `PipelineOptions`.
+{:.language-go}
+Run the pipeline by passing it to a runner.
+
```java
p.run().waitUntilFinish();
```
@@ -237,6 +300,12 @@ p.run().waitUntilFinish();
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_minimal_run
%}```
+```go
+direct.Execute(context.Background(), p)
+```
+
+{:.language-java}
+{:.language-py}
Note that the `run` method is asynchronous. For a blocking execution, call the
<span class="language-java">`waitUntilFinish`</span>
<span class="language-py">`wait_until_finish`</span> method on the result object
@@ -340,6 +409,48 @@ python -m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespea
To view the full code in Python, see
**[wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount.py).**
+**To run this example in Go:**
+
+{:.runner-direct}
+```
+$ go install github.com/apache/beam/sdks/go/examples/wordcount
+$ wordcount --input <PATH_TO_INPUT_FILE> --output counts
+```
+
+{:.runner-apex}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-local}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-cluster}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-spark}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-dataflow}
+```
+$ go install github.com/apache/beam/sdks/go/examples/wordcount
+$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://<your-gcs-bucket>/counts \
+ --runner dataflow \
+ --project your-gcp-project \
+ --temp_location gs://<your-gcs-bucket>/tmp/ \
+ --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+```
+
+To view the full code in Go, see
+**[wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/wordcount/wordcount.go).**
+
**New Concepts:**
* Applying `ParDo` with an explicit `DoFn`
@@ -351,6 +462,8 @@ pipeline code into smaller sections.
### Specifying explicit DoFns
+{:.language-java}
+{:.language-py}
When using `ParDo` transforms, you need to specify the processing operation that
gets applied to each element in the input `PCollection`. This processing
operation is a subclass of the SDK class `DoFn`. You can create the `DoFn`
@@ -359,6 +472,15 @@ done in the previous example (MinimalWordCount). However, it's often a good
idea to define the `DoFn` at the global level, which makes it easier to unit
test and can make the `ParDo` code more readable.
+{:.language-go}
+When using `ParDo` transforms, you need to specify the processing operation that
+gets applied to each element in the input `PCollection`. This processing
+operation is either a named function or a struct with specially-named methods. You
+can use anomynous functions (but not closures). However, it's often a good
+idea to define the `DoFn` at the global level, which makes it easier to unit
+test and can make the `ParDo` code more readable.
+
+
```java
// In this example, ExtractWordsFn is a DoFn that is defined as a static class:
@@ -378,17 +500,38 @@ static class ExtractWordsFn extends DoFn<String, String> {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_wordcount_dofn
%}```
+```go
+// In this example, extractFn is a DoFn that is defined as a function:
+
+func extractFn(ctx context.Context, line string, emit func(string)) {
+ ...
+}
+```
+
### Creating composite transforms
+{:.language-java}
+{:.language-py}
If you have a processing operation that consists of multiple transforms or
`ParDo` steps, you can create it as a subclass of `PTransform`. Creating a
`PTransform` subclass allows you to encapsulate complex transforms, can make
your pipeline's structure more clear and modular, and makes unit testing easier.
+{:.language-go}
+If you have a processing operation that consists of multiple transforms or
+`ParDo` steps, you can use a normal Go function to encapsulate them. You can
+furthermore use a named subscope to group them as a composite transform visible
+for monitoring.
+
+{:.language-java}
+{:.language-py}
In this example, two transforms are encapsulated as the `PTransform` subclass
`CountWords`. `CountWords` contains the `ParDo` that runs `ExtractWordsFn` and
the SDK-provided `Count` transform.
+{:.language-go}
+In this example, two transforms are encapsulated as a `CountWords` function.
+
When `CountWords` is defined, we specify its ultimate input and output; the
input is the `PCollection<String>` for the extraction operation, and the output
is the `PCollection<KV<String, Long>>` produced by the count operation.
@@ -424,6 +567,18 @@ public static void main(String[] args) throws IOException {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_wordcount_composite
%}```
+```go
+func CountWords(s beam.Scope, lines beam.PCollection) beam.PCollection {
+ s = s.Scope("CountWords")
+
+ // Convert lines of text into individual words.
+ col := beam.ParDo(s, extractFn, lines)
+
+ // Count the number of times each word occurs.
+ return stats.Count(s, col)
+}
+```
+
### Using parameterizable PipelineOptions
You can hard-code various execution options when you run your pipeline. However,
@@ -431,9 +586,15 @@ the more common way is to define your own configuration options via command-line
argument parsing. Defining your configuration options via the command-line makes
the code more easily portable across different runners.
+{:.language-java}
+{:.language-py}
Add arguments to be processed by the command-line parser, and specify default
values for them. You can then access the options values in your pipeline code.
+{:.language-go}
+You can use the standard `flag` package for this purpose.
+
+
```java
public static interface WordCountOptions extends PipelineOptions {
@Description("Path of the file to read from")
@@ -455,6 +616,18 @@ public static void main(String[] args) {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:examples_wordcount_wordcount_options
%}```
+```go
+var input = flag.String("input", "gs://apache-beam-samples/shakespeare/kinglear.txt", "File(s) to read.")
+
+func main() {
+ ...
+ p := beam.NewPipeline()
+ s := p.Root()
+
+ lines := textio.Read(s, *input)
+ ...
+```
+
## DebuggingWordCount example
The DebuggingWordCount example demonstrates some best practices for
@@ -547,6 +720,48 @@ python -m apache_beam.examples.wordcount_debugging --input gs://dataflow-samples
To view the full code in Python, see
**[wordcount_debugging.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_debugging.py).**
+**To run this example in Go:**
+
+{:.runner-direct}
+```
+$ go install github.com/apache/beam/sdks/go/examples/debugging_wordcount
+$ debugging_wordcount --input <PATH_TO_INPUT_FILE> --output counts
+```
+
+{:.runner-apex}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-local}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-cluster}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-spark}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-dataflow}
+```
+$ go install github.com/apache/beam/sdks/go/examples/debugging_wordcount
+$ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://<your-gcs-bucket>/counts \
+ --runner dataflow \
+ --project your-gcp-project \
+ --temp_location gs://<your-gcs-bucket>/tmp/ \
+ --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+```
+
+To view the full code in Go, see
+**[debugging_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/debugging_wordcount/debugging_wordcount.go).**
+
**New Concepts:**
* Logging
@@ -585,6 +800,22 @@ public class DebuggingWordCount {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:example_wordcount_debugging_logging
%}```
+```go
+type filterFn struct {
+ ...
+}
+
+func (f *filterFn) ProcessElement(ctx context.Context, word string, count int, emit func(string, int)) {
+ if f.re.MatchString(word) {
+ // Log at the "INFO" level each element that we match.
+ log.Infof(ctx, "Matched: %v", word)
+ emit(word, count)
+ } else {
+ // Log at the "DEBUG" level each element that is not matched.
+ log.Debugf(ctx, "Did not match: %v", word)
+ }
+}
+```
#### Direct Runner
@@ -602,6 +833,8 @@ that Cloud Dataflow has spun up to complete your job. Logging statements in your
pipeline's `DoFn` instances will appear in Stackdriver Logging as your pipeline
runs.
+{:.language-java}
+{:.language-py}
You can also control the worker log levels. Cloud Dataflow workers that execute
user code are configured to log to Stackdriver Logging by default at "INFO" log
level and higher. You can override log levels for specific logging namespaces by
@@ -611,6 +844,8 @@ when executing a pipeline using the Cloud Dataflow service, Stackdriver Logging
will contain only "DEBUG" or higher level logs for the package in addition to
the default "INFO" or higher level logs.
+{:.language-java}
+{:.language-py}
The default Cloud Dataflow worker logging configuration can be overridden by
specifying `--defaultWorkerLogLevel=<one of TRACE, DEBUG, INFO, WARN, ERROR>`.
For example, by specifying `--defaultWorkerLogLevel=DEBUG` when executing a
@@ -635,18 +870,26 @@ or DEBUG significantly increases the amount of logs output.
### Testing your pipeline with asserts
+{:.language-java}
+{:.language-py}
<span class="language-java">`PAssert`</span><span class="language-py">`assert_that`</span>
is a set of convenient PTransforms in the style of Hamcrest's collection
matchers that can be used when writing pipeline level tests to validate the
contents of PCollections. Asserts are best used in unit tests with small data
sets.
+{:.language-go}
+The `passert` package contains convenient PTransforms that can be used when
+writing pipeline level tests to validate the contents of PCollections. Asserts
+are best used in unit tests with small data sets.
+
{:.language-java}
The following example verifies that the set of filtered words matches our
expected counts. The assert does not produce any output, and the pipeline only
succeeds if all of the expectations are met.
{:.language-py}
+{:.language-go}
The following example verifies that two collections contain the same values. The
assert does not produce any output, and the pipeline only succeeds if all of the
expectations are met.
@@ -670,6 +913,11 @@ with TestPipeline() as p:
assert_that(p | Create([1, 2, 3]), equal_to([1, 2, 3]))
```
+```go
+...
+passert.Equals(s, formatted, "Flourish: 3", "stomach: 1")
+```
+
{:.language-java}
See [DebuggingWordCountTest](https://github.com/apache/beam/blob/master/examples/java/src/test/java/org/apache/beam/examples/DebuggingWordCountTest.java)
for an example unit test.
@@ -781,6 +1029,49 @@ python -m apache_beam.examples.windowed_wordcount --input YOUR_INPUT_FILE \
To view the full code in Python, see
**[windowed_wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/windowed_wordcount.py).**
+**To run this example in Python:**
+
+{:.runner-direct}
+```
+$ go install github.com/apache/beam/sdks/go/examples/windowed_wordcount
+$ windowed_wordcount --input <PATH_TO_INPUT_FILE> --output counts
+```
+
+{:.runner-apex}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-local}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-flink-cluster}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-spark}
+```
+This runner is not yet available for the Go SDK.
+```
+
+{:.runner-dataflow}
+```
+$ go install github.com/apache/beam/sdks/go/examples/windowed_wordcount
+$ windowed_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://<your-gcs-bucket>/counts \
+ --runner dataflow \
+ --project your-gcp-project \
+ --temp_location gs://<your-gcs-bucket>/tmp/ \
+ --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+```
+
+To view the full code in Go, see
+**[windowed_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/windowed_wordcount/windowed_wordcount.go).**
+
+
### Unbounded and bounded pipeline input modes
Beam allows you to create a single pipeline that can handle both bounded and
@@ -811,6 +1102,17 @@ public static void main(String[] args) throws IOException {
# This feature is not yet available in the Beam SDK for Python.
```
+```go
+func main() {
+ ...
+ p := beam.NewPipeline()
+ s := p.Root()
+
+ lines := textio.Read(s, *input)
+ ...
+}
+```
+
### Adding timestamps to data
Each element in a `PCollection` has an associated [timestamp]({{ site.baseurl }}/documentation/programming-guide#element-timestamps).
@@ -832,6 +1134,10 @@ each element in the `PCollection`.
# This feature is not yet available in the Beam SDK for Python.
```
+```go
+timestampedLines := beam.ParDo(s, &addTimestampFn{Min: mtime.Now()}, lines)
+```
+
Below is the code for `AddTimestampFn`, a `DoFn` invoked by `ParDo`, that sets
the data element of the timestamp given the element itself. For example, if the
elements were log lines, this `ParDo` could parse the time out of the log string
@@ -869,6 +1175,21 @@ static class AddTimestampFn extends DoFn<String, String> {
# This feature is not yet available in the Beam SDK for Python.
```
+```go
+type addTimestampFn struct {
+ Min beam.EventTime `json:"min"`
+}
+
+func (f *addTimestampFn) ProcessElement(x beam.X) (beam.EventTime, beam.X) {
+ timestamp := f.Min.Add(time.Duration(rand.Int63n(2 * time.Hour.Nanoseconds())))
+ return timestamp, x
+}
+```
+
+{:.language-go}
+Note that the use of the `beam.X` "type variable" allows the transform to be
+used for any type.
+
### Windowing
Beam uses a concept called **Windowing** to subdivide a `PCollection` into
@@ -890,6 +1211,11 @@ PCollection<String> windowedWords = input
# This feature is not yet available in the Beam SDK for Python.
```
+```go
+windowedLines := beam.WindowInto(s, window.NewFixedWindows(time.Minute), timestampedLines)
+
+```
+
### Reusing PTransforms over windowed PCollections
You can reuse existing PTransforms that were created for manipulating simple
@@ -903,6 +1229,10 @@ PCollection<KV<String, Long>> wordCounts = windowedWords.apply(new WordCount.Cou
# This feature is not yet available in the Beam SDK for Python.
```
+```go
+counted := wordcount.CountWords(s, windowedLines)
+```
+
## StreamingWordCount example
The StreamingWordCount example is a streaming pipeline that reads Pub/Sub
@@ -967,6 +1297,11 @@ python -m apache_beam.examples.streaming_wordcount \
To view the full code in Python, see
**[streaming_wordcount.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/streaming_wordcount.py).**
+**To run this example in Go:**
+
+> **Note:** StreamingWordCount is not yet available for the Go SDK. There is an open issue for this
+([BEAM-4292](https://issues.apache.org/jira/browse/BEAM-4292)).
+
### Reading an unbounded data set
@@ -985,6 +1320,10 @@ messages from a Pub/Sub subscription or topic using
else:
lines = p | beam.io.ReadStringsFromPubSub(topic=known_args.input_topic)
```
+
+```go
+ // This example is not currently available for the Beam SDK for Go.
+```
### Writing unbounded results
When the input is unbounded, the same is true of the output `PCollection`. As
@@ -1003,4 +1342,7 @@ using [`beam.io.WriteStringsToPubSub`]({{ site.baseurl }}/documentation/sdks/pyd
# Write to Pub/Sub
output | beam.io.WriteStringsToPubSub(known_args.output_topic)
```
+```go
+ // This example is not currently available for the Beam SDK for Go.
+```
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 03/04: Fix mailing list links and docker image
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit 6c16369bc75ab89c97665d8e94218e26ab9899c0
Author: Henning Rohde <he...@google.com>
AuthorDate: Tue May 15 16:43:03 2018 -0700
Fix mailing list links and docker image
---
src/get-started/quickstart-go.md | 6 +++---
src/get-started/wordcount-example.md | 6 +++---
2 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/src/get-started/quickstart-go.md b/src/get-started/quickstart-go.md
index 17dc3a2..9cb0408 100644
--- a/src/get-started/quickstart-go.md
+++ b/src/get-started/quickstart-go.md
@@ -53,7 +53,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
- --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+ --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
```
## Next Steps
@@ -62,6 +62,6 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
and look through the [godoc](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam).
* Walk through these WordCount examples in the [WordCount Example Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
* Dive in to some of our favorite [articles and presentations]({{ site.baseurl }}/documentation/resources).
-* Join the Beam [users@]({{ site.baseurl }}/get-started/support#mailing-lists) mailing list.
+* Join the Beam [users@]({{ site.baseurl }}/community/contact-us) mailing list.
-Please don't hesitate to [reach out]({{ site.baseurl }}/get-started/support) if you encounter any issues!
+Please don't hesitate to [reach out]({{ site.baseurl }}/community/contact-us) if you encounter any issues!
diff --git a/src/get-started/wordcount-example.md b/src/get-started/wordcount-example.md
index 3933c8d..41f21f5 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -445,7 +445,7 @@ $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
- --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+ --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
```
To view the full code in Go, see
@@ -756,7 +756,7 @@ $ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
- --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+ --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
```
To view the full code in Go, see
@@ -1065,7 +1065,7 @@ $ windowed_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
--runner dataflow \
--project your-gcp-project \
--temp_location gs://<your-gcs-bucket>/tmp/ \
- --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+ --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
```
To view the full code in Go, see
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 01/04: Add Go SDK quickstart and minimal documentation
page
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit b040c928c11d3c515ae52603fe27c1208f7b1c86
Author: Henning Rohde <he...@google.com>
AuthorDate: Mon May 14 14:28:22 2018 -0700
Add Go SDK quickstart and minimal documentation page
---
src/_includes/section-menu/get-started.html | 1 +
src/_includes/section-menu/sdks.html | 11 +++++
src/documentation/index.md | 3 +-
src/documentation/sdks/go.md | 20 +++++++++
src/get-started/beam-overview.md | 3 +-
src/get-started/index.md | 2 +-
src/get-started/quickstart-go.md | 67 ++++++++++++++++++++++++++++
src/images/logos/sdks/go.png | Bin 0 -> 15475 bytes
src/index.md | 2 +
9 files changed, 106 insertions(+), 3 deletions(-)
diff --git a/src/_includes/section-menu/get-started.html b/src/_includes/section-menu/get-started.html
index e60ebd9..74a5a71 100644
--- a/src/_includes/section-menu/get-started.html
+++ b/src/_includes/section-menu/get-started.html
@@ -6,6 +6,7 @@
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/get-started/quickstart-java/">Quickstart - Java</a></li>
<li><a href="{{ site.baseurl }}/get-started/quickstart-py/">Quickstart - Python</a></li>
+ <li><a href="{{ site.baseurl }}/get-started/quickstart-go/">Quickstart - Go</a></li>
</ul>
</li>
<li>
diff --git a/src/_includes/section-menu/sdks.html b/src/_includes/section-menu/sdks.html
index ddcd1e2..ed988fe 100644
--- a/src/_includes/section-menu/sdks.html
+++ b/src/_includes/section-menu/sdks.html
@@ -30,6 +30,17 @@
</li>
<li>
+ <span class="section-nav-list-title">Go SDK</span>
+ <ul class="section-nav-list">
+ <li><a href="{{ site.baseurl }}/documentation/sdks/go/">Go SDK overview</a></li>
+ <li><a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam" target="_blank">Go SDK API reference <img src="{{ site.baseurl }}/images/external-link-icon.png"
+ width="14" height="14"
+ alt="External link."></a>
+ </li>
+ </ul>
+</li>
+
+<li>
<span class="section-nav-list-title">DSLs</span>
<ul class="section-nav-list">
<li><a href="{{ site.baseurl }}/documentation/dsls/sql/">SQL</a></li>
diff --git a/src/documentation/index.md b/src/documentation/index.md
index d54543d..e7b36cc 100644
--- a/src/documentation/index.md
+++ b/src/documentation/index.md
@@ -32,6 +32,7 @@ Find status and reference information on all of the available Beam SDKs.
* [Java SDK]({{ site.baseurl }}/documentation/sdks/java/)
* [Python SDK]({{ site.baseurl }}/documentation/sdks/python/)
+* [Go SDK]({{ site.baseurl }}/documentation/sdks/go/)
## Runners
@@ -50,4 +51,4 @@ A Beam Runner runs a Beam pipeline on a specific (often distributed) data proces
Beam is designed to enable pipelines to be portable across different runners. However, given every runner has different capabilities, they also have different abilities to implement the core concepts in the Beam model. The [Capability Matrix]({{ site.baseurl }}/documentation/runners/capability-matrix/) provides a detailed comparison of runner functionality.
-Once you have chosen which runner to use, see that runner's page for more information about any initial runner-specific setup as well as any required or optional `PipelineOptions` for configuring it's execution. You may also want to refer back to the Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java) or [Python]({{ site.baseurl }}/get-started/quickstart-py) for instructions on executing the sample WordCount pipeline.
+Once you have chosen which runner to use, see that runner's page for more information about any initial runner-specific setup as well as any required or optional `PipelineOptions` for configuring it's execution. You may also want to refer back to the Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java), [Python]({{ site.baseurl }}/get-started/quickstart-py) or [Go]({{ site.baseurl }}/get-started/quickstart-go) for instructions on executing the sample WordCount pipeline.
diff --git a/src/documentation/sdks/go.md b/src/documentation/sdks/go.md
new file mode 100644
index 0000000..14891b1
--- /dev/null
+++ b/src/documentation/sdks/go.md
@@ -0,0 +1,20 @@
+---
+layout: section
+title: "Beam Go SDK"
+section_menu: section-menu/sdks.html
+permalink: /documentation/sdks/go/
+---
+# Apache Beam Go SDK
+
+The Go SDK for Apache Beam provides a simple, powerful API for building both batch and streaming parallel data processing pipelines. It is based on the following
+[design](https://s.apache.org/beam-go-sdk-design-rfc).
+
+## Get Started with the Go SDK
+
+Get started with the [Beam Go SDK quickstart]({{ site.baseurl }}/get-started/quickstart-go) to set up your development environment and run an example pipeline. Then, read through the [Beam programming guide]({{ site.baseurl }}/documentation/programming-guide) to learn the basic concepts that apply to all SDKs in Beam.
+
+See the [godoc](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam) for more detailed information.
+
+## Status
+
+The Go SDK is currently experimental, does not yet offer any compatibility guarantees and is not recommended for production usage. It supports most features, but not all.
diff --git a/src/get-started/beam-overview.md b/src/get-started/beam-overview.md
index b23811c..1b2fbfc 100644
--- a/src/get-started/beam-overview.md
+++ b/src/get-started/beam-overview.md
@@ -22,6 +22,7 @@ Beam currently supports the following language-specific SDKs:
* Java ![Java logo]({{ "/images/logos/sdks/java.png" | prepend: site.baseurl }})
* Python ![Python logo]({{ "/images/logos/sdks/python.png" | prepend: site.baseurl }})
+* Go ![Go logo]({{ "/images/logos/sdks/go.png" | prepend: site.baseurl }}){: height="45px"}
A Scala ![Scala logo]({{ "/images/logos/sdks/scala.png" | prepend: site.baseurl }}){: height="45px"} interface is also available as [Scio](https://github.com/spotify/scio).
@@ -43,7 +44,7 @@ Beam currently supports Runners that work with the following distributed process
Get started using Beam for your data processing tasks.
-1. Follow the Quickstart for the [Java SDK]({{ site.baseurl }}/get-started/quickstart-java) or the [Python SDK]({{ site.baseurl }}/get-started/quickstart-py).
+1. Follow the Quickstart for the [Java SDK]({{ site.baseurl }}/get-started/quickstart-java), the [Python SDK]({{ site.baseurl }}/get-started/quickstart-py) or the [Go SDK]({{ site.baseurl }}/get-started/quickstart-go).
2. See the [WordCount Examples Walkthrough]({{ site.baseurl }}/get-started/wordcount-example) for examples that introduce various features of the SDKs.
diff --git a/src/get-started/index.md b/src/get-started/index.md
index 41bfd57..7f9d3e8 100644
--- a/src/get-started/index.md
+++ b/src/get-started/index.md
@@ -15,7 +15,7 @@ Learn to use Beam to create data processing pipelines that run on supported proc
Learn about the Beam model, the currently available Beam SDKs and Runners, and Beam's native I/O connectors.
-#### Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java) or [Python]({{ site.baseurl }}/get-started/quickstart-py)
+#### Quickstart for [Java]({{ site.baseurl }}/get-started/quickstart-java), [Python]({{ site.baseurl }}/get-started/quickstart-py) or [Go]({{ site.baseurl }}/get-started/quickstart-go)
Learn how to set up a Beam project and run a simple example Beam pipeline on your local machine.
diff --git a/src/get-started/quickstart-go.md b/src/get-started/quickstart-go.md
new file mode 100644
index 0000000..17dc3a2
--- /dev/null
+++ b/src/get-started/quickstart-go.md
@@ -0,0 +1,67 @@
+---
+layout: section
+title: "Beam Quickstart for Go"
+permalink: /get-started/quickstart-go/
+section_menu: section-menu/get-started.html
+---
+
+# Apache Beam Go SDK Quickstart
+
+This Quickstart will walk you through executing your first Beam pipeline to run [WordCount]({{ site.baseurl }}/get-started/wordcount-example), written using Beam's [Go SDK]({{ site.baseurl }}/documentation/sdks/go), on a [runner]({{ site.baseurl }}/documentation#runners) of your choice.
+
+* TOC
+{:toc}
+
+## Set up your environment
+
+The Beam SDK for Go requires `go` version 1.10 or newer. It can be downloaded [here](https://golang.org/). Check that you have version 1.10 by running:
+
+```
+$ go --version
+```
+
+## Get the SDK and the examples
+
+The easiest way to obtain the Apache Beam Go SDK is via `go get`:
+
+```
+$ go get -u github.com/apache/beam/sdks/go/...
+```
+
+For development of the Go SDK itself, see [BUILD.md](https://github.com/apache/beam/blob/master/sdks/go/BUILD.md) for details.
+
+## Run wordcount
+
+The Apache Beam
+[examples](https://github.com/apache/beam/tree/master/sdks/go/examples)
+directory has many examples. All examples can be run by passing the
+required arguments described in the examples.
+
+For example, to run `wordcount`, run:
+
+{:.runner-direct}
+```
+$ go install github.com/apache/beam/sdks/go/examples/wordcount
+$ wordcount --input <PATH_TO_INPUT_FILE> --output counts
+```
+
+{:.runner-dataflow}
+```
+$ go install github.com/apache/beam/sdks/go/examples/wordcount
+$ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
+ --output gs://<your-gcs-bucket>/counts \
+ --runner dataflow \
+ --project your-gcp-project \
+ --temp_location gs://<your-gcs-bucket>/tmp/ \
+ --worker_harness_container_image=herohde-docker-apache.bintray.io/beam/go:20180514
+```
+
+## Next Steps
+
+* Learn more about the [Beam SDK for Go]({{ site.baseurl }}/documentation/sdks/go/)
+ and look through the [godoc](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam).
+* Walk through these WordCount examples in the [WordCount Example Walkthrough]({{ site.baseurl }}/get-started/wordcount-example).
+* Dive in to some of our favorite [articles and presentations]({{ site.baseurl }}/documentation/resources).
+* Join the Beam [users@]({{ site.baseurl }}/get-started/support#mailing-lists) mailing list.
+
+Please don't hesitate to [reach out]({{ site.baseurl }}/get-started/support) if you encounter any issues!
diff --git a/src/images/logos/sdks/go.png b/src/images/logos/sdks/go.png
new file mode 100644
index 0000000..d6d98c3
Binary files /dev/null and b/src/images/logos/sdks/go.png differ
diff --git a/src/index.md b/src/index.md
index 94a4c66..bb5eaec 100644
--- a/src/index.md
+++ b/src/index.md
@@ -53,6 +53,7 @@ cards:
<div class="hero__ctas">
<a class="button" href="{{'/get-started/quickstart-java/'|prepend:site.baseurl}}">Java Quickstart</a>
<a class="button" href="{{'/get-started/quickstart-py/'|prepend:site.baseurl}}">Python Quickstart</a>
+ <a class="button" href="{{'/get-started/quickstart-go/'|prepend:site.baseurl}}">Go Quickstart</a>
</div>
</div>
</div>
@@ -153,5 +154,6 @@ cards:
<div class="ctas__ctas">
<a class="button" href="{{'/get-started/quickstart-java/'|prepend:site.baseurl}}">Java Quickstart</a>
<a class="button" href="{{'/get-started/quickstart-py/'|prepend:site.baseurl}}">Python Quickstart</a>
+ <a class="button" href="{{'/get-started/quickstart-go/'|prepend:site.baseurl}}">Go Quickstart</a>
</div>
</div>
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.
[beam-site] 04/04: This closes #440
Posted by me...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git
commit b89b181669f9849ffa361b6b97452979468dcf17
Merge: 9441be0 6c16369
Author: Mergebot <me...@apache.org>
AuthorDate: Tue May 15 21:08:10 2018 -0700
This closes #440
src/_includes/section-menu/get-started.html | 1 +
src/_includes/section-menu/sdks.html | 11 +
src/documentation/index.md | 3 +-
src/documentation/sdks/go.md | 20 ++
src/get-started/beam-overview.md | 3 +-
src/get-started/index.md | 2 +-
src/get-started/mobile-gaming-example.md | 3 +
src/get-started/quickstart-go.md | 67 ++++++
src/get-started/wordcount-example.md | 342 ++++++++++++++++++++++++++++
src/images/logos/sdks/go.png | Bin 0 -> 15475 bytes
src/index.md | 2 +
11 files changed, 451 insertions(+), 3 deletions(-)
--
To stop receiving notification emails like this one, please contact
mergebot-role@apache.org.