You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by me...@apache.org on 2019/01/07 18:14:56 UTC
[beam] branch master updated: Add some Go examples to the code
snippets. (#7084)
This is an automated email from the ASF dual-hosted git repository.
melap pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 185eca3 Add some Go examples to the code snippets. (#7084)
185eca3 is described below
commit 185eca3ea8cb94ce3e89bbfa75189035fa9e83df
Author: Antonio D'souza <ad...@gmail.com>
AuthorDate: Mon Jan 7 13:14:45 2019 -0500
Add some Go examples to the code snippets. (#7084)
---
website/src/documentation/programming-guide.md | 46 ++++++++++++++++++++++++--
1 file changed, 43 insertions(+), 3 deletions(-)
diff --git a/website/src/documentation/programming-guide.md b/website/src/documentation/programming-guide.md
index 48f4ead..7ee0a4a 100644
--- a/website/src/documentation/programming-guide.md
+++ b/website/src/documentation/programming-guide.md
@@ -87,7 +87,7 @@ A typical Beam driver program works as follows:
* **Apply** `PTransforms` to each `PCollection`. Transforms can change, filter,
group, analyze, or otherwise process the elements in a `PCollection`. A
transform creates a new output `PCollection` *without modifying the input
- collection*. A typical pipeline applies subsequent transforms to the each new
+ collection*. A typical pipeline applies subsequent transforms to each new
output `PCollection` in turn until processing is complete. However, note that
a pipeline does not have to be a single straight line of transforms applied
one after another: think of `PCollection`s as variables and `PTransform`s as
@@ -129,6 +129,10 @@ Pipeline p = Pipeline.create(options);
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipelines_constructing_creating
%}
```
+```go
+// In order to start creating the pipeline for execution, a Pipeline object and a Scope object are needed.
+p, s := beam.NewPipelineWithRoot()
+```
### 2.1. Configuring pipeline options {#configuring-pipeline-options}
@@ -159,6 +163,10 @@ PipelineOptions options =
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipelines_constructing_creating
%}
```
+```go
+// If beamx or Go flags are used, flags must be parsed first.
+flag.Parse()
+```
This interprets command-line arguments that follow the format:
@@ -192,6 +200,12 @@ public interface MyOptions extends PipelineOptions {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipeline_options_define_custom
%}
```
+```go
+var (
+ input = flag.String("input", "", "")
+ output = flag.String("output", "", "")
+)
+```
You can also specify a description, which appears when a user passes `--help` as
a command-line argument, and a default value.
@@ -210,6 +224,13 @@ public interface MyOptions extends PipelineOptions {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipeline_options_define_custom_with_help_and_default
%}
```
+```go
+var (
+ input = flag.String("input", "gs://my-bucket/input", "File(s) to read.")
+ output = flag.String("output", "gs://my-bucket/output", "Output file.")
+)
+```
+
{:.language-java}
It's recommended that you register your interface with `PipelineOptionsFactory`
@@ -288,6 +309,9 @@ public static void main(String[] args) {
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py tag:pipelines_constructing_reading
%}
```
+```go
+lines := textio.Read(s, "protocol://path/to/some/inputData.txt")
+```
See the [section on I/O](#pipeline-io) to learn more about how to read from the
various data sources supported by the Beam SDK.
@@ -590,6 +614,16 @@ words = ...
%}
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_pardo_apply
%}```
+```go
+// words is the input PCollection of strings
+var words beam.PCollection = ...
+
+func computeWordLengthFn(word string) int {
+ return len(word)
+}
+
+wordLengths := beam.ParDo(s, computeWordLengthFn, words)
+```
In the example, our input `PCollection` contains `String` values. We apply a
`ParDo` transform that specifies a function (`ComputeWordLengthFn`) to compute
@@ -704,7 +738,6 @@ PCollection<Integer> wordLengths = words.apply(
}
}));
```
-
```py
# The input PCollection of strings.
words = ...
@@ -713,6 +746,14 @@ words = ...
# Save the result as the PCollection word_lengths.
{% github_sample /apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py tag:model_pardo_using_flatmap
%}```
+```go
+// words is the input PCollection of strings
+var words beam.PCollection = ...
+
+lengths := beam.ParDo(s, func (word string) int {
+ return len(word)
+}, words)
+```
If your `ParDo` performs a one-to-one mapping of input elements to output
elements--that is, for each input element, it applies a function that produces
@@ -734,7 +775,6 @@ PCollection<Integer> wordLengths = words.apply(
MapElements.into(TypeDescriptors.integers())
.via((String word) -> word.length()));
```
-
```py
# The input PCollection of string.
words = ...