You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/24 17:19:55 UTC
[GitHub] [beam] damondouglas opened a new pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
damondouglas opened a new pull request #11803:
URL: https://github.com/apache/beam/pull/11803
This pull request adds a CoGroupByKey lesson to the Go SDK katas. I would like to request the following reviewers:
(R: @lostluck )
(R: @henryken )
If accepted, please wait until the [Stepik course](https://stepik.org/course/70387) is updated before finally merging this PR.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [*] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [*] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a change in pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
damondouglas commented on a change in pull request #11803:
URL: https://github.com/apache/beam/pull/11803#discussion_r430474979
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
Review comment:
I agree with you and realizing for code readability purposes it makes sense. I created [BEAM-10091](https://issues.apache.org/jira/browse/BEAM-10091) when we are all complete with the series of Go SDK katas. I'd like to perform an overall cleanup for naming consistency and code readability.
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
Review comment:
I agree with you and realizing for code readability purposes it makes sense. I created [BEAM-10091](https://issues.apache.org/jira/browse/BEAM-10091) when we are all complete with the series of Go SDK katas. I'd like to perform an overall cleanup for naming consistency and code readability.
For now, I will adjust the current CoGroupByKey lesson task for code readability. Thank you, Henry.
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
I couldn't find any and solved this through analyzing the error output. @lostluck if/whenever you have the chance, could you let us know if this is correct?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
Yes I believe this is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #11803:
URL: https://github.com/apache/beam/pull/11803
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] henryken commented on a change in pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
henryken commented on a change in pull request #11803:
URL: https://github.com/apache/beam/pull/11803#discussion_r431167018
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
Review comment:
Thanks for creating the cleanup task!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11803:
URL: https://github.com/apache/beam/pull/11803#discussion_r430512377
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
Review comment:
+1 to using unambiguous names here.
In this case it's especially important to disambiguate between the iterator parameters (currently f and c) and the emiter parameter since they're both function parameters.
eg, using iter1, iter2 would be better than unrelated single character names. Also +1 to doing a global cleanup instead of immediately back updating everything previously. That said, there's no harm in starting in *this* PR.
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
[GroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam?utm_source=backtogodoc#GroupByKey) by itself does have a single example attached to it, but [CoGBK](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam?utm_source=backtogodoc#CoGroupByKey) does not, nor does it explain it's relationship to GBK (it's a generalization of GBK that groups multiple PCollection KVs into a CoGBK<K, V1, V2..> )
The go docs definitely be improved in this regard.
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
I did write a better guide for the Go SDK last year, but never go around to exporting it from Google internal. It's on my list of things to do before declaring the SDK is no longer experimental.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] henryken commented on a change in pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
henryken commented on a change in pull request #11803:
URL: https://github.com/apache/beam/pull/11803#discussion_r429965845
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
+ v := &WordsAlphabet{
Review comment:
Why not using `wa` here so that not to confuse with Value?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
If I understand correctly, it seems that `func(*string) bool` should assign the value of V to the passed-in variable pointer, not the KV?
Borrowing the concept of the Java version, these functions behave similarly to CoGbkResult.get() based on the sequence of the passed in PCollection?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/go.mod
##########
@@ -0,0 +1,26 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+module cogroupbykey
+
+go 1.14
Review comment:
Just realized that the GroupByKey and CoGroupByKey are not using the same Go version as the others. Can we make all of them uniform?
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
+ v := &WordsAlphabet{
+ Alphabet: key,
+ }
+ f(&v.Fruit)
+ c(&v.Country)
+ emit(v.String())
+ }, grouped)
+}
+
+type WordsAlphabet struct {
+ Alphabet string
+ Fruit string
+ Country string
+}
+
+func (wa *WordsAlphabet) String() string {
+ return fmt.Sprintf("WordsAlphabet%+v", *wa)
Review comment:
Nice trick!
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package task
+
+import (
+ "fmt"
+ "github.com/apache/beam/sdks/go/pkg/beam"
+)
+
+func ApplyTransform(s beam.Scope, fruits beam.PCollection, countries beam.PCollection) beam.PCollection {
+ fruitsKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, fruits)
+
+ countriesKV := beam.ParDo(s, func(e string) (string, string) {
+ return string(e[0]), e
+ }, countries)
+
+ grouped := beam.CoGroupByKey(s, fruitsKV, countriesKV)
+ return beam.ParDo(s, func(key string, f func(*string) bool, c func(*string) bool, emit func(string)) {
Review comment:
I understand the convention of Go that prefers shorter variable name, but I personally find that `f` and `c` are cryptic.
But if it's well understood, I guess I'm just too old for this :)
Btw, I like the variable naming in the examples at the top of the godoc.
https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam
##########
File path: learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
##########
@@ -0,0 +1,104 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# CoGroupByKey
+
+CoGroupByKey performs a relational join of two or more key/value PCollections that have the same
+key type.
+
+**Kata:** Implement a [beam.CoGroupByKey](https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey)
+transform that join words by the first alphabetical letter, and then produces the string representation of the
+WordsAlphabet model.
+
+<div class="hint">
+ Refer to
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ to solve this problem.
+</div>
+
+<div class="hint">
+ Refer to the Beam Programming Guide
+ <a href="https://beam.apache.org/documentation/programming-guide/#cogroupbykey">
+ "CoGroupByKey"</a> section for more information.
+</div>
+
+<div class="hint">
+ Think of this problem in three stages. First, create key/value pairs of PCollections called KV
+ for fruits and countries, pairing the first character with the word. Next, apply CoGroupByKey to the KVs
+ followed by a ParDo.
+</div>
+
+<div class="hint">
+ In the last lesson we learned how to make key/value PCollections called KV. Now we have
+ two to make from fruits and countries.
+
+ To return as a KV, you can return two values from your DoFn. The first return value represents the Key, and
+ the second return value represents the Value. An example is shown below.
+
+```
+func doFn(element string) (string, string) {
+ key := string(element[0])
+ value := element
+ return key, value
+}
+```
+</div>
+
+<div class="hint">
+ In the last lesson we learned that
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#GroupByKey">
+ beam.GroupByKey</a> takes a single KV.
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#CoGroupByKey">beam.CoGroupByKey</a>
+ takes more than one KV.
+</div>
+
+<div class="hint">
+ Our final step in this problem requires a
+ <a href="https://godoc.org/github.com/apache/beam/sdks/go/pkg/beam#ParDo">beam.ParDo</a>
+ with a DoFn that's different than what we've seen in previous lessons. In the previous step we should
+ have a PCollection acquired from CoGroupByKey. A ParDo for that PCollection expects a DoFn that looks
+ like the following.
+
+ ```
+ func doFn(key string, aKV func(*string) bool, anotherKV func(*string) bool, emit func(string)){
Review comment:
Is there any documentation available that explains about this DoFn variation?
I still couldn't find a good programming guide for the Go version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] henryken commented on pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
henryken commented on pull request #11803:
URL: https://github.com/apache/beam/pull/11803#issuecomment-635811715
Thanks @damondouglas! This PR can be merged now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] henryken commented on pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
henryken commented on pull request #11803:
URL: https://github.com/apache/beam/pull/11803#issuecomment-634696399
LGTM now. Thanks for the changes @damondouglas.
For updating to Stepik, can you wait to do it together with [PR #11806](https://github.com/apache/beam/pull/11806)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on pull request #11803: [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section
Posted by GitBox <gi...@apache.org>.
damondouglas commented on pull request #11803:
URL: https://github.com/apache/beam/pull/11803#issuecomment-635644834
I updated [the stepik course](https://stepik.org/course/70387) and commited the updated `*-remote.yaml` files to this PR. It is ready to merge into master. Thank you, @henryken and @lostluck for your help.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org