You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/02 23:13:10 UTC
[GitHub] [beam] damondouglas opened a new pull request #12448: [BEAM-9679] Add Additional Parameters lesson to Go SDK Katas
damondouglas opened a new pull request #12448:
URL: https://github.com/apache/beam/pull/12448
This pull request adds an additional parameters lesson to the Go SDK katas. I would like to request the following reviewer:
(R: @lostluck )
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a change in pull request #12448: [BEAM-9679] Add Additional Parameters lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
damondouglas commented on a change in pull request #12448:
URL: https://github.com/apache/beam/pull/12448#discussion_r466143420
##########
File path: learning/katas/go/core_transforms/windowing/additional_parameters/task.md
##########
@@ -0,0 +1,77 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Windowing
+
+This lesson introduces the concept of windowed PCollection elements. A window is a view into a fixed beginning and
+fixed end to a set of data. In the beam model, windowing subdivides a PCollection according to the
+timestamps of its individual elements.
+
+Beam can pass information about the window and timestamp to your elements in your DoFn. All your previous
+lessons' DoFn had this information available, yet you never made use of it in your DoFn parameters. In this
+lesson you will. The simple toy dataset has five git commit messages and their timestamps
Review comment:
I like the idea of more windowing lessons after the rest of the originally planned katas are complete.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on a change in pull request #12448: [BEAM-9679] Add Additional Parameters lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
damondouglas commented on a change in pull request #12448:
URL: https://github.com/apache/beam/pull/12448#discussion_r464765533
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
+is not unique to unbounded data. The beam model windows all data whether it is bounded or unbounded.
+Yet, when you read from a fixed size source such as a file, beam applies the same timestamp to all the elements.
Review comment:
My gut was telling me "beam applies the timestamp" was not quite it but now the way you describe it makes sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #12448: [BEAM-9679] Add a windowed PCollection lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #12448:
URL: https://github.com/apache/beam/pull/12448
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12448: [BEAM-9679] Add Additional Parameters lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12448:
URL: https://github.com/apache/beam/pull/12448#discussion_r464583093
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
Review comment:
Additional Parameters is a weird name for this section. It's simply how this is implemented in the Go SDK. There are also other additional parameters that aren't being covered which may get confusing as we add them to the SDK (Pane, Timers, State...)
Windowing or windows and timestamps stands fairly well on it's own.
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
+is not unique to unbounded data. The beam model windows all data whether it is bounded or unbounded.
+Yet, when you read from a fixed size source such as a file, beam applies the same timestamp to all the elements.
Review comment:
Beam doesn't specify timestamps. It's transform or runner dependent. If the framework receives timestamps, it propagates them or updates them as the transforms require.
eg. "The reading transform applies a timestamp...." not "beam applies the timestamp"
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
+is not unique to unbounded data. The beam model windows all data whether it is bounded or unbounded.
+Yet, when you read from a fixed size source such as a file, beam applies the same timestamp to all the elements.
+
+Beam will include information about the window and timestamp to your elements in your DoFn. All your previous
+lessons' DoFn had this information provided, yet you never made use of it in your DoFn parameters. In this
+lesson you will. The simple toy dataset has five git commit messages and their timestamps
+from the [Apache Beam public repository](https://github.com/apache/beam). Their timestamps have been
+applied to the PCollection input to simulate an unbounded dataset.
Review comment:
Probably repeating myself now, but bounded datasets can have timestamps as well.
Speaking outside of the context of this lesson:
Consider you have a stream of data from pubsub or something. Each element has the publishing time associated with it. However, data can be late*, which means you might emit less than accurate results if you want to maintain your ~1 minute averages or similar. To have the daily graphs be correct after the fact, you could preserve the incoming datastream somewhere, timestamps and all in some files. Then after the fact you could run the same pipeline against those files, to get the correct running averages throughout the day instead, just by replacing the streaming source transform, with the batch source transform, along with the respective sinks. Fun eh?
*which you can configure beam to handle, but that's not implemented in the Go SDK yet.
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
+is not unique to unbounded data. The beam model windows all data whether it is bounded or unbounded.
+Yet, when you read from a fixed size source such as a file, beam applies the same timestamp to all the elements.
+
+Beam will include information about the window and timestamp to your elements in your DoFn. All your previous
+lessons' DoFn had this information provided, yet you never made use of it in your DoFn parameters. In this
Review comment:
I'd say "available" rather than "provided".
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
Review comment:
Bounded vs Unbounded is orthogonal to windowing/event times. There's no need to understand it to understand the other. Windowing is useful and available to both kinds of PCollection. I'd recommend not mentioning it at all at this juncture.
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
Review comment:
WRT elements, size refers to how many bytes it takes up. You probably mean counts.
Windowing doesn't set things to fixed sizes or element sizes, or even counts.
WRT to bounded/unbounded, note that the text is saying "It's true for A!" "it's also true for not A!" It's true for both A and not A!"
So my recommendation is to not mention A at all.
##########
File path: learning/katas/go/core_transforms/additional_parameters/additional_parameters/task.md
##########
@@ -0,0 +1,84 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Additional Parameters - Window and Timestamp
+
+This lesson introduces the concept of windowing and timestamped PCollection elements.
+Before discussing windowing, we need to distinguish bounded from unbounded data.
+Bounded data is of a fixed size such as a file or database query. Unbounded data comes
+from a continuously updated source such as a subscription or stream.
+
+A window is a view into a fixed beginning and fixed end to a set of data. In the beam model, windowing subdivides
+a PCollection according to the timestamps of its individual elements. This is useful
+for unbounded data because it allows the model to work with fixed element sizes. Note that windowing
+is not unique to unbounded data. The beam model windows all data whether it is bounded or unbounded.
+Yet, when you read from a fixed size source such as a file, beam applies the same timestamp to all the elements.
+
+Beam will include information about the window and timestamp to your elements in your DoFn. All your previous
Review comment:
Beam can pass information about....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12448: [BEAM-9679] Add Additional Parameters lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12448:
URL: https://github.com/apache/beam/pull/12448#discussion_r466074785
##########
File path: learning/katas/go/core_transforms/windowing/additional_parameters/task.md
##########
@@ -0,0 +1,77 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Windowing
+
+This lesson introduces the concept of windowed PCollection elements. A window is a view into a fixed beginning and
+fixed end to a set of data. In the beam model, windowing subdivides a PCollection according to the
+timestamps of its individual elements.
+
+Beam can pass information about the window and timestamp to your elements in your DoFn. All your previous
+lessons' DoFn had this information available, yet you never made use of it in your DoFn parameters. In this
+lesson you will. The simple toy dataset has five git commit messages and their timestamps
Review comment:
Beam's not passing information to the elements, it's passing it to the dofn.
Consider...
`A DoFn can request timestamp and windowing information about the element it's processing. All your previous lessons had this information available as well. In this lesson you will make use of these parameters.`
It's probably a personal nit of mine, but the "yet you" sounds accusatory (not saying that was your intent).
##########
File path: learning/katas/go/core_transforms/windowing/additional_parameters/task.md
##########
@@ -0,0 +1,77 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Windowing
+
+This lesson introduces the concept of windowed PCollection elements. A window is a view into a fixed beginning and
+fixed end to a set of data. In the beam model, windowing subdivides a PCollection according to the
+timestamps of its individual elements.
Review comment:
I don't know how precise we need to be in the kata's, but consider adding.
"An element can be a part of one or more windows."
##########
File path: learning/katas/go/core_transforms/windowing/additional_parameters/task.md
##########
@@ -0,0 +1,77 @@
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+# Windowing
+
+This lesson introduces the concept of windowed PCollection elements. A window is a view into a fixed beginning and
+fixed end to a set of data. In the beam model, windowing subdivides a PCollection according to the
+timestamps of its individual elements.
+
+Beam can pass information about the window and timestamp to your elements in your DoFn. All your previous
+lessons' DoFn had this information available, yet you never made use of it in your DoFn parameters. In this
+lesson you will. The simple toy dataset has five git commit messages and their timestamps
Review comment:
I'd remove either `simple` or `toy`, they're redundant together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damondouglas commented on pull request #12448: [BEAM-9679] Add a windowed PCollection lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
damondouglas commented on pull request #12448:
URL: https://github.com/apache/beam/pull/12448#issuecomment-670084653
@lostluck the [stepik course](https://stepik.org/course/70387) has been updated and the `*-remote.yaml` files committed to this PR. This PR is ready to merge. Thank you so much for patiently walking with me through this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #12448: [BEAM-9679] Add a windowed PCollection lesson to Go SDK Katas
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #12448:
URL: https://github.com/apache/beam/pull/12448#issuecomment-670089621
No worries! You're getting to the point where we need more features in the SDK before more Katas can be written :D
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org