You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/05/22 02:41:48 UTC
[GitHub] [beam] epicfaace opened a new pull request #11790: Programming guide - Fix typos and add clearer Python text
epicfaace opened a new pull request #11790:
URL: https://github.com/apache/beam/pull/11790
[draft - in progress]
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11790:
URL: https://github.com/apache/beam/pull/11790#issuecomment-645001983
thanks @epicfaace -I'll merge after tests pass
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #11790:
URL: https://github.com/apache/beam/pull/11790#issuecomment-634198356
retest this please
(note this is just a command to jenkins to run tests on your changes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11790:
URL: https://github.com/apache/beam/pull/11790#discussion_r441130698
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -1842,6 +1847,9 @@ transform's intermediate data changes type multiple times.
{{< github_sample "/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py" pipeline_monitoring_composite >}}
{{< /highlight >}}
+Note that because `Count` is itself a composite transform,
Review comment:
done
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -527,7 +530,7 @@ The graph of this pipeline looks like the following:
*Figure 1: A linear pipeline with three sequential transforms.*
However, note that a transform *does not consume or otherwise alter* the input
-collection--remember that a `PCollection` is immutable by definition. This means
+collection - remember that a `PCollection` is immutable by definition. This means
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11790:
URL: https://github.com/apache/beam/pull/11790
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11790:
URL: https://github.com/apache/beam/pull/11790#issuecomment-634198306
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11790:
URL: https://github.com/apache/beam/pull/11790#issuecomment-645016623
thanks @epicfaace @rosetn !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rosetn commented on a change in pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
rosetn commented on a change in pull request #11790:
URL: https://github.com/apache/beam/pull/11790#discussion_r433569849
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -527,7 +530,7 @@ The graph of this pipeline looks like the following:
*Figure 1: A linear pipeline with three sequential transforms.*
However, note that a transform *does not consume or otherwise alter* the input
-collection--remember that a `PCollection` is immutable by definition. This means
+collection - remember that a `PCollection` is immutable by definition. This means
Review comment:
Em dash with no spaces between words is the correct usage here: —
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -1842,6 +1847,9 @@ transform's intermediate data changes type multiple times.
{{< github_sample "/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py" pipeline_monitoring_composite >}}
{{< /highlight >}}
+Note that because `Count` is itself a composite transform,
Review comment:
I'd either format this as a note (see other notes on this page) or remove "Note that"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] epicfaace commented on a change in pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
epicfaace commented on a change in pull request #11790:
URL: https://github.com/apache/beam/pull/11790#discussion_r430653038
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -683,18 +687,18 @@ that for you. Your `@ProcessElement` method should accept a parameter tagged wit
`@Element`, which will be populated with the input element. In order to output
elements, the method can also take a parameter of type `OutputReceiver` which
provides a method for emitting elements. The parameter types must match the input
-and output types of your `DoFn` or the framework will raise an error. Note: @Element and
-OutputReceiver were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-ProcessContext parameter should be used instead.
+and output types of your `DoFn` or the framework will raise an error. Note: `@Element` and
+`OutputReceiver` were introduced in Beam 2.5.0; if using an earlier release of Beam, a
+`ProcessContext` parameter should be used instead.
{{< /paragraph >}}
{{< paragraph class="language-py" >}}
Inside your `DoFn` subclass, you'll write a method `process` where you provide
the actual processing logic. You don't need to manually extract the elements
from the input collection; the Beam SDKs handle that for you. Your `process`
-method should accept an object of type `element`. This is the input element and
-output is emitted by using `yield` or `return` statement inside `process`
-method.
+method should accept an argument `element`, which is the input element, emit
+output value(s) by using `yield` statements. You can also use a `return`
+statement if you are sure the method will return only a single output value.
Review comment:
That sounds good!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11790: [BEAM-9926] Programming guide - Fix typos and add clearer Python text
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11790:
URL: https://github.com/apache/beam/pull/11790#discussion_r430617430
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -683,18 +687,18 @@ that for you. Your `@ProcessElement` method should accept a parameter tagged wit
`@Element`, which will be populated with the input element. In order to output
elements, the method can also take a parameter of type `OutputReceiver` which
provides a method for emitting elements. The parameter types must match the input
-and output types of your `DoFn` or the framework will raise an error. Note: @Element and
-OutputReceiver were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-ProcessContext parameter should be used instead.
+and output types of your `DoFn` or the framework will raise an error. Note: `@Element` and
+`OutputReceiver` were introduced in Beam 2.5.0; if using an earlier release of Beam, a
+`ProcessContext` parameter should be used instead.
{{< /paragraph >}}
{{< paragraph class="language-py" >}}
Inside your `DoFn` subclass, you'll write a method `process` where you provide
the actual processing logic. You don't need to manually extract the elements
from the input collection; the Beam SDKs handle that for you. Your `process`
-method should accept an object of type `element`. This is the input element and
-output is emitted by using `yield` or `return` statement inside `process`
-method.
+method should accept an argument `element`, which is the input element, emit
+output value(s) by using `yield` statements. You can also use a `return`
+statement if you are sure the method will return only a single output value.
Review comment:
The reason that `yield` is used is that `process` should return an _iterable_. `yield` is the way to pass individual elements from an iterable (does that make sense?)
Return statements can be used, but they must return an iterable - for example a list. So, for example, these functions are equivalent:
```
def process(self, elm):
yield elm
```
and
```
def process(self, elm):
return [elm]
```
##########
File path: website/www/site/content/en/documentation/programming-guide.md
##########
@@ -683,18 +687,18 @@ that for you. Your `@ProcessElement` method should accept a parameter tagged wit
`@Element`, which will be populated with the input element. In order to output
elements, the method can also take a parameter of type `OutputReceiver` which
provides a method for emitting elements. The parameter types must match the input
-and output types of your `DoFn` or the framework will raise an error. Note: @Element and
-OutputReceiver were introduced in Beam 2.5.0; if using an earlier release of Beam, a
-ProcessContext parameter should be used instead.
+and output types of your `DoFn` or the framework will raise an error. Note: `@Element` and
+`OutputReceiver` were introduced in Beam 2.5.0; if using an earlier release of Beam, a
+`ProcessContext` parameter should be used instead.
{{< /paragraph >}}
{{< paragraph class="language-py" >}}
Inside your `DoFn` subclass, you'll write a method `process` where you provide
the actual processing logic. You don't need to manually extract the elements
from the input collection; the Beam SDKs handle that for you. Your `process`
-method should accept an object of type `element`. This is the input element and
-output is emitted by using `yield` or `return` statement inside `process`
-method.
+method should accept an argument `element`, which is the input element, emit
+output value(s) by using `yield` statements. You can also use a `return`
+statement if you are sure the method will return only a single output value.
Review comment:
So, perhaps I would write something like...
```
Your `process` method should accept an argument `element`, which is the input element,
and return an iterable with its output values. You can accomplish this by emitting individual
elements with `yield` statements. You can also use a `return` statement with an iterable,
like a list or a generator.
```
I am not married to the phrasing - whichever phrasing you choose is fine. But I think it's important to mention the fact that an iterable is expected.
Thanks @epicfaace
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org