You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "YashaswiniTB (via GitHub)" <gi...@apache.org> on 2025/05/11 18:18:07 UTC
[PR] Add documentation for return vs yield usage in DoFn.process() [beam]
YashaswiniTB opened a new pull request, #34912:
URL: https://github.com/apache/beam/pull/34912
Issue Link : https://github.com/apache/beam/issues/34906
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "YashaswiniTB (via GitHub)" <gi...@apache.org>.
YashaswiniTB commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-2891438805
@damccorm kindly review the PR and approve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-2997413093
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2330244749
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1183,6 +1183,100 @@ func init() {
> parameters to a single `emitter function`.
</span>
+{{< paragraph class="language-python">}}
Review Comment:
```suggestion
{{< paragraph class="language-python">}}
```
Nit
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1183,6 +1183,100 @@ func init() {
> parameters to a single `emitter function`.
</span>
+{{< paragraph class="language-python">}}
+Proper use of return vs yield in Python Functions.
+{{< /paragraph >}}
+
+<span class="language-python">
+
+> **Returning a single element (e.g., `return element`) is incorrect**
+> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string
+> (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`) or incorrect results since the return value
+> will be treated as an iterable. Always ensure your return type is iterable.
+
+</span>
+
+{{< highlight python >}}
+# Incorrectly Returning a single string instead of a sequence
+class ReturnIndividualElement(beam.DoFn):
+ def process(self, element):
+ return element
+
+with beam.Pipeline() as pipeline:
+ (
+ pipeline
+ | "CreateExamples" >> beam.Create(["foo"])
+ | "MapIncorrect" >> beam.ParDo(ReturnIndividualElement())
+ | "Print" >> beam.Map(print)
+ )
+ # prints:
+ # f
+ # o
+ # o
+{{< /highlight >}}
+
+<span class="language-python">
+
+> **Returning a list (e.g., `return [element1, element2]`) is valid because List is Iterable**
+> This approach works well when emitting multiple outputs from a single call and is easy to read for small datasets.
+
+</span>
+
+{{< highlight python >}}
+# Returning a list of strings
+class ReturnWordsFn(beam.DoFn):
+ def process(self, element):
+ # Split the sentence and return all words longer than 2 characters as a list
+ return [word for word in element.split() if len(word) > 2]
+
+with beam.Pipeline() as pipeline:
+ (
+ pipeline
+ | "CreateSentences_Return" >> beam.Create([ # Create a collection of sentences
+ "Apache Beam is powerful", # Sentence 1
+ "Try it now" # Sentence 2
+ ])
+ | "SplitWithReturn" >> beam.ParDo(ReturnWordsFn()) # Apply the custom DoFn to split words
+ | "PrintWords_Return" >> beam.Map(print) # Print each List of words
+ )
+ # prints:
+ # ['Apache', 'Beam', 'powerful']
+ # ['Try', 'now']
Review Comment:
```suggestion
# Apache
# Beam
# powerful
# Try
# now
```
When you return an iterable, each element in the iterable is part of the pcollection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2162326649
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,101 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+{{< /paragraph >}}
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
+class ReturnIndividualElement(beam.DoFn):
+ def process(self, element):
+ return element
+
+with beam.Pipeline() as pipeline:
+ (
+ pipeline
+ | "CreateExamples" >> beam.Create(["foo"])
+ | "MapIncorrect" >> beam.ParDo(ReturnIndividualElement())
+ | "Print" >> beam.Map(print)
+ )
+ # prints:
+ # f
+ # o
+ # o
+{{< /highlight >}}
+
+<span class="language-python">
+
+> **Returning a single element (e.g., `return element`) is incorrect**
Review Comment:
It does not look like you addressed this - please move the "Returning a single element" section to the back of the order
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2084679998
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,45 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+In Python, functions can return results either all at once (return) or lazily one at a time (yield). The right choice depends on your use case, especially when dealing with large datasets or streaming scenarios.
Review Comment:
We don't need to include information on how Python generally handles return vs yield, we should specifically talk about how Beam handles them
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,45 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+In Python, functions can return results either all at once (return) or lazily one at a time (yield). The right choice depends on your use case, especially when dealing with large datasets or streaming scenarios.
+{{< /paragraph >}}
+
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
+def get_lines_wrong():
+ return "Line 1" # Not iterable in the way most expect
+
+# Returning a list of strings
+def get_lines_as_list():
+ return ["Line 1", "Line 2", "Line 3"] # Eager, but valid
+
+# Yielding each line one at a time
+def get_lines_generator():
+ with open("data.txt") as f:
+ for line in f:
+ yield line.strip() # Lazy and memory-efficient
+
+{{< /highlight >}}
+
+
+<span class="language-python">
+
+> **Note:**
+>
+> - **Returning a single element (e.g., `return element`) is incorrect**
+> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`). Always ensure your return type is iterable.
Review Comment:
```suggestion
> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`) or incorrect results since the return value will be treated as an iterable. Always ensure your return type is iterable.
```
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,45 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+In Python, functions can return results either all at once (return) or lazily one at a time (yield). The right choice depends on your use case, especially when dealing with large datasets or streaming scenarios.
+{{< /paragraph >}}
+
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
+def get_lines_wrong():
+ return "Line 1" # Not iterable in the way most expect
+
+# Returning a list of strings
+def get_lines_as_list():
+ return ["Line 1", "Line 2", "Line 3"] # Eager, but valid
+
+# Yielding each line one at a time
+def get_lines_generator():
+ with open("data.txt") as f:
+ for line in f:
+ yield line.strip() # Lazy and memory-efficient
+
+{{< /highlight >}}
+
+
+<span class="language-python">
+
+> **Note:**
+>
+> - **Returning a single element (e.g., `return element`) is incorrect**
+> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`). Always ensure your return type is iterable.
+>
+> - **Returning a list (e.g., `return [element1, element2]`) is valid but eager**
+> This method is syntactically correct and works for small numbers of outputs. However, it builds the entire list in memory before returning it, which can increase memory consumption and impact performance for large data sets.
+>
+> - **Using `yield` (e.g., `yield element`) is preferred for scalability**
+> Using `yield` turns the method into a generator function. This enables lazy evaluation, where each element is processed and emitted one at a time. It’s more memory-efficient and better suited for large pipelines or streaming workloads.
Review Comment:
We can get rid of the eager vs not eager piece. In the context of Beam it won't necessarily matter since the data may be materialized by Beam anyways
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,45 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+In Python, functions can return results either all at once (return) or lazily one at a time (yield). The right choice depends on your use case, especially when dealing with large datasets or streaming scenarios.
+{{< /paragraph >}}
+
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
+def get_lines_wrong():
+ return "Line 1" # Not iterable in the way most expect
+
+# Returning a list of strings
+def get_lines_as_list():
+ return ["Line 1", "Line 2", "Line 3"] # Eager, but valid
+
+# Yielding each line one at a time
+def get_lines_generator():
+ with open("data.txt") as f:
+ for line in f:
+ yield line.strip() # Lazy and memory-efficient
+
+{{< /highlight >}}
+
+
+<span class="language-python">
+
+> **Note:**
+>
+> - **Returning a single element (e.g., `return element`) is incorrect**
+> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`). Always ensure your return type is iterable.
Review Comment:
Could we add examples for each of these? For example, something like:
```
# Incorrect return of element instead of iterable
class IdentityFunction(beam.DoFn):
def process(self, element):
return element
with beam.Pipeline() as pipeline:
examples = (
pipeline
| "CreateExamples" >> beam.Create(["foo"])
| "Map" >> beam.ParDo(ReturnIndividualElement())
| "Print" >> beam.ParDo(FormatOutput())
)
# prints:
# f
# o
# o
```
and then similar examples with correct return types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2162327101
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1183,6 +1183,100 @@ func init() {
> parameters to a single `emitter function`.
</span>
+{{< paragraph class="language-python">}}
+Proper use of return vs yield in Python Functions.
+{{< /paragraph >}}
+
+<span class="language-python">
+
+> **Returning a single element (e.g., `return element`) is incorrect**
+> The `process` method in Beam must return an *iterable* of elements. Returning a single value like an integer or string
+> (e.g., `return element`) leads to a runtime error (`TypeError: 'int' object is not iterable`) or incorrect results since the return value
+> will be treated as an iterable. Always ensure your return type is iterable.
+
+</span>
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
Review Comment:
```suggestion
# Incorrectly returning a single string instead of a sequence
```
Maybe helpful to add this too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #34912:
URL: https://github.com/apache/beam/pull/34912
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2086853847
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,91 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+{{< /paragraph >}}
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
Review Comment:
Rather than having a list of examples at the top, could we interleave these examples with the descriptions below?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-2870064324
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #34912:
URL: https://github.com/apache/beam/pull/34912#discussion_r2105241495
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,101 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
+{{< /paragraph >}}
+
+{{< highlight python >}}
+# Returning a single string instead of a sequence
+class ReturnIndividualElement(beam.DoFn):
+ def process(self, element):
+ return element
+
+with beam.Pipeline() as pipeline:
+ (
+ pipeline
+ | "CreateExamples" >> beam.Create(["foo"])
+ | "MapIncorrect" >> beam.ParDo(ReturnIndividualElement())
+ | "Print" >> beam.Map(print)
+ )
+ # prints:
+ # f
+ # o
+ # o
+{{< /highlight >}}
+
+<span class="language-python">
+
+> **Returning a single element (e.g., `return element`) is incorrect**
Review Comment:
Could you please flip the order of the descriptions and the examples? Otherwise, I worry people will just read the example and follow it without seeing the description saying it is wrong
##########
website/www/site/content/en/documentation/programming-guide.md:
##########
@@ -1184,6 +1184,101 @@ func init() {
</span>
+{{< paragraph class="language-python">}}
+Proper Use of return vs yield in Python Functions.
Review Comment:
```suggestion
Proper use of return vs yield in Python Functions.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-2994340754
Assigning reviewers:
R: @damccorm for label website.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "derrickaw (via GitHub)" <gi...@apache.org>.
derrickaw commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-3179563822
waiting on author
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Added documentation for return vs yield usage in DoFn.process() [beam]
Posted by "YashaswiniTB (via GitHub)" <gi...@apache.org>.
YashaswiniTB commented on PR #34912:
URL: https://github.com/apache/beam/pull/34912#issuecomment-2997502090
> waiting on author
Thanks @damccorm! I believe I’ve addressed all the comments. Please let me know if anything else is required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org