You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org> on 2026/07/22 17:55:01 UTC
[PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
SreeramaYeshwanthGowd opened a new pull request, #39442:
URL: https://github.com/apache/beam/pull/39442
### Problem
Java has `Sample.any(n)`, which returns up to n arbitrary elements from a
PCollection without the cost of a uniform random sample. The Python SDK has no
equivalent. Its only global option is `Sample.FixedSizeGlobally(n)`, which runs
a uniform reservoir sample and returns a single `list[T]`. This gap was reported
in #18552.
### Solution
Add `Sample.Any`, the Python equivalent of Java's `Sample.any`. It keeps up to n
arbitrary elements using a small `CombineFn` and returns them as a PCollection
of individual elements. It makes no uniformity guarantee and is cheaper than
`FixedSizeGlobally` because it skips the per element random sampling. If the
input has fewer than n elements, all of them are returned. This mirrors Java's
semantics and its element returning result type.
### Testing
Added unit tests in `combiners_test.py` that run on the DirectRunner and cover
the normal case, an input smaller than n, a windowed input, an empty input, and
n equal to zero. Also extended the existing display data test to cover the new
transform.
Fixes #18552
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [x] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org>.
SreeramaYeshwanthGowd commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5149995442
Thanks @bvolpato. Done, Sample.Any now rejects negative n with a ValueError matching Java's Sample.any, and added a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5050581831
Assigning reviewers:
R: @claudevdm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org>.
SreeramaYeshwanthGowd commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5138085487
Hi @claudevdm, following up when you have a chance to take a look. CI is green and the merge conflict has been resolved. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org>.
SreeramaYeshwanthGowd commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5230469437
@claudevdm The CI is green. Ready to get merged whenever you are ready. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org>.
SreeramaYeshwanthGowd commented on code in PR #39442:
URL: https://github.com/apache/beam/pull/39442#discussion_r3703547842
##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -597,6 +597,34 @@ def display_data(self):
def default_label(self):
return 'FixedSizePerKey(%d)' % self._n
+ @with_input_types(T)
+ @with_output_types(T)
+ class Any(ptransform.PTransform):
+ """Returns up to n arbitrary elements from the input PCollection.
+
+ This is the Python equivalent of Java's ``Sample.any``. Unlike
+ ``FixedSizeGlobally`` it does not sample uniformly at random, and it returns
+ the selected elements rather than a single list. If the input has fewer than
+ n elements, all of them are returned.
+ """
+ def __init__(self, n):
+ if n < 0:
+ raise ValueError('Expected non-negative n, received %s.' % n)
+ self._n = n
+
+ def expand(self, pcoll):
+ return (
+ pcoll
+ | core.CombineGlobally(_SampleAnyCombineFn(
+ self._n)).without_defaults()
+ | core.FlatMap(lambda elements: elements))
Review Comment:
Done, added with_input_types(list[T]) and with_output_types(T) to the FlatMap. Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "SreeramaYeshwanthGowd (via GitHub)" <gi...@apache.org>.
SreeramaYeshwanthGowd commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5165850859
Thanks for the review @claudevdm. I added the type hints you requested to the FlatMap call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5049578203
> [!CAUTION]
> The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm commented on code in PR #39442:
URL: https://github.com/apache/beam/pull/39442#discussion_r3703507039
##########
sdks/python/apache_beam/transforms/combiners.py:
##########
@@ -597,6 +597,34 @@ def display_data(self):
def default_label(self):
return 'FixedSizePerKey(%d)' % self._n
+ @with_input_types(T)
+ @with_output_types(T)
+ class Any(ptransform.PTransform):
+ """Returns up to n arbitrary elements from the input PCollection.
+
+ This is the Python equivalent of Java's ``Sample.any``. Unlike
+ ``FixedSizeGlobally`` it does not sample uniformly at random, and it returns
+ the selected elements rather than a single list. If the input has fewer than
+ n elements, all of them are returned.
+ """
+ def __init__(self, n):
+ if n < 0:
+ raise ValueError('Expected non-negative n, received %s.' % n)
+ self._n = n
+
+ def expand(self, pcoll):
+ return (
+ pcoll
+ | core.CombineGlobally(_SampleAnyCombineFn(
+ self._n)).without_defaults()
+ | core.FlatMap(lambda elements: elements))
Review Comment:
Can you please add an explicit .with_input_types(list[T]).with_output_types(T) to FlatMap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "claudevdm (via GitHub)" <gi...@apache.org>.
claudevdm merged PR #39442:
URL: https://github.com/apache/beam/pull/39442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Sample.Any to the Python SDK to match Java's Sample.any [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39442:
URL: https://github.com/apache/beam/pull/39442#issuecomment-5130967658
Reminder, please take a look at this pr: @claudevdm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org