You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mtauha (via GitHub)" <gi...@apache.org> on 2026/02/22 16:01:18 UTC
[PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
mtauha opened a new pull request, #37679:
URL: https://github.com/apache/beam/pull/37679
## Description
This PR adds unbounded streaming support to the native Python `GenerateSequence` PTransform, building on top of the bounded implementation introduced in #37675.
Addresses #18088
### Motivation
Phase 1 (#37675) introduced the bounded `GenerateSequence`. This PR completes
Phase 2 by adding unbounded support — allowing Python pipelines to generate infinite integer sequences with optional rate limiting and custom timestamps, equivalent to Java's `UnboundedCountingSource`.
The existing external Flink-only version at `apache_beam/io/external/generate_sequence.py` remains untouched.
### Changes
- Updated `sdks/python/apache_beam/io/generate_sequence.py`:
- `GenerateSequence` now supports unbounded mode (when `stop` is not provided)
- Added `_UnboundedCountingSource` backed by `UnboundedSource`
- Added rate limiting support via `elements_per_second` parameter
- Added custom timestamp support via `timestamp_fn` parameter
- Added checkpointing via `_CounterMark` (equivalent to Java's `CounterMark`)
- Updated `sdks/python/apache_beam/io/generate_sequence_test.py` with tests
covering unbounded mode, rate limiting, and checkpointing behavior
### Notes
- This is **Phase 2 (unbounded)**. The bounded implementation from Phase 1 remains fully intact and untouched.
- Works on **all runners** without a Java expansion service.
- Modelled after Java's `UnboundedCountingSource` and `CounterMark`.
### Testing
```bash
cd sdks/python
python -m pytest apache_beam/io/generate_sequence_test.py -v
```
## Checklist
- [X] Addresses #18088
- [X] Updated `CHANGES.md` with noteworthy changes
- [X] All existing and new tests pass locally
- [X] No pylint errors (`pylint apache_beam/io/generate_sequence.py`)
- [X] Formatted with yapf (`yapf --in-place --parallel --recursive apache_beam/io/generate_sequence.py`)
- [X] Apache license header present in all new/modified files
R: @shunping @damccorm @robertwb @claudevdm @kennknowles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed pull request #37679: [python][io] Add native unbounded GenerateSequence PTransform (#18088)
URL: https://github.com/apache/beam/pull/37679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #37679:
URL: https://github.com/apache/beam/pull/37679#issuecomment-4155965461
Closing per Yi's comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #37679:
URL: https://github.com/apache/beam/pull/37679#issuecomment-3947461727
We already have many counting sources served as similar purposes. I would suggest improving existing Python PeriodicImpulse, for example, porting Java's stopAfter and catchUpToNow configs into Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37679:
URL: https://github.com/apache/beam/pull/37679#issuecomment-3941548751
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][io] Add native unbounded GenerateSequence PTransform (#18088) [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37679:
URL: https://github.com/apache/beam/pull/37679#issuecomment-3941269381
## Summary of Changes
Hello @mtauha, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the native Python `GenerateSequence` PTransform by introducing full unbounded streaming capabilities. Building upon the existing bounded implementation, it now allows Python pipelines to generate infinite sequences of integers, complete with configurable rate limiting and custom timestamp assignment. This brings the Python `GenerateSequence` to parity with its Java SDK counterpart, `UnboundedCountingSource`, and ensures compatibility across all runners without requiring an external Java expansion service.
### Highlights
* **Unbounded GenerateSequence**: Added native Python support for the `GenerateSequence` PTransform to generate infinite integer sequences, completing Phase 2 of its implementation.
* **Rate Limiting**: Implemented optional rate limiting for unbounded sequences using `elements_per_period` and `period` parameters.
* **Custom Timestamps**: Enabled custom timestamp assignment for generated elements in the unbounded mode.
* **Checkpointing**: Introduced `_CounterMark` for checkpointing in unbounded mode, mirroring Java's `UnboundedCountingSource`.
* **Comprehensive Testing**: Included new unit tests covering unbounded mode, rate limiting, and checkpointing behavior for the `GenerateSequence` PTransform.
🧠 **New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Changelog</b></summary>
* **CHANGES.md**
* Added native Python GenerateSequence transform for generating bounded and unbounded sequences of integers, with optional rate limiting support.
* **sdks/python/apache_beam/io/generate_sequence.py**
* Implemented the `GenerateSequence` PTransform to support both bounded and unbounded integer sequence generation.
* Introduced `_UnboundedCountingDoFn` and `_UnboundedCountingRestrictionProvider` for unbounded mode using the Splittable DoFn (SDF) framework.
* Added logic for rate limiting and timestamp generation within the unbounded source.
* Included `_BoundedCountingSource` for the bounded mode.
* **sdks/python/apache_beam/io/generate_sequence_test.py**
* Added extensive unit tests for `GenerateSequence`, covering bounded and unbounded modes, various parameter validations, `_BoundedCountingSource` behavior (estimate_size, split, read, display_data), and `_UnboundedCountingRestrictionProvider` logic (initial_restriction, restriction_size).
</details>
<details>
<summary><b>Activity</b></summary>
* No human activity (comments, reviews, or progress updates) has been recorded for this pull request yet.
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org