You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "stankiewicz (via GitHub)" <gi...@apache.org> on 2026/03/11 13:09:17 UTC
[PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
stankiewicz opened a new pull request, #37825:
URL: https://github.com/apache/beam/pull/37825
1. exposes
```
@ProcessElement
public void process(ProcessContext c, CausedByDrain causedByDrain) {}
@OnTimer(timerId)
public void onTimer(CausedByDrain causedByDrain) {}
```
2. added tests
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #37825:
URL: https://github.com/apache/beam/pull/37825#discussion_r2919129529
##########
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java:
##########
@@ -555,6 +555,11 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
return timestamp();
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return elem.causedByDrain();
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```java
/**
* Provide a reference to the caused by drain.
*/
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return elem.causedByDrain();
}
```
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java:
##########
@@ -514,6 +524,11 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
return delegate.timestamp(doFn);
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return delegate.causedByDrain(doFn);
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```suggestion
/** Provide a reference to the caused by drain. */
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return delegate.causedByDrain(doFn);
}
```
##########
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java:
##########
@@ -831,6 +836,11 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
return timestamp();
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return causedByDrain;
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```java
/**
* Provide a reference to the caused by drain.
*/
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return causedByDrain;
}
```
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/SplittableParDoNaiveBounded.java:
##########
@@ -543,6 +543,11 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
return outerContext.timestamp();
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return outerContext.causedByDrain();
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```suggestion
/** Provide a reference to the caused by drain. */
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return outerContext.causedByDrain();
}
```
##########
runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java:
##########
@@ -1119,6 +1129,11 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
return timestamp;
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ throw new UnsupportedOperationException("CausedByDrain parameters are not supported.");
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```java
/**
* Provide a reference to the caused by drain.
*/
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
throw new UnsupportedOperationException("CausedByDrain parameters are not supported.");
}
```
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java:
##########
@@ -325,6 +329,12 @@ public Instant timestamp(DoFn<InputT, OutputT> doFn) {
String.format("Timestamp unsupported in %s", getErrorContext()));
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ throw new UnsupportedOperationException(
+ String.format("CausedByDrain unsupported in %s", getErrorContext()));
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```suggestion
/** Provide a reference to the caused by drain. */
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
throw new UnsupportedOperationException(
String.format("CausedByDrain unsupported in %s", getErrorContext()));
}
```
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java:
##########
@@ -1357,6 +1362,11 @@ private static Parameter analyzeExtraParameter(
return Parameter.keyT(paramT);
} else if (rawType.equals(TimeDomain.class)) {
return Parameter.timeDomainParameter();
+ } else if (CausedByDrain.class.isAssignableFrom(rawType)) {
+ methodErrors.checkArgument(
+ rawType.equals(CausedByDrain.class),
+ "CausedByDrain argument must have type org.apache.beam.sdk.values.CausedByDrain.");
+ return Parameter.causedByDrainParameter();
Review Comment:

medium: Consider adding a comment to explain why `isAssignableFrom` is used here instead of `equals`.
```suggestion
} else if (CausedByDrain.class.isAssignableFrom(rawType)) {
methodErrors.checkArgument(
rawType.equals(CausedByDrain.class), // Ensure exact type match
"CausedByDrain argument must have type org.apache.beam.sdk.values.CausedByDrain.");
return Parameter.causedByDrainParameter();
```
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java:
##########
@@ -1804,6 +1804,11 @@ public <T> void outputWindowedValue(
outputTo(consumer, WindowedValues.of(output, timestamp, windows, paneInfo));
}
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return currentElement.causedByDrain();
+ }
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```suggestion
/** Provide a reference to the caused by drain. */
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return currentElement.causedByDrain();
}
```
##########
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java:
##########
@@ -1946,6 +1951,11 @@ public <T> void outputWindowedValue(
public CausedByDrain causedByDrain() {
return currentElement.causedByDrain();
}
+
+ @Override
+ public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
+ return currentElement.causedByDrain();
Review Comment:

medium: This method should include a Javadoc comment explaining its purpose and behavior, similar to other methods in this class.
```java
/** Provide a reference to the caused by drain. */
@Override
public CausedByDrain causedByDrain(DoFn<InputT, OutputT> doFn) {
return currentElement.causedByDrain();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4039292946
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enhances Apache Beam's `DoFn` capabilities by exposing the `CausedByDrain` parameter to both `@ProcessElement` and `@OnTimer` methods. This change allows developers to write more robust data processing pipelines that can gracefully handle job draining scenarios, enabling custom logic to be executed when a pipeline is being shut down or reconfigured.
### Highlights
* **CausedByDrain Parameter for ProcessElement**: Introduced the `CausedByDrain` parameter to `@ProcessElement` methods, allowing `DoFn`s to react to job draining events.
* **CausedByDrain Parameter for OnTimer**: Enabled the `CausedByDrain` parameter for `@OnTimer` methods, providing timer-based `DoFn`s with draining context.
* **Enhanced DoFn Signature Reflection**: Updated the `DoFn` signature reflection mechanism to correctly identify and handle the new `CausedByDrain` parameter type.
* **Comprehensive Testing**: Added comprehensive unit tests to validate the proper recognition and handling of the `CausedByDrain` parameter in `DoFn` signatures.
🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Changelog</b></summary>
* **runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java**
* Implemented the `causedByDrain` method in `DoFnProcessContext`, `DoFnTimerContext`, and `DoFnContext` to provide access to the `CausedByDrain` object.
* **sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnInvoker.java**
* Added `CausedByDrain` import and introduced `causedByDrain` to the `ArgumentProvider` interface and its default/delegating implementations.
* **sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignature.java**
* Extended the `Parameter` hierarchy with `CausedByDrainParameter` and updated the visitor pattern to include it.
* **sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/reflect/DoFnSignatures.java**
* Updated allowed parameter lists for various `DoFn` lifecycle methods to include `CausedByDrainParameter` and added logic to recognize `CausedByDrain` as a valid parameter type.
* **sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/SplittableParDoNaiveBounded.java**
* Updated the `ProcessContext` implementation to expose the `causedByDrain` method.
* **sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/reflect/DoFnSignaturesTest.java**
* Added new test cases and updated existing ones to verify the correct handling and reflection of the `CausedByDrain` parameter in `DoFn` signatures for both `@ProcessElement` and `@OnTimer` methods.
</details>
<details>
<summary><b>Activity</b></summary>
* No specific activity (comments, reviews, progress) has been recorded for this pull request yet.
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4045305045
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4042751542
Run Java_GCP_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4039865483
Run Java_Examples_Dataflow PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles merged PR #37825:
URL: https://github.com/apache/beam/pull/37825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4039867494
Run Java_PVR_Prism_Loopback PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4046083297
Assigning reviewers:
R: @chamikaramj for label java.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4039893387
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Drain] Expose drain to dofn processElement and onTimer [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37825:
URL: https://github.com/apache/beam/pull/37825#issuecomment-4040432545
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org