You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2026/04/29 14:44:04 UTC
[PR] Deflake testMatchWatchForNewFiles [beam]
Abacn opened a new pull request, #38320:
URL: https://github.com/apache/beam/pull/38320
mitigate #19480
FileIOTest.testMatchWatchForNewFiles is very flaky, probably due to a heavy pipeline test 4 scenarios put into a single test. Consider split it into two test, for matchMetadata and matchAllUpdatedMetadata, construct and run pipeline; for matchAllMetadata and matchUpdatedMetadata, only construct pipeline and check the output PCollection is unbounded.
Previously it was testing full combination of two parameters. Picking two out of four covering all parameters the test coverage remains unchanged.
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake testMatchWatchForNewFiles [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4344796136
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request aims to improve the stability of the test suite by refactoring a flaky test case in FileIOTest. Additionally, it includes minor build configuration improvements for the DirectRunner and removes deprecated compatibility logic in the Python SDK to streamline the codebase.
### Highlights
* **Test Stability**: Split the flaky FileIOTest.testMatchWatchForNewFiles into two separate tests to improve reliability and maintain coverage.
* **Build Configuration**: Updated Gradle tasks in runners/direct-java/build.gradle to use tasks.register for better performance and configuration avoidance.
* **Python SDK Cleanup**: Removed legacy compatibility code for Apache Beam versions prior to 2.73.0 in sdks/python/apache_beam/transforms/core.py.
🧠**New Feature in Public Preview:** You can now enable **Memory** to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java watchForNewFiles [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4359374658
still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake testMatchWatchForNewFiles [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #38320:
URL: https://github.com/apache/beam/pull/38320#discussion_r3161922492
##########
sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java:
##########
@@ -376,6 +350,35 @@ public void testMatchWatchForNewFiles() throws IOException, InterruptedException
p.run();
}
+ @Test
+ public void testMatchWatchForNewFiles_UnboundedPCollection() {
+ // Additional scenarios for testMatchWatchForNewFiles. Only construct the pipeline and check
+ // output pcoll
+ final Path watchPath = tmpFolder.getRoot().toPath().resolve("watch");
+
Review Comment:
the enforcement isn't present when there is no runner in classpath, which is the case for core unit test
##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2448,7 +2448,7 @@ def check_threshold(bad, total, threshold, window=DoFn.WindowParam):
else:
return result
- def expand_2_72_0(self, pcoll):
+ def expand(self, pcoll):
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java watchForNewFiles [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38320: Deflake PreCommit Java watchForNewFiles
URL: https://github.com/apache/beam/pull/38320
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java and PostCommit Java Dataflow [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4356292261
Need a windows machine to debug the failing test (moved from needsRunnerTest to core unit test making it gets exercised on windows test workflow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java and PostCommit Java Dataflow [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #38320:
URL: https://github.com/apache/beam/pull/38320#discussion_r3171085429
##########
sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java:
##########
@@ -376,6 +349,35 @@ public void testMatchWatchForNewFiles() throws IOException, InterruptedException
p.run();
}
+ @Test
+ public void testMatchWatchForNewFiles_UnboundedPCollection() {
+ // Additional scenarios for testMatchWatchForNewFiles. Only construct the pipeline and check
+ // output pcoll
+ final Path watchPath = tmpFolder.getRoot().toPath().resolve("watch");
+
+ PCollection<MatchResult.Metadata> matchAllMetadata =
+ p.apply("create for matchAll new files", Create.of(watchPath + "/*"))
+ .apply(
+ "match filename through matchAll",
+ FileIO.matchAll()
+ .continuously(
+ Duration.millis(100),
+ Watch.Growth.afterTimeSinceNewOutput(Duration.standardSeconds(1))));
+
+ PCollection<MatchResult.Metadata> matchUpdatedMetadata =
+ p.apply(
+ "match updated",
+ FileIO.match()
+ .filepattern(watchPath.resolve("first").toString())
+ .continuously(
+ Duration.millis(100),
+ Watch.Growth.afterTimeSinceNewOutput(Duration.standardSeconds(1)),
+ true));
+
+ assertEquals(PCollection.IsBounded.UNBOUNDED, matchAllMetadata.isBounded());
+ assertEquals(PCollection.IsBounded.UNBOUNDED, matchUpdatedMetadata.isBounded());
+ }
Review Comment:
This new test method has two significant issues:
1. **Compilation Error**: At line 359, `watchPath + "/*"` will not compile because `watchPath` is a `java.nio.file.Path` object and Java does not support the `+` operator for `Path` and `String`. It should use `watchPath.resolve("*").toString()` to be correct and consistent with the rest of the file.
2. **Test Failure (Abandoned Nodes)**: The test uses the `TestPipeline` rule `p` but does not call `p.run()`. By default, `TestPipeline` throws an exception if nodes are added but the pipeline is not executed. Since this test only checks the boundedness of the PCollections, you should disable the abandoned node enforcement using `p.enableAbandonedNodeEnforcement(false)`.
Here is the corrected method:
```java
public void testMatchWatchForNewFiles_UnboundedPCollection() {
// Additional scenarios for testMatchWatchForNewFiles. Only construct the pipeline and check
// output pcoll
p.enableAbandonedNodeEnforcement(false);
final Path watchPath = tmpFolder.getRoot().toPath().resolve("watch");
PCollection<MatchResult.Metadata> matchAllMetadata =
p.apply("create for matchAll new files", Create.of(watchPath.resolve("*").toString()))
.apply(
"match filename through matchAll",
FileIO.matchAll()
.continuously(
Duration.millis(100),
Watch.Growth.afterTimeSinceNewOutput(Duration.standardSeconds(1))));
PCollection<MatchResult.Metadata> matchUpdatedMetadata =
p.apply(
"match updated",
FileIO.match()
.filepattern(watchPath.resolve("first").toString())
.continuously(
Duration.millis(100),
Watch.Growth.afterTimeSinceNewOutput(Duration.standardSeconds(1)),
true));
assertEquals(PCollection.IsBounded.UNBOUNDED, matchAllMetadata.isBounded());
assertEquals(PCollection.IsBounded.UNBOUNDED, matchUpdatedMetadata.isBounded());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake testMatchWatchForNewFiles [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #38320:
URL: https://github.com/apache/beam/pull/38320#discussion_r3161898151
##########
sdks/python/apache_beam/transforms/core.py:
##########
@@ -2448,7 +2448,7 @@ def check_threshold(bad, total, threshold, window=DoFn.WindowParam):
else:
return result
- def expand_2_72_0(self, pcoll):
+ def expand(self, pcoll):
Review Comment:
This change appears to be an accidental regression. It renames `expand_2_72_0` to `expand` and deletes the existing `expand` method (which was located at lines 2463-2483 in the previous version). The deleted method contained critical logic for handling type hints and dead letter tags for Beam versions >= 2.73.0. This change is unrelated to the PR's objective of deflaking a Java test and will break functionality in the Python SDK.
##########
sdks/java/core/src/test/java/org/apache/beam/sdk/io/FileIOTest.java:
##########
@@ -376,6 +350,35 @@ public void testMatchWatchForNewFiles() throws IOException, InterruptedException
p.run();
}
+ @Test
+ public void testMatchWatchForNewFiles_UnboundedPCollection() {
+ // Additional scenarios for testMatchWatchForNewFiles. Only construct the pipeline and check
+ // output pcoll
+ final Path watchPath = tmpFolder.getRoot().toPath().resolve("watch");
+
Review Comment:
The `TestPipeline` rule enforces that `run()` must be called if any transforms are applied to the pipeline. Since this new test only verifies the boundedness of the `PCollection` and intentionally avoids calling `p.run()` to save time/resources, it will fail with an `IllegalStateException: TestPipeline.run() was never called.`
You should disable the abandoned node enforcement for this specific test to allow it to pass without calling `run()`.
```java
public void testMatchWatchForNewFiles_UnboundedPCollection() {
p.enableAbandonedNodeEnforcement(false);
// Additional scenarios for testMatchWatchForNewFiles. Only construct the pipeline and check
// output pcoll
final Path watchPath = tmpFolder.getRoot().toPath().resolve("watch");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java and PostCommit Java Dataflow [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4346691367
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java watchForNewFiles [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4843853477
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deflake PreCommit Java watchForNewFiles [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38320:
URL: https://github.com/apache/beam/pull/38320#issuecomment-4904234308
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org