You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2026/10/19 18:45:38 UTC
[I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
github-actions[bot] opened a new issue, #38710:
URL: https://github.com/apache/beam/issues/38710
The PostCommit Java ValidatesRunner Dataflow Streaming is failing over 50% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=2&var-Workflow=PostCommit%20Java%20ValidatesRunner%20Dataflow%20Streaming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-6361545754
I investigated into the recent failures and this doesn’t look like a ValidatesRunner regression as the failing tests all hit the same GCE stockout in us-central1-c (ZONE_RESOURCE_POOL_EXHAUSTED), while most of the suite still passes
@Abacn We already swapped the schedule in #40351, so I think about reducing how many Dataflow jobs we launch at once
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-4799346139
Stabilized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-6365792866
There is ongoing capacity issue in us-central1. We are solving in the backends. As long as it's confirmed not Beam issue (and not permared) we're good for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-5219153288
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "durgaprasadml (via GitHub)" <gi...@apache.org>.
durgaprasadml commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-4583590691
### Summary of Investigation and Stabilization Plan for #38710
I investigated the flakiness of the PostCommit Java ValidatesRunner Dataflow Streaming workflow and identified several structural causes contributing to the >50% failure rate.
### Root Causes Identified
1. Unthrottled Parallelism / GCP Quota Exhaustion
- The validatesRunner task currently runs with:
groovy maxParallelForks Integer.MAX_VALUE
- Combined with GitHub Actions max-workers: 12, this can launch up to 12 concurrent Dataflow streaming jobs.
- This frequently exhausts GCP quotas/resources (Compute Engine IPs, CPUs, concurrent Dataflow jobs) and overloads the self-hosted runner.
2. Legacy Streaming Worker Non-Termination
- The workflow currently uses the legacy VM-based streaming path.
- Bounded streaming pipelines do not reliably terminate automatically under the legacy runner, causing jobs to remain in RUNNING until the 15-minute timeout expires.
3. Overly Aggressive Failure Detection
- TestDataflowRunner immediately cancels jobs upon encountering any JOB_MESSAGE_ERROR.
- In distributed streaming environments, transient worker/network errors are expected and are often automatically recovered by Dataflow retries.
- This behavior causes false negatives and unnecessary test failures.
### Implemented Fixes
#### 1. Throttled Concurrency
Reduced validatesRunner parallelism from unbounded forks to:
groovy maxParallelForks = 4
This significantly reduces:
- quota exhaustion,
- worker startup starvation,
- and CI runner overload.
#### 2. Migration to Dataflow Streaming Engine
Updated the workflow to use:
bash :runners:google-cloud-dataflow-java:validatesRunnerStreamingEngine
instead of the legacy streaming path.
Benefits:
- faster worker startup,
- better bounded-source termination behavior,
- lower infrastructure overhead,
- improved stability.
#### 3. Metrics-Driven Early Success/Failure Cancellation
Enhanced TestDataflowRunner.java to poll:
- PAssertSuccess
- PAssertFailure
during streaming execution.
Behavior:
- If assertions succeed → cancel immediately as success.
- If assertions fail → cancel immediately as failure.
This reduces successful test runtime from ~15 minutes to ~2–3 minutes and lowers flakiness exposure.
#### 4. Delayed Cancellation on Transient Errors
Modified the background cancellation logic to allow a recovery window before aborting on transient JOB_MESSAGE_ERROR entries.
This allows Dataflow self-healing/retries to stabilize the pipeline before the test is failed.
#### 5. Gradle Test Retry Support
Added the org.gradle.test-retry plugin for CI integration tests to automatically retry transient infrastructure failures.
### Validation
Added/updated tests for:
- early success termination,
- early failure termination,
- streaming metrics polling behavior.
Suggested verification command:
bash ./gradlew :runners:google-cloud-dataflow-java:test \ --tests "org.apache.beam.runners.dataflow.TestDataflowRunnerTest"
### Expected Outcome
These changes should:
- drastically reduce workflow flakiness,
- reduce GCP resource pressure,
- shorten overall workflow runtime,
- improve streaming test reliability,
- and stabilize CI execution behavior.
I’ve prepared the implementation changes and validation updates for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #38710: The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky
URL: https://github.com/apache/beam/issues/38710
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #38710: The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky
URL: https://github.com/apache/beam/issues/38710
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-6412506987
> 19\. There is ongoing capacity issue in us-central1. We are solving in the backends. As long as it's confirmed not Beam issue (and not permared) we're good for now.
@abac I also checked a few related PostCommit failures:
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2 #40764
beam_PostCommit_Java_DataflowV2 #39274
And those look like the same us-central1 stockout / quota issue (ZONE_RESOURCE_POOL_EXHAUSTED) as most tests pass and only a few fail when workers can’t start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-5241192428
Looks like this stabilized - it looks like temporary stockout issues:
```
Failed tests
[Tests](https://github.com/apache/beam/issues/38710#tab1)
[Standard error](https://github.com/apache/beam/issues/38710#tab2)
testInGlobalWindowBatchSizeByteSize
java.lang.RuntimeException: Startup of the worker pool in us-central1-c failed to bring up any of the desired 1 workers. This is likely a quota issue or a Compute Engine stockout. The service will retry. For troubleshooting steps, see https://cloud.google.com/dataflow/docs/guides/common-errors#worker-pool-failure for help troubleshooting. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: Instance 'groupintobatchestest0test-08072212-e23h-harness-6hzt' creation failed: The zone 'projects/apache-beam-testing/zones/us-central1-c' does not have enough resources available to fulfill the request. 'NULL:0/NULL:0/NULL:0 (state:STOCKOUT, sub-state:STOCKOUT, resource type:compute)'.
at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:149)
at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:101)
at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:56)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:442)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:381)
at org.apache.beam.sdk.transforms.GroupIntoBatchesTest.testInGlobalWindowBatchSizeByteSize(GroupIntoBatchesTest.java:164)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.lang.Thread.run(Thread.java:1583)
Wrap lines
Generated by [Gradle 8.14.3](https://www.gradle.org/) at Aug 8, 2026, 10:25:33 AM
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java ValidatesRunner Dataflow Streaming job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38710:
URL: https://github.com/apache/beam/issues/38710#issuecomment-6274657913
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org