You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2026/01/09 18:47:33 UTC
[I] The PostCommit Java PVR Spark Batch job is flaky [beam]
github-actions[bot] opened a new issue, #30512:
URL: https://github.com/apache/beam/issues/30512
The PostCommit Java PVR Spark Batch is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_PVR_Spark_Batch.yml?query=is%3Afailure+branch%3Amaster to see the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2402962813
Run tests locally (macOS). There are lots of this log:
```
[spark-runner-job-invoker-0] INFO org.apache.spark.executor.Executor - Adding file:/private/var/folders/wg/hwmcqjwd4zz75mjs0r5z_3f400y2yj/T/spark-fad6dd06-5673-4920-9176-3a801211fa9d/userFiles-f57ec6cf-273d-46c9-b251-ff365803ad6a/commons-lang-2.6.jar to class loader
```
and there is also intermittent test failure, but now FileNotFoundException has a hint (Too many files open)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-4924132101
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2586295750
Job has been stable for 2 months. Closing as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2439785365
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-3625138347
Stabilzied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-3730159591
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-4533761470
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-3620885352
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-4939074990
Stabilized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-4677350851
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2441997749
Likely due to worker upgraded to Java 11, however the actual error is not surfaced. A first step would be making the "FnHarness Startup failed" report the underlying Exception, making it debuggable.
This test has been a trouble maker as Spark in memory runner doing dirty ops on class loader and often caused conflict. Remove milestone for now until we have bandwidth to look into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-3736656720
Fixed by #37270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30512: The PostCommit Java PVR Spark Batch job is flaky
URL: https://github.com/apache/beam/issues/30512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2402923733
Taking a closer look, tests are failing on initializing TestPipeline:
```
Caused by: java.io.InvalidClassException: org.apache.beam.sdk.transforms.ParDoTest$LifecycleTests; unable to create instance
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.util.ServiceConfigurationError: org.apache.beam.sdk.io.FileSystemRegistrar: Error reading configuration file
...
at org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Lists.newArrayList(Lists.java:132)
at org.apache.beam.sdk.io.FileSystems.setDefaultPipelineOptions(FileSystems.java:587)
at org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:518)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:273)
...
Caused by: java.io.FileNotFoundException: /tmp/spark-f76d7a05-d782-41a9-9926-089251e24bdd/userFiles-7506344e-2e1b-43ea-8ccb-5bbc936d955f/beam-runners-direct-java-2.60.0-SNAPSHOT.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
at java.net.URL.openStream(URL.java:1093)
at java.util.ServiceLoader.parse(ServiceLoader.java:304)
```
the failing call is "setDefaultPipelineOptions", which loads filesystem registrar that are autoService. It sounds similar to https://github.com/google/auto/issues/718
Another observation is that the test trying to read from `/tmp/spark-***` which looks like the staged jars for Spark runner pipeline. However the invocation is inside junit code, and it looks like the staged jar paths leaked into the host test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2301960699
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-2110951142
Random tests in this test suite failing due to tmp file get deleted half way, likely a racing issue.
This is recurring for a long time.
```
java.lang.RuntimeException: The Runner experienced the following error during execution:
java.lang.RuntimeException: Error received from SDK harness for instruction 4: java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With Execution Info
at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:78)
at org.apache.beam.sdk.util.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:767)
at org.apache.beam.sdk.util.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:391)
at org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:379)
...
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.InvalidClassException: org.apache.beam.sdk.transforms.ParDoTest$StateTests; unable to create instance
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2200)
...
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:75)
... 19 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedSerializationConstructorAccessor390.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.io.ObjectStreamClass.newInstance(ObjectStreamClass.java:1102)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2197)
... 31 more
Caused by: java.util.ServiceConfigurationError: org.apache.beam.sdk.io.FileSystemRegistrar: Error reading configuration file
...
at org.apache.beam.sdk.transforms.ParDoTest$SharedTestBase.<init>(ParDoTest.java:193)
... 35 more
Caused by: java.io.FileNotFoundException: /tmp/spark-4426f648-ee8b-49e3-9545-dffde22d1a6f/userFiles-b02ee903-820b-4b77-b1b7-69f020d54c8b/beam-sdks-java-extensions-google-cloud-platform-core-2.57.0-SNAPSHOT.jar (No such file or directory)
at java.util.zip.ZipFile.open(Native Method)
...
at java.net.URL.openStream(URL.java:1093)
at java.util.ServiceLoader.parse(ServiceLoader.java:304)
... 46 more
at org.apache.beam.runners.portability.JobServicePipelineResult.propagateErrors(JobServicePipelineResult.java:176)
at org.apache.beam.runners.portability.JobServicePipelineResult.waitUntilFinish(JobServicePipelineResult.java:117)
at org.apache.beam.runners.portability.testing.TestPortableRunner.run(TestPortableRunner.java:81)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:325)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:404)
...
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit Java PVR Spark Batch job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30512:
URL: https://github.com/apache/beam/issues/30512#issuecomment-4701330730
Stabilized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org