You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/06/07 04:43:39 UTC
[GitHub] [beam] Abacn opened a new issue, #27040: [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition
Abacn opened a new issue, #27040:
URL: https://github.com/apache/beam/issues/27040
### What happened?
Test is flaky. Error message:
```
java.lang.NullPointerException
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.schemaUpdateOptionsTest(BigQueryIOWriteTest.java:2945)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldRelaxation(BigQueryIOWriteTest.java:2958)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:323)
at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest$1$1.evaluate(BigQueryIOWriteTest.java:229)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
```
https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Cron/1360/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testWriteFileSchemaUpdateOptionAllowFieldAddition_2_/
https://ci-beam.apache.org/job/beam_PreCommit_Java_GCP_IO_Direct_Commit/2779/testReport/junit/org.apache.beam.sdk.io.gcp.bigquery/BigQueryIOWriteTest/testWriteFileSchemaUpdateOptionAllowFieldRelaxation_2_/
### Issue Failure
Failure: Test is flaky
### Issue Priority
Priority: 2 (backlog / disabled test but we think the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition [beam]
Posted by "mxtymoshyk (via GitHub)" <gi...@apache.org>.
mxtymoshyk commented on issue #27040:
URL: https://github.com/apache/beam/issues/27040#issuecomment-6398122249
I couldn't reproduce this on current master, either in CI or locally.
### CI history
I checked every `beam_PreCommit_Java_GCP_IO_Direct` run from **2026-07-27 to 2026-10-25**. That goes back about as far as GitHub keeps logs.
| Window | Successful runs | Failed runs | Failed-run logs searched | Failures of these tests |
|---|---|---|---|---|
| 2026-09-16 to 2026-10-25 | 395 | 124 | 124 | **0** |
| 2026-07-27 to 2026-09-15 | 362 | 67 | 67 | **0** |
| **Total** | **757** | **191** | **191** | **0** |
Neither `testWriteFileSchemaUpdateOptionAllowFieldAddition` nor `testWriteFileSchemaUpdateOptionAllowFieldRelaxation` failed in any of these runs. The failed runs broke on integration tests and compile errors, plus a few other `BigQueryIOWriteTest` cases.
### Local runs
I ran the full `BigQueryIOWriteTest` class **5 times** on `a16836191dd`:
```
./gradlew :sdks:java:io:google-cloud-platform:test --tests 'org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest' --rerun
```
There were **no failures in any run**, and both of these tests passed in the batch and streaming variants every time.
### Suggestion
I'd **close this as not reproducible**. It can be reopened if it comes back.
<details>
<summary>Likely cause and a defensive fix, if it does come back</summary>
`schemaUpdateOptionsTest` calls this on every job returned by `fakeJobService.getAllJobs()`, with no null check:
```java
JobConfigurationLoad configuration = job.getConfiguration().getLoad();
assertEquals(expectedOptions, configuration.getSchemaUpdateOptions());
```
`FakeJobService.allJobs` is **static**. A non-load job in that list, such as a copy job, would make `getLoad()` return `null` and throw the NPE from the original report.
The fix would be to:
- skip jobs where `getLoad()` is `null`
- only assert on jobs whose destination is `project-id:dataset-id.table-id`
I haven't opened a PR for this because nothing shows the test failing now.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27040:
URL: https://github.com/apache/beam/issues/27040#issuecomment-5981014215
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #27040: [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition
URL: https://github.com/apache/beam/issues/27040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: BigQueryIOWriteTest.testWriteFileSchemaUpdateOptionAllowFieldAddition [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #27040:
URL: https://github.com/apache/beam/issues/27040#issuecomment-6478579001
Thanks for the investigation. Close this bug for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org