You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2025/08/14 15:16:40 UTC
[I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Abacn opened a new issue, #35868:
URL: https://github.com/apache/beam/issues/35868
### What happened?
see https://github.com/apache/beam/runs/48025477476
error message:
no result returned
```
java.lang.AssertionError: ParDo(ModsToString)/ParMultiDo(ModsToString).output:
Expected: iterable with items [...] in any order
but: no item matches: "..." in []
at org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:176)
at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:461)
at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:453)
at org.apache.beam.sdk.io.gcp.spanner.changestreams.it.SpannerChangeStreamIT.testReadSpannerChangeStreamImpl(SpannerChangeStreamIT.java:162)
```
```
java.lang.AssertionError:
Expected: (stack trace contains string 'SpannerException' and stack trace contains string 'NOT_FOUND: Session not found')
but: stack trace contains string 'NOT_FOUND: Session not found' was <org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted>
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted
```
and other errors.
this is blocking GCP BOM upgrade #35854.
### Issue Failure
Failure: Test is flaky
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [ ] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3617278375
The spanner issue relevant to this bug is fixed at 6.104.0. Some failed unit tests have been disabled as further refactoring is needed(#37003), while IT tests will remain unchanged and functional.
This bug is superseded by #37008.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3539447950
The change stream IT tests are broken by google_cloud_spanner_version to 6.98.0 and later, when multiplex session is used, the transaction tag is missing. This need to be fixed or the change stream connector customers will be impacted. @jiangzzhu please follow up. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #35868: [Failing Test]: Spanner change stream tests failing on latest GCP BOM
URL: https://github.com/apache/beam/issues/35868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3198335371
According to the error message, the issue is around MultiplexedSessionDatabaseClient in the cloud spanner java client lib.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3573547677
Per my conversation with @jiangzzhu offline, the fix for the client library will be delayed past the 2.70.0 release due to the internal code freeze.
Moving this to the milestone of the 2.71.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3518413458
The tests still failing in the latest GCP_BOM (26.71.0): https://github.com/apache/beam/pull/36787/checks?check_run_id=55108217053
```
testReadSpannerChangeStream (org.apache.beam.sdk.io.gcp.spanner.changestreams.it.SpannerChangeStreamIT) failed
Expected: iterable with items [...] in any order
but: no item matches: ... in []
```
basically no output. Same symptom as https://github.com/apache/beam/issues/35868#issuecomment-3201031891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "chenxuesdu (via GitHub)" <gi...@apache.org>.
chenxuesdu commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3530190715
I ran the test SpannerChangeStreamIT locally and found that the test started to fail after using the version of com.google.cloud:google-cloud-spanner to 6.98.0. The previous versions works fine.
The error is caused by missing transaction_tag value of change stream record, where the transaction_tag is set when updating database using databaseClient, for example: [ Options.tag("app=beam;action=update"))](https://github.com/apache/beam/blob/db08b7c8c0b9e6d0fba5183851fc0b74f9267129/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/it/SpannerChangeStreamIT.java#L334).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3198347732
The multiplex session issue is not relevant to the recent change stream connector v2 implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3201031891
To reproduce locally
- checkout #35854 (`git fetch upstream pull/35854/head:pr-35854 && git switch pr-35854`)
- Run
```
./gradlew :sdks:java:io:google-cloud-platform:integrationTest --tests org.apache.beam.sdk.io.gcp.spanner.changestreams.it.SpannerChangeStreamIT -DintegrationTestPipelineOptions='["--project=<your project>","--instanceId=<spanner instance>"]'
```
It seems not fully deterministic. Assertion error happens if I run `SpannerChangeStreamIT` suite, but sometimes goes away when run single test under `SpannerChangeStreamIT`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "changliiu (via GitHub)" <gi...@apache.org>.
changliiu commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3198376755
Example error message:
```
Expected: (stack trace contains string 'SpannerException' and stack trace contains string 'NOT_FOUND: Session not found')
but: stack trace contains string 'NOT_FOUND: Session not found' was <org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted>
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping closed issue #35868: [Failing Test]: Spanner change stream tests failing on latest GCP BOM
URL: https://github.com/apache/beam/issues/35868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "rahul2393 (via GitHub)" <gi...@apache.org>.
rahul2393 commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3207586841
@Abacn @changliiu Multiplex session can't be deleted so this is something not intended in tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Spanner change stream tests failing on latest GCP BOM [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #35868:
URL: https://github.com/apache/beam/issues/35868#issuecomment-3534579345
@chenxuesdu thanks for triaging this. Is this expected behavior change, and if so, what would need to change for SpannerChangeStreamIT to make test pass again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org