You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "gouthammylu (via GitHub)" <gi...@apache.org> on 2023/10/02 15:10:39 UTC
[I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
gouthammylu opened a new issue, #28767:
URL: https://github.com/apache/beam/issues/28767
### What happened?
We are creating a data pipeline to read the CSV files and write records to Bigquery and Spanner tables. In our pipeline we are connecting to 3 spanner databases, based on the file name we push data to one of the spanner database tables. The dataflow job was working fine till version 2.48.0. when we upgrade to 2.49.0 or 2.50.0 Dataflow job is in running state for a long time but no data hasbeen processed. We have seen below exception in the Diagnostics section.
Logs:
Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process
at java.base@11.0.9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at app//com.google.common.util.concurrent.OverflowAvoidingLockSupport.parkNanos(OverflowAvoidingLockSupport.java:36)
at app//com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:452)
at app//com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:121)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.pollUninterruptiblyWithTimeout(SessionPool.java:1591)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1568)
at app//com.google.cloud.spanner.SessionPool$WaiterFuture.get(SessionPool.java:1545)
at app//com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:66)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture.get(SessionPool.java:1302)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture.lambda$readOnlyTransaction$4(SessionPool.java:1195)
at app//com.google.cloud.spanner.SessionPool$PooledSessionFuture$$Lambda$372/0x0000000800882440.apply(Unknown Source)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext.getReadContextDelegate(SessionPool.java:234)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$6.load(SessionPool.java:472)
at app//com.google.cloud.spanner.SessionPool$CachedResultSetSupplier.get(SessionPool.java:153)
at app//com.google.cloud.spanner.SessionPool$CachedResultSetSupplier.get(SessionPool.java:141)
at app//com.google.cloud.spanner.ForwardingResultSet.next(ForwardingResultSet.java:54)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.internalNext(SessionPool.java:273)
at app//com.google.cloud.spanner.SessionPool$AutoClosingReadContext$1.next(SessionPool.java:253)
at app//org.apache.beam.sdk.io.gcp.spanner.ReadSpannerSchema.processElement(ReadSpannerSchema.java:95)
at app//org.apache.beam.sdk.io.gcp.spanner.ReadSpannerSchema$DoFnInvoker.invokeProcessElement(Unknown Source)
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [X] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-1743223655
Our SpannerIO load tests shows spanner write is functioning without issue, though metrics shows Spanner write could be slower than bigquery write and optimization is possible.
Please open a support ticket to GCP Dataflow; Job logs are important for this kind of investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-4742538168
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-5015776535
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-1743727412
Please check https://cloud.google.com/dataflow/docs/support/getting-support and feel free to mention this GitHub issue. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28767: [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process
URL: https://github.com/apache/beam/issues/28767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "gouthammylu (via GitHub)" <gi...@apache.org>.
gouthammylu commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-1743235197
How to open a support ticket to GCP Dataflow ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Operation ongoing in step EU/Write mutations to Cloud Spanner/Read information schema for at least 05m00s without outputting or completing in state process [beam]
Posted by "gouthammylu (via GitHub)" <gi...@apache.org>.
gouthammylu commented on issue #28767:
URL: https://github.com/apache/beam/issues/28767#issuecomment-1743238038
Why reading information schema is taking lot of time? even after two hours we were getting the same exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org