You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/15 16:01:13 UTC
[GitHub] [beam] Abacn opened a new issue, #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Abacn opened a new issue, #22299:
URL: https://github.com/apache/beam/issues/22299
### What happened?
Re-raise BEAM-9629 as this seems not fully resolved.
Affect versions: at least >= 2.25, likely >=2.18 as in BEAM-9629.
log using 2.35.0
```
Operation ongoing in step Write Blacklisted Posts/ParDo(Write) for at least 45m00s without outputting or completing in state finish at ...
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:581) at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:437) at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:354) at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.getConnection(JdbcIO.java:2029) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.executeBatch(JdbcIO.java:2087) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.finishBundle(JdbcIO.java:2048) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)
```
log using 2.25.0:
```
warnings:
Operation ongoing in step Write Blacklisted Posts/ParDo(Write) for at least 45m00s without outputting or completing in state finish at ... at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at
org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:581) at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:437) at
org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:354) at
org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134) at
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:734) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.getConnection(JdbcIO.java:2029) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.executeBatch(JdbcIO.java:2087) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn.finishBundle(JdbcIO.java:2048) at
org.apache.beam.sdk.io.jdbc.JdbcIO$WriteFn$DoFnInvoker.invokeFinishBundle(Unknown Source)
```
### Issue Priority
Priority: 1
### Issue Component
Component: io-java-jdbc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #22299:
URL: https://github.com/apache/beam/issues/22299#issuecomment-1185697215
A possible cause is that a racing condition could occur here:
https://github.com/apache/beam/blob/2f13855aff620f0df020aead22e5d14ccab0f85e/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java#L2323
getConnection() gets called by many process element at the same time, with connection == null, and all of them trying to establish the dataSource.getConnection at the same time. Marking this async may solve the issue.
However, verifying this is blocked by broken JdbcIOIT, where the fix is currently in progress (#21796)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #22299:
URL: https://github.com/apache/beam/issues/22299#issuecomment-1285850225
looking at this more, it can't be in @setup, but it may be possible to use @startbundle instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #22299:
URL: https://github.com/apache/beam/issues/22299#issuecomment-1285893167
Context of last change: #12209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on issue #22299:
URL: https://github.com/apache/beam/issues/22299#issuecomment-1285762119
is there a particular reason to not just put connection setup logic into @setup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey closed issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
Posted by GitBox <gi...@apache.org>.
johnjcasey closed issue #22299: [Bug]: JDBCIO Write freeze at getConnection() in WriteFn
URL: https://github.com/apache/beam/issues/22299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org