You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/12/01 19:18:00 UTC

[jira] [Commented] (QPID-8387) [Broker-J] JDBC based message stores do not wait for asynchronous message removal threads to complete when store is being closed

    [ https://issues.apache.org/jira/browse/QPID-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16985650#comment-16985650 ] 

ASF subversion and git services commented on QPID-8387:
-------------------------------------------------------

Commit 7193e26f6fad83f5ce81fd9ef855edb0fd5594ea in qpid-broker-j's branch refs/heads/master from Alex Rudyy
[ https://gitbox.apache.org/repos/asf?p=qpid-broker-j.git;h=7193e26 ]

QPID-8387: [Broker-J] Await for asyncroronous message removal tasks to complete on message store close


> [Broker-J] JDBC based message stores do not wait for asynchronous message removal threads to complete when store is being closed
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: QPID-8387
>                 URL: https://issues.apache.org/jira/browse/QPID-8387
>             Project: Qpid
>          Issue Type: Bug
>          Components: Broker-J
>    Affects Versions: qpid-java-broker-7.1.0, qpid-java-broker-7.1.1, qpid-java-broker-7.1.2, qpid-java-broker-7.1.3, qpid-java-broker-7.1.4, qpid-java-broker-7.1.5, qpid-java-broker-7.1.6
>            Reporter: Alex Rudyy
>            Priority: Major
>
> The asynchronous message removal was added into JDBC based message stores (Derby and JDBC) in version 7.1.0 of Qpid Broker-J as part of  [QPID-8242|https://issues.apache.org/jira/browse/QPID-8242]. The message removal now happens asynchronously in internal threads of JDBC based message stores.
> However, when store is closed, the closing functionality does not wait for the asynchronous message removal threads to complete. This issue causes several side effects:
> * the messages scheduled for removal might not be removed due to the close of the VirtualHost. Though, it is not an issue, as Broker on restart deletes all orphaned messages
> * the asynchronous removal might result in ServerScopedRuntimeException being thrown from the removal thread. At the moment, those ServerScopedRuntimeException are ignored, but the intention of ServerScopedRuntimeException is to halt the JVM, and, this does not happen
> * The restart of VirtualHost in integration tests can hang due to issues with store recovery
> The following exception can be thrown from the tests in {{PersistentMessagingTest}} when any of derby profiles is used
> {noformat}
> 019-11-29 12:33:23,062 DEBUG [Broker-Config] o.a.q.s.m.AbstractConfiguredObject All children closed DerbyVirtualHostNodeImplWithAccessChecking : PersistentMessagingTest_transactedAcknowledgementPersistence
> 2019-11-29 12:33:23,070 ERROR [PersistentMessagingTest_transactedAcknowledgementPersistence-store-1] o.a.q.s.u.ServerScopedRuntimeException Error removing messages with ids [3] from database: Java exception: ': org.apache.derby.shared.common.error.ShutdownException'.
> java.sql.SQLException: Java exception: ': org.apache.derby.shared.common.error.ShutdownException'.
> 	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedConnection.close(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedConnection.close(Unknown Source)
> 	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.removeMessages(AbstractJDBCMessageStore.java:505)
> 	at org.apache.qpid.server.store.jdbc.AbstractJDBCMessageStore.lambda$scheduleMessageRemoval$0(AbstractJDBCMessageStore.java:461)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.derby.iapi.error.StandardException: Java exception: ': org.apache.derby.shared.common.error.ShutdownException'.
> 	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> 	at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
> 	... 18 common frames omitted
> Caused by: org.apache.derby.shared.common.error.ShutdownException: 
> 	at org.apache.derby.iapi.services.context.ContextManager.checkInterrupt(Unknown Source)
> 	at org.apache.derby.iapi.services.context.ContextManager.getContext(Unknown Source)
> 	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.getStatementContext(Unknown Source)
> 	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.doRollback(Unknown Source)
> 	at org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.userRollback(Unknown Source)
> 	at org.apache.derby.impl.jdbc.TransactionResourceImpl.rollback(Unknown Source)
> 	... 11 common frames omitted
> 2019-11-29 12:33:24,673 INFO  [Broker-Config] q.m.c.close <<UNKNOWN>> [vh(/PersistentMessagingTest_transactedAcknowledgementPersistence)/ms(DerbyConfigurationStore)] CFG-1003 : Closed
> 2019-11-29 12:33:24,673 INFO  [main] o.a.q.t.u.EmbeddedBrokerPerClassAdminImpl Starting VirtualHostNode for restart
> {noformat}
> As minimum, the following changes need to be implemented
> * {{ServerScopedRuntimeException}} thrown from async message removal threads should halt Broker JVM
> * The message store close functionality should either wait for in-progress removal to complete, or, cancel removal tasks  gracefully
> * The issue with test hanging should be resolved



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org