You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by magellings <ma...@qg.com> on 2009/04/08 00:13:19 UTC
Re: Is there JDBC master/slave retry logic if database goes down?
How do you exactly provide your own implementation?
On the JDBCPersistenceAdaptor the createDatabaseLocker will always use the
DefaultDatabaseLocker:
protected DatabaseLocker createDatabaseLocker() throws IOException {
return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
}
Can I provide my own implemenation declaratively through configuration
provided I write a class deriving from JDBCPersistenceAdaptor of which
overrides the createDatabaseLocker class?
Gary Tully wrote:
>
> One thing to note: you can provide your own implementation of the
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> keepAlive to be aware of your database backup policy.
>
> If you go down this route and conceive a better default or optional
> strategy, feel free to submit it as a patch.
>
>
>
> 2009/3/11 magellings <ma...@qg.com>
>
>>
>> Well it was butt-simple to get working like the c3p0 documentation said.
>> I
>> upped the delay to 20s, unfortunately the master broker still failed.
>>
>> I'm kind of wondering how the slave would respond. My guess is it would
>> probably time out too.
>>
>> Any thoughts?
>>
>> See further below for the errors.
>>
>>
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
>> xbean:activemq.xml
>> ACTIVEMQ_HOME:
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> ACTIVEMQ_BASE:
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> Loading message broker from: xbean:activemq.xml
>> INFO DefaultCamelContext - JMX enabled. Using
>> InstrumentationLifecycleStrategy.
>> INFO MLog - MLog clients using log4j logging.
>> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
>> 21-May-2007 15:04:56; debug? true; trace: 10]
>> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
>> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> acquireRetryAttemp
>> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
>> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> checkoutTimeout -
>> > 30000, connectionCustomizerClassName -> null, connectionTesterClassName
>> ->
>> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
>> 2rvy1
>> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
>> description -> null, driverClass ->
>> com.microsoft.sqlserver.jdbc.SQLServerDri
>> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
>> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> idleConnectionTestPerio
>> d -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$word,
>> maxAdministrativeTas
>> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
>> 200,
>> maxStatementsPerCon
>> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
>> properties -> {user=**
>> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
>> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout ->
>> 0,
>> use
>> sTraditionalReflectiveProxies -> false ]
>> INFO BrokerService - Using Persistence Adapter:
>> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
>> acquireIncrement
>> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
>> autoCommitOnClose -> false, automaticTestTable -> con_test,
>> breakAfterAcquireFailure ->
>> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName ->
>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> debugUnreturnedConnectionStackTraces -> false, description -> null,
>> driverClass -> com.microsoft.sql
>> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
>> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> 2rvy117z1wqpy2hxzkebp|18105e8,
>> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$wo
>> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime ->
>> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10, numHelperThreads
>> ->
>> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null
>> , properties -> {user=******, password=******}, propertyCycle -> 0,
>> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
>> unreturnedCon
>> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
>> INFO JDBCPersistenceAdapter - Database driver recognized:
>> [microsoft_sql_server_2005_jdbc_driver]
>> INFO DefaultDatabaseLocker - Attempting to acquire the
>> exclusive
>> lock to become the Master broker
>> INFO DefaultDatabaseLocker - Becoming the master on dataSource:
>> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> acquireRe
>> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
>> false,
>> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> checkout
>> Timeout -> 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName ->
>> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
>> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces
>> ->
>> false, description -> null, driverClass ->
>> com.microsoft.sqlserver.jdbc.SQL
>> ServerDriver, factoryClassLocation -> null,
>> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
>> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$word,
>> maxAdminist
>> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
>> 200,
>> maxStateme
>> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
>> properties ->
>> {user=******, password=******}, propertyCycle -> 0,
>> testConnectionOnCheckin
>> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
>> -> 0, usesTraditionalReflectiveProxies -> false ]
>> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
>> (master) is starting
>> INFO BrokerService - For help or more information
>> please
>> see: http://activemq.apache.org/
>> INFO TransportServerThreadSupport - Listening for connections at:
>> tcp://QG076619:61616
>> INFO TransportConnector - Connector openwire Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> ssl://QG076619:61617
>> INFO TransportConnector - Connector ssl Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> stomp://QG076619:61613
>> INFO TransportConnector - Connector stomp Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> xmpp://QG076619:61222
>> INFO TransportConnector - Connector xmpp Started
>> INFO BrokerService - ActiveMQ JMS Message Broker
>> (master,
>> ID:QG076619-3632-1236812198396-0:0) started
>> INFO log - Logging to
>> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> INFO log - jetty-6.1.9
>> INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
>> INFO /admin - Initializing Spring
>> FrameworkServlet
>> 'dispatcher'
>> INFO log - ActiveMQ Console at
>> http://0.0.0.0:8162/admin
>> INFO log - ActiveMQ Web Demos at
>> http://0.0.0.0:8162/demo
>> INFO log - RESTful file access application at
>> http://0.0.0.0:8162/fileserver
>> INFO log - Started
>> SelectChannelConnector@0.0.0.0:8162
>> WARN BrokerRegistry - Broker localhost not started so
>> using
>> master instead
>> INFO TransportConnector - Connector vm://localhost Started
>>
>>
>> Then BOOM! Although behavior is different. Now it takes longer to stop.
>> :) "The connection to '/10.0.1.5:3333' is taking a long time to
>> shutdown."
>> If you look closely you can see activeMQ is timing out.
>>
>>
>>
>>
>> ERROR DefaultDatabaseLocker - Failed to update database lock:
>> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
>> et write error
>> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> peer:
>> socket write error
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
>> Source)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
>> at
>>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> INFO GooGooStatementCache - Problem with checked-in Statement,
>> discarding.
>> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
>> closed.
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
>> Source)
>> at
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
>> at
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
>> at
>>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> INFO JDBCPersistenceAdapter - No longer able to keep the
>> exclusive
>> lock so giving up being a master
>> INFO BrokerService - ActiveMQ Message Broker (master,
>> ID:QG076619-3332-1236811661934-0:0) is shutting down
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> WARN ThreadPoolAsynchronousRunner -
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> -- APPARENT DEADLOCK!!! Crea
>> rgency threads for unassigned pending tasks!
>> WARN ThreadPoolAsynchronousRunner -
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> -- APPARENT DEADLOCK!!! Comp
>> tus:
>> Managed Threads: 3
>> Active Threads: 3
>> Active Tasks:
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
>> Pending Tasks:
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
>> Pool thread stack traces:
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> ERROR QueueStorePrefetch - Failed to get message count
>> java.io.IOException: An attempt by a client to checkout a Connection has
>> timed out.
>> at
>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> at
>>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> at
>>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> at
>>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> at
>>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> at
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> at
>>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> at
>>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> at
>>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> at
>>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> Connection has timed out.
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> at
>>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> ... 20 more
>> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
>> out
>> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> ResourcePool@d964af -- timeout at awaitAvailable()
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> ... 22 more
>> WARN TransportConnection - Failed to remove consumer:
>> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
>> java.lang.RuntimeException
>> o.IOException: An attempt by a client to checkout a Connection has timed
>> out.
>> java.lang.RuntimeException: java.io.IOException: An attempt by a client
>> to
>> checkout a Connection has timed out.
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
>> at
>>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> at
>>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> at
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> at
>>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> at
>>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> at
>>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> at
>>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> Caused by: java.io.IOException: An attempt by a client to checkout a
>> Connection has timed out.
>> at
>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> at
>>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> at
>>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> ... 16 more
>> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> Connection has timed out.
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> at
>>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> ... 20 more
>> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
>> out
>> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> ResourcePool@d964af -- timeout at awaitAvailable()
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> ... 22 more
>> INFO TransportConnector - Connector openwire Stopped
>>
>>
>> bsnyder wrote:
>> >
>> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
>> wrote:
>> >>
>> >> Thanks for the response Bruce. :)
>> >>
>> >> To my knowledge we aren't using a connection pooler. We're just
>> >> configuring
>> >> the activemq xml and letting activeMQ do its thing. One of my
>> colleagues
>> >> mentioned SNAC since we are using Sql Server, not sure if that is
>> similar
>> >> to
>> >> a connection pooler or not.
>> >
>> > The job of a JDBC connection pooler is to keep a pool of JDBC
>> > connections available for use by a Java application. IIRC, SNAC stands
>> > for SQL Native Client which has nothing to do with a JDBC connection
>> > pooler.
>> >
>> >> In our scenario during a Sql Server failover it may take 15 minutes.
>> We
>> >> aren't using a high availability cluster. Does a connection pooler
>> give
>> >> the
>> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
>> >
>> > One of the options in a good JDBC connection pooler is the ability to
>> > test JDBC connections as it hands them out to the Java application. If
>> > the connection is stale for any reason the JDBC connection pooler will
>> > evict that connection from the pool and try the next one until it
>> > reaches one that is alive. To my knowledge, some JDBC connection
>> > poolers provide a the ability to configure some attributes like number
>> > of retries, etc. E.g., here is some information about what c3p0
>> > offers:
>> >
>> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
>> >
>> > The one big caveat I will mention is that longer outage periods such
>> > as the 15 minute outage you mention are not typically handled by a
>> > JDBC connection pooler as they're mainly geared toward handling
>> > momentary outages. As an example, notice that the default value for
>> > acquireRetryDelay in c3p0 is only one second:
>> >
>> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
>> >
>> > But you could certainly configure this to any value you want. I
>> > encourage you to test this in your environment before deploying to
>> > production.
>> >
>> > Bruce
>> > --
>> > perl -e 'print
>> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> > );'
>> >
>> > Apache ActiveMQ - http://activemq.apache.org/
>> > Apache Camel - http://camel.apache.org/
>> > Apache ServiceMix - http://servicemix.apache.org/
>> >
>> > Blog: http://bruceblog.org/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Gary Tully <ga...@gmail.com>.
<bean id='dbLocker" class="org.example.newAndBetter.DataBaseLocker" />
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#mssql-ds"
dataBaseLocker="#dbLocker"/>
</persistenceAdapter>
You have got to implement the DataBaseLocker interface in code though or
extend the current default to make it behave as you need.
2009/4/9 magellings <ma...@qg.com>
>
> This implies that I'd need to modify activemq source though doesn't it?
>
> Do you have an example of how it could be done through configuration?
>
>
> Gary Tully wrote:
> >
> > in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
> > set a locker instance.
> > see:
> >
> http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)<http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker%28org.apache.activemq.store.jdbc.DatabaseLocker%29>
> >
> > Otherwise, yea, as you describe, derive and inject using spring via the
> > XML
> > configuration.
> >
> > hope this helps,
> > Gary.
> > 2009/4/7 magellings <ma...@qg.com>
> >
> >>
> >> How do you exactly provide your own implementation?
> >>
> >> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use
> >> the
> >> DefaultDatabaseLocker:
> >>
> >> protected DatabaseLocker createDatabaseLocker() throws IOException {
> >> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
> >> }
> >>
> >> Can I provide my own implemenation declaratively through configuration
> >> provided I write a class deriving from JDBCPersistenceAdaptor of which
> >> overrides the createDatabaseLocker class?
> >>
> >>
> >> Gary Tully wrote:
> >> >
> >> > One thing to note: you can provide your own implementation of the
> >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> >> > keepAlive to be aware of your database backup policy.
> >> >
> >> > If you go down this route and conceive a better default or optional
> >> > strategy, feel free to submit it as a patch.
> >> >
> >> >
> >> >
> >> > 2009/3/11 magellings <ma...@qg.com>
> >> >
> >> >>
> >> >> Well it was butt-simple to get working like the c3p0 documentation
> >> said.
> >> >> I
> >> >> upped the delay to 20s, unfortunately the master broker still failed.
> >> >>
> >> >> I'm kind of wondering how the slave would respond. My guess is it
> >> would
> >> >> probably time out too.
> >> >>
> >> >> Any thoughts?
> >> >>
> >> >> See further below for the errors.
> >> >>
> >> >>
> >> >>
> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
> >> >> xbean:activemq.xml
> >> >> ACTIVEMQ_HOME:
> >> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> >> ACTIVEMQ_BASE:
> >> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> >> Loading message broker from: xbean:activemq.xml
> >> >> INFO DefaultCamelContext - JMX enabled. Using
> >> >> InstrumentationLifecycleStrategy.
> >> >> INFO MLog - MLog clients using log4j
> >> logging.
> >> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2
> >> [built
> >> >> 21-May-2007 15:04:56; debug? true; trace: 10]
> >> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
> >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> >> acquireRetryAttemp
> >> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> >> checkoutTimeout -
> >> >> > 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName
> >> >> ->
> >> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
> >> >> 2rvy1
> >> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
> >> false,
> >> >> description -> null, driverClass ->
> >> >> com.microsoft.sqlserver.jdbc.SQLServerDri
> >> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions
> >> ->
> >> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> idleConnectionTestPerio
> >> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> >> maxAdministrativeTas
> >> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> >> 200,
> >> >> maxStatementsPerCon
> >> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null,
> >> >> properties -> {user=**
> >> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin
> ->
> >> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
> >> ->
> >> >> 0,
> >> >> use
> >> >> sTraditionalReflectiveProxies -> false ]
> >> >> INFO BrokerService - Using Persistence Adapter:
> >> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
> >> >> acquireIncrement
> >> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
> >> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
> >> >> breakAfterAcquireFailure ->
> >> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName ->
> >> null,
> >> >> connectionTesterClassName ->
> >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> >> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
> >> >> driverClass -> com.microsoft.sql
> >> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
> >> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> >> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$wo
> >> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0,
> maxIdleTime
> >> ->
> >> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100,
> >> maxStatements
> >> >> ->
> >> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
> >> numHelperThreads
> >> >> ->
> >> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null
> >> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
> >> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
> >> >> unreturnedCon
> >> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
> >> >> INFO JDBCPersistenceAdapter - Database driver recognized:
> >> >> [microsoft_sql_server_2005_jdbc_driver]
> >> >> INFO DefaultDatabaseLocker - Attempting to acquire the
> >> >> exclusive
> >> >> lock to become the Master broker
> >> >> INFO DefaultDatabaseLocker - Becoming the master on
> >> dataSource:
> >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> >> acquireRe
> >> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
> >> >> false,
> >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> >> checkout
> >> >> Timeout -> 30000, connectionCustomizerClassName -> null,
> >> >> connectionTesterClassName ->
> >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
> >> >> -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> debugUnreturnedConnectionStackTraces
> >> >> ->
> >> >> false, description -> null, driverClass ->
> >> >> com.microsoft.sqlserver.jdbc.SQL
> >> >> ServerDriver, factoryClassLocation -> null,
> >> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
> >> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> >> maxAdminist
> >> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> >> 200,
> >> >> maxStateme
> >> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null,
> >> >> properties ->
> >> >> {user=******, password=******}, propertyCycle -> 0,
> >> >> testConnectionOnCheckin
> >> >> -> false, testConnectionOnCheckout -> false,
> >> unreturnedConnectionTimeout
> >> >> -> 0, usesTraditionalReflectiveProxies -> false ]
> >> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message
> >> Broker
> >> >> (master) is starting
> >> >> INFO BrokerService - For help or more information
> >> >> please
> >> >> see: http://activemq.apache.org/
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> tcp://QG076619:61616
> >> >> INFO TransportConnector - Connector openwire Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> ssl://QG076619:61617
> >> >> INFO TransportConnector - Connector ssl Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> stomp://QG076619:61613
> >> >> INFO TransportConnector - Connector stomp Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> xmpp://QG076619:61222
> >> >> INFO TransportConnector - Connector xmpp Started
> >> >> INFO BrokerService - ActiveMQ JMS Message Broker
> >> >> (master,
> >> >> ID:QG076619-3632-1236812198396-0:0) started
> >> >> INFO log - Logging to
> >> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
> >> >> org.mortbay.log.Slf4jLog
> >> >> INFO log - jetty-6.1.9
> >> >> INFO WebConsoleStarter - ActiveMQ WebConsole
> >> initialized.
> >> >> INFO /admin - Initializing Spring
> >> >> FrameworkServlet
> >> >> 'dispatcher'
> >> >> INFO log - ActiveMQ Console at
> >> >> http://0.0.0.0:8162/admin
> >> >> INFO log - ActiveMQ Web Demos at
> >> >> http://0.0.0.0:8162/demo
> >> >> INFO log - RESTful file access
> application
> >> at
> >> >> http://0.0.0.0:8162/fileserver
> >> >> INFO log - Started
> >> >> SelectChannelConnector@0.0.0.0:8162
> >> >> WARN BrokerRegistry - Broker localhost not started
> so
> >> >> using
> >> >> master instead
> >> >> INFO TransportConnector - Connector vm://localhost
> >> Started
> >> >>
> >> >>
> >> >> Then BOOM! Although behavior is different. Now it takes longer to
> >> stop.
> >> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
> >> >> shutdown."
> >> >> If you look closely you can see activeMQ is timing out.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ERROR DefaultDatabaseLocker - Failed to update database
> lock:
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> pe
> >> >> et write error
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> >> peer:
> >> >> socket write error
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown
> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown
> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
> >> >> Source)
> >> >> at
> >> com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
> >> Source)
> >> >> at
> >> >>
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
> >> >> Source)
> >> >> at
> >> >>
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> >> at
> >> >>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> >> at
> >> >>
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> >> at
> >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> INFO GooGooStatementCache - Problem with checked-in
> >> Statement,
> >> >> discarding.
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
> >> >> closed.
> >> >> at
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
> >> >> Source)
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
> >> >> Source)
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> >> at
> >> >>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> >> at
> >> >>
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> >> at
> >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> INFO JDBCPersistenceAdapter - No longer able to keep the
> >> >> exclusive
> >> >> lock so giving up being a master
> >> >> INFO BrokerService - ActiveMQ Message Broker
> >> (master,
> >> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> WARN ThreadPoolAsynchronousRunner -
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> >> -- APPARENT DEADLOCK!!! Crea
> >> >> rgency threads for unassigned pending tasks!
> >> >> WARN ThreadPoolAsynchronousRunner -
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> >> -- APPARENT DEADLOCK!!! Comp
> >> >> tus:
> >> >> Managed Threads: 3
> >> >> Active Threads: 3
> >> >> Active Tasks:
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
> >> >> Pending Tasks:
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
> >> >>
> >> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
> >> >> Pool thread stack traces:
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> ERROR QueueStorePrefetch - Failed to get message count
> >> >> java.io.IOException: An attempt by a client to checkout a Connection
> >> has
> >> >> timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> >> at
> >> >>
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> >> Caused by: java.sql.SQLException: An attempt by a client to checkout
> a
> >> >> Connection has timed out.
> >> >> at
> >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> >> at
> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> >> ... 20 more
> >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
> >> timed
> >> >> out
> >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> >> ... 22 more
> >> >> WARN TransportConnection - Failed to remove consumer:
> >> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
> >> >> java.lang.RuntimeException
> >> >> o.IOException: An attempt by a client to checkout a Connection has
> >> timed
> >> >> out.
> >> >> java.lang.RuntimeException: java.io.IOException: An attempt by a
> >> client
> >> >> to
> >> >> checkout a Connection has timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> >> at
> >> >>
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> >> Caused by: java.io.IOException: An attempt by a client to checkout a
> >> >> Connection has timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> >> ... 16 more
> >> >> Caused by: java.sql.SQLException: An attempt by a client to checkout
> a
> >> >> Connection has timed out.
> >> >> at
> >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> >> at
> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> >> ... 20 more
> >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
> >> timed
> >> >> out
> >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> >> ... 22 more
> >> >> INFO TransportConnector - Connector openwire Stopped
> >> >>
> >> >>
> >> >> bsnyder wrote:
> >> >> >
> >> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
> >> >> wrote:
> >> >> >>
> >> >> >> Thanks for the response Bruce. :)
> >> >> >>
> >> >> >> To my knowledge we aren't using a connection pooler. We're just
> >> >> >> configuring
> >> >> >> the activemq xml and letting activeMQ do its thing. One of my
> >> >> colleagues
> >> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
> >> >> similar
> >> >> >> to
> >> >> >> a connection pooler or not.
> >> >> >
> >> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
> >> >> > connections available for use by a Java application. IIRC, SNAC
> >> stands
> >> >> > for SQL Native Client which has nothing to do with a JDBC
> connection
> >> >> > pooler.
> >> >> >
> >> >> >> In our scenario during a Sql Server failover it may take 15
> >> minutes.
> >> >> We
> >> >> >> aren't using a high availability cluster. Does a connection
> pooler
> >> >> give
> >> >> >> the
> >> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
> >> >> >
> >> >> > One of the options in a good JDBC connection pooler is the ability
> >> to
> >> >> > test JDBC connections as it hands them out to the Java application.
> >> If
> >> >> > the connection is stale for any reason the JDBC connection pooler
> >> will
> >> >> > evict that connection from the pool and try the next one until it
> >> >> > reaches one that is alive. To my knowledge, some JDBC connection
> >> >> > poolers provide a the ability to configure some attributes like
> >> number
> >> >> > of retries, etc. E.g., here is some information about what c3p0
> >> >> > offers:
> >> >> >
> >> >> >
> http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
> >> >> >
> >> >> > The one big caveat I will mention is that longer outage periods
> such
> >> >> > as the 15 minute outage you mention are not typically handled by a
> >> >> > JDBC connection pooler as they're mainly geared toward handling
> >> >> > momentary outages. As an example, notice that the default value for
> >> >> > acquireRetryDelay in c3p0 is only one second:
> >> >> >
> >> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
> >> >> >
> >> >> > But you could certainly configure this to any value you want. I
> >> >> > encourage you to test this in your environment before deploying to
> >> >> > production.
> >> >> >
> >> >> > Bruce
> >> >> > --
> >> >> > perl -e 'print
> >> >> >
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> >> >> > );'
> >> >> >
> >> >> > Apache ActiveMQ - http://activemq.apache.org/
> >> >> > Apache Camel - http://camel.apache.org/
> >> >> > Apache ServiceMix - http://servicemix.apache.org/
> >> >> >
> >> >> > Blog: http://bruceblog.org/
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://blog.garytully.com
> >> >
> >> > Open Source SOA
> >> > http://FUSESource.com
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source SOA
> > http://FUSESource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22973315.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by magellings <ma...@qg.com>.
This implies that I'd need to modify activemq source though doesn't it?
Do you have an example of how it could be done through configuration?
Gary Tully wrote:
>
> in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
> set a locker instance.
> see:
> http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)
>
> Otherwise, yea, as you describe, derive and inject using spring via the
> XML
> configuration.
>
> hope this helps,
> Gary.
> 2009/4/7 magellings <ma...@qg.com>
>
>>
>> How do you exactly provide your own implementation?
>>
>> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use
>> the
>> DefaultDatabaseLocker:
>>
>> protected DatabaseLocker createDatabaseLocker() throws IOException {
>> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
>> }
>>
>> Can I provide my own implemenation declaratively through configuration
>> provided I write a class deriving from JDBCPersistenceAdaptor of which
>> overrides the createDatabaseLocker class?
>>
>>
>> Gary Tully wrote:
>> >
>> > One thing to note: you can provide your own implementation of the
>> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
>> > keepAlive to be aware of your database backup policy.
>> >
>> > If you go down this route and conceive a better default or optional
>> > strategy, feel free to submit it as a patch.
>> >
>> >
>> >
>> > 2009/3/11 magellings <ma...@qg.com>
>> >
>> >>
>> >> Well it was butt-simple to get working like the c3p0 documentation
>> said.
>> >> I
>> >> upped the delay to 20s, unfortunately the master broker still failed.
>> >>
>> >> I'm kind of wondering how the slave would respond. My guess is it
>> would
>> >> probably time out too.
>> >>
>> >> Any thoughts?
>> >>
>> >> See further below for the errors.
>> >>
>> >>
>> >>
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
>> >> xbean:activemq.xml
>> >> ACTIVEMQ_HOME:
>> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> >> ACTIVEMQ_BASE:
>> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> >> Loading message broker from: xbean:activemq.xml
>> >> INFO DefaultCamelContext - JMX enabled. Using
>> >> InstrumentationLifecycleStrategy.
>> >> INFO MLog - MLog clients using log4j
>> logging.
>> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2
>> [built
>> >> 21-May-2007 15:04:56; debug? true; trace: 10]
>> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
>> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> >> acquireRetryAttemp
>> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
>> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> >> checkoutTimeout -
>> >> > 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName
>> >> ->
>> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
>> >> 2rvy1
>> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
>> false,
>> >> description -> null, driverClass ->
>> >> com.microsoft.sqlserver.jdbc.SQLServerDri
>> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions
>> ->
>> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> idleConnectionTestPerio
>> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$word,
>> >> maxAdministrativeTas
>> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> >> 200,
>> >> maxStatementsPerCon
>> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null,
>> >> properties -> {user=**
>> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
>> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
>> ->
>> >> 0,
>> >> use
>> >> sTraditionalReflectiveProxies -> false ]
>> >> INFO BrokerService - Using Persistence Adapter:
>> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
>> >> acquireIncrement
>> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
>> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
>> >> breakAfterAcquireFailure ->
>> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName ->
>> null,
>> >> connectionTesterClassName ->
>> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
>> >> driverClass -> com.microsoft.sql
>> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
>> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> >> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$wo
>> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime
>> ->
>> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100,
>> maxStatements
>> >> ->
>> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
>> numHelperThreads
>> >> ->
>> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null
>> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
>> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
>> >> unreturnedCon
>> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
>> >> INFO JDBCPersistenceAdapter - Database driver recognized:
>> >> [microsoft_sql_server_2005_jdbc_driver]
>> >> INFO DefaultDatabaseLocker - Attempting to acquire the
>> >> exclusive
>> >> lock to become the Master broker
>> >> INFO DefaultDatabaseLocker - Becoming the master on
>> dataSource:
>> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> >> acquireRe
>> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
>> >> false,
>> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> >> checkout
>> >> Timeout -> 30000, connectionCustomizerClassName -> null,
>> >> connectionTesterClassName ->
>> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
>> >> -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> debugUnreturnedConnectionStackTraces
>> >> ->
>> >> false, description -> null, driverClass ->
>> >> com.microsoft.sqlserver.jdbc.SQL
>> >> ServerDriver, factoryClassLocation -> null,
>> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
>> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$word,
>> >> maxAdminist
>> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> >> 200,
>> >> maxStateme
>> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null,
>> >> properties ->
>> >> {user=******, password=******}, propertyCycle -> 0,
>> >> testConnectionOnCheckin
>> >> -> false, testConnectionOnCheckout -> false,
>> unreturnedConnectionTimeout
>> >> -> 0, usesTraditionalReflectiveProxies -> false ]
>> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message
>> Broker
>> >> (master) is starting
>> >> INFO BrokerService - For help or more information
>> >> please
>> >> see: http://activemq.apache.org/
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> tcp://QG076619:61616
>> >> INFO TransportConnector - Connector openwire Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> ssl://QG076619:61617
>> >> INFO TransportConnector - Connector ssl Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> stomp://QG076619:61613
>> >> INFO TransportConnector - Connector stomp Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> xmpp://QG076619:61222
>> >> INFO TransportConnector - Connector xmpp Started
>> >> INFO BrokerService - ActiveMQ JMS Message Broker
>> >> (master,
>> >> ID:QG076619-3632-1236812198396-0:0) started
>> >> INFO log - Logging to
>> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
>> >> org.mortbay.log.Slf4jLog
>> >> INFO log - jetty-6.1.9
>> >> INFO WebConsoleStarter - ActiveMQ WebConsole
>> initialized.
>> >> INFO /admin - Initializing Spring
>> >> FrameworkServlet
>> >> 'dispatcher'
>> >> INFO log - ActiveMQ Console at
>> >> http://0.0.0.0:8162/admin
>> >> INFO log - ActiveMQ Web Demos at
>> >> http://0.0.0.0:8162/demo
>> >> INFO log - RESTful file access application
>> at
>> >> http://0.0.0.0:8162/fileserver
>> >> INFO log - Started
>> >> SelectChannelConnector@0.0.0.0:8162
>> >> WARN BrokerRegistry - Broker localhost not started so
>> >> using
>> >> master instead
>> >> INFO TransportConnector - Connector vm://localhost
>> Started
>> >>
>> >>
>> >> Then BOOM! Although behavior is different. Now it takes longer to
>> stop.
>> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
>> >> shutdown."
>> >> If you look closely you can see activeMQ is timing out.
>> >>
>> >>
>> >>
>> >>
>> >> ERROR DefaultDatabaseLocker - Failed to update database lock:
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> pe
>> >> et write error
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> >> peer:
>> >> socket write error
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown
>> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
>> >> Source)
>> >> at
>> com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
>> Source)
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
>> >> Source)
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> >> at
>> >>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >> at
>> >>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> >> at
>> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> at java.lang.Thread.run(Thread.java:619)
>> >> INFO GooGooStatementCache - Problem with checked-in
>> Statement,
>> >> discarding.
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
>> >> closed.
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> >> at
>> >>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >> at
>> >>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> >> at
>> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> at java.lang.Thread.run(Thread.java:619)
>> >> INFO JDBCPersistenceAdapter - No longer able to keep the
>> >> exclusive
>> >> lock so giving up being a master
>> >> INFO BrokerService - ActiveMQ Message Broker
>> (master,
>> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> WARN ThreadPoolAsynchronousRunner -
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> >> -- APPARENT DEADLOCK!!! Crea
>> >> rgency threads for unassigned pending tasks!
>> >> WARN ThreadPoolAsynchronousRunner -
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> >> -- APPARENT DEADLOCK!!! Comp
>> >> tus:
>> >> Managed Threads: 3
>> >> Active Threads: 3
>> >> Active Tasks:
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
>> >> Pending Tasks:
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
>> >> Pool thread stack traces:
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> ERROR QueueStorePrefetch - Failed to get message count
>> >> java.io.IOException: An attempt by a client to checkout a Connection
>> has
>> >> timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> >> at
>> >>
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> >> ... 20 more
>> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
>> timed
>> >> out
>> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> >> ResourcePool@d964af -- timeout at awaitAvailable()
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> >> ... 22 more
>> >> WARN TransportConnection - Failed to remove consumer:
>> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
>> >> java.lang.RuntimeException
>> >> o.IOException: An attempt by a client to checkout a Connection has
>> timed
>> >> out.
>> >> java.lang.RuntimeException: java.io.IOException: An attempt by a
>> client
>> >> to
>> >> checkout a Connection has timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> >> at
>> >>
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> >> Caused by: java.io.IOException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> >> ... 16 more
>> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> >> ... 20 more
>> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
>> timed
>> >> out
>> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> >> ResourcePool@d964af -- timeout at awaitAvailable()
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> >> ... 22 more
>> >> INFO TransportConnector - Connector openwire Stopped
>> >>
>> >>
>> >> bsnyder wrote:
>> >> >
>> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
>> >> wrote:
>> >> >>
>> >> >> Thanks for the response Bruce. :)
>> >> >>
>> >> >> To my knowledge we aren't using a connection pooler. We're just
>> >> >> configuring
>> >> >> the activemq xml and letting activeMQ do its thing. One of my
>> >> colleagues
>> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
>> >> similar
>> >> >> to
>> >> >> a connection pooler or not.
>> >> >
>> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
>> >> > connections available for use by a Java application. IIRC, SNAC
>> stands
>> >> > for SQL Native Client which has nothing to do with a JDBC connection
>> >> > pooler.
>> >> >
>> >> >> In our scenario during a Sql Server failover it may take 15
>> minutes.
>> >> We
>> >> >> aren't using a high availability cluster. Does a connection pooler
>> >> give
>> >> >> the
>> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
>> >> >
>> >> > One of the options in a good JDBC connection pooler is the ability
>> to
>> >> > test JDBC connections as it hands them out to the Java application.
>> If
>> >> > the connection is stale for any reason the JDBC connection pooler
>> will
>> >> > evict that connection from the pool and try the next one until it
>> >> > reaches one that is alive. To my knowledge, some JDBC connection
>> >> > poolers provide a the ability to configure some attributes like
>> number
>> >> > of retries, etc. E.g., here is some information about what c3p0
>> >> > offers:
>> >> >
>> >> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
>> >> >
>> >> > The one big caveat I will mention is that longer outage periods such
>> >> > as the 15 minute outage you mention are not typically handled by a
>> >> > JDBC connection pooler as they're mainly geared toward handling
>> >> > momentary outages. As an example, notice that the default value for
>> >> > acquireRetryDelay in c3p0 is only one second:
>> >> >
>> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
>> >> >
>> >> > But you could certainly configure this to any value you want. I
>> >> > encourage you to test this in your environment before deploying to
>> >> > production.
>> >> >
>> >> > Bruce
>> >> > --
>> >> > perl -e 'print
>> >> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> >> > );'
>> >> >
>> >> > Apache ActiveMQ - http://activemq.apache.org/
>> >> > Apache Camel - http://camel.apache.org/
>> >> > Apache ServiceMix - http://servicemix.apache.org/
>> >> >
>> >> > Blog: http://bruceblog.org/
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source SOA
>> > http://FUSESource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22973315.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Gary Tully <ga...@gmail.com>.
in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
set a locker instance.
see:
http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)
Otherwise, yea, as you describe, derive and inject using spring via the XML
configuration.
hope this helps,
Gary.
2009/4/7 magellings <ma...@qg.com>
>
> How do you exactly provide your own implementation?
>
> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use the
> DefaultDatabaseLocker:
>
> protected DatabaseLocker createDatabaseLocker() throws IOException {
> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
> }
>
> Can I provide my own implemenation declaratively through configuration
> provided I write a class deriving from JDBCPersistenceAdaptor of which
> overrides the createDatabaseLocker class?
>
>
> Gary Tully wrote:
> >
> > One thing to note: you can provide your own implementation of the
> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> > keepAlive to be aware of your database backup policy.
> >
> > If you go down this route and conceive a better default or optional
> > strategy, feel free to submit it as a patch.
> >
> >
> >
> > 2009/3/11 magellings <ma...@qg.com>
> >
> >>
> >> Well it was butt-simple to get working like the c3p0 documentation said.
> >> I
> >> upped the delay to 20s, unfortunately the master broker still failed.
> >>
> >> I'm kind of wondering how the slave would respond. My guess is it would
> >> probably time out too.
> >>
> >> Any thoughts?
> >>
> >> See further below for the errors.
> >>
> >>
> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
> >> xbean:activemq.xml
> >> ACTIVEMQ_HOME:
> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> ACTIVEMQ_BASE:
> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> Loading message broker from: xbean:activemq.xml
> >> INFO DefaultCamelContext - JMX enabled. Using
> >> InstrumentationLifecycleStrategy.
> >> INFO MLog - MLog clients using log4j logging.
> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
> >> 21-May-2007 15:04:56; debug? true; trace: 10]
> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> acquireRetryAttemp
> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> checkoutTimeout -
> >> > 30000, connectionCustomizerClassName -> null,
> connectionTesterClassName
> >> ->
> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
> >> 2rvy1
> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
> >> description -> null, driverClass ->
> >> com.microsoft.sqlserver.jdbc.SQLServerDri
> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> idleConnectionTestPerio
> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> maxAdministrativeTas
> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> >> 200,
> >> maxStatementsPerCon
> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> >> properties -> {user=**
> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout ->
> >> 0,
> >> use
> >> sTraditionalReflectiveProxies -> false ]
> >> INFO BrokerService - Using Persistence Adapter:
> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
> >> acquireIncrement
> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
> >> breakAfterAcquireFailure ->
> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName ->
> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
> >> driverClass -> com.microsoft.sql
> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$wo
> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime
> ->
> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
> numHelperThreads
> >> ->
> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> null
> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
> >> unreturnedCon
> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
> >> INFO JDBCPersistenceAdapter - Database driver recognized:
> >> [microsoft_sql_server_2005_jdbc_driver]
> >> INFO DefaultDatabaseLocker - Attempting to acquire the
> >> exclusive
> >> lock to become the Master broker
> >> INFO DefaultDatabaseLocker - Becoming the master on
> dataSource:
> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> acquireRe
> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
> >> false,
> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> checkout
> >> Timeout -> 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName ->
> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
> >> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces
> >> ->
> >> false, description -> null, driverClass ->
> >> com.microsoft.sqlserver.jdbc.SQL
> >> ServerDriver, factoryClassLocation -> null,
> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> maxAdminist
> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> >> 200,
> >> maxStateme
> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> >> properties ->
> >> {user=******, password=******}, propertyCycle -> 0,
> >> testConnectionOnCheckin
> >> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
> >> -> 0, usesTraditionalReflectiveProxies -> false ]
> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
> >> (master) is starting
> >> INFO BrokerService - For help or more information
> >> please
> >> see: http://activemq.apache.org/
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> tcp://QG076619:61616
> >> INFO TransportConnector - Connector openwire Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> ssl://QG076619:61617
> >> INFO TransportConnector - Connector ssl Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> stomp://QG076619:61613
> >> INFO TransportConnector - Connector stomp Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> xmpp://QG076619:61222
> >> INFO TransportConnector - Connector xmpp Started
> >> INFO BrokerService - ActiveMQ JMS Message Broker
> >> (master,
> >> ID:QG076619-3632-1236812198396-0:0) started
> >> INFO log - Logging to
> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
> >> org.mortbay.log.Slf4jLog
> >> INFO log - jetty-6.1.9
> >> INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
> >> INFO /admin - Initializing Spring
> >> FrameworkServlet
> >> 'dispatcher'
> >> INFO log - ActiveMQ Console at
> >> http://0.0.0.0:8162/admin
> >> INFO log - ActiveMQ Web Demos at
> >> http://0.0.0.0:8162/demo
> >> INFO log - RESTful file access application
> at
> >> http://0.0.0.0:8162/fileserver
> >> INFO log - Started
> >> SelectChannelConnector@0.0.0.0:8162
> >> WARN BrokerRegistry - Broker localhost not started so
> >> using
> >> master instead
> >> INFO TransportConnector - Connector vm://localhost Started
> >>
> >>
> >> Then BOOM! Although behavior is different. Now it takes longer to
> stop.
> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
> >> shutdown."
> >> If you look closely you can see activeMQ is timing out.
> >>
> >>
> >>
> >>
> >> ERROR DefaultDatabaseLocker - Failed to update database lock:
> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
> >> et write error
> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> peer:
> >> socket write error
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
> >> Source)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> at
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> at
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >> INFO GooGooStatementCache - Problem with checked-in
> Statement,
> >> discarding.
> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
> >> closed.
> >> at
> >>
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
> >> Source)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> at
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> at
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >> INFO JDBCPersistenceAdapter - No longer able to keep the
> >> exclusive
> >> lock so giving up being a master
> >> INFO BrokerService - ActiveMQ Message Broker (master,
> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> WARN ThreadPoolAsynchronousRunner -
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> -- APPARENT DEADLOCK!!! Crea
> >> rgency threads for unassigned pending tasks!
> >> WARN ThreadPoolAsynchronousRunner -
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> -- APPARENT DEADLOCK!!! Comp
> >> tus:
> >> Managed Threads: 3
> >> Active Threads: 3
> >> Active Tasks:
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
> >> Pending Tasks:
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
> >> Pool thread stack traces:
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> ERROR QueueStorePrefetch - Failed to get message count
> >> java.io.IOException: An attempt by a client to checkout a Connection has
> >> timed out.
> >> at
> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> at
> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> at
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> at
> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> ... 20 more
> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
> >> out
> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> ... 22 more
> >> WARN TransportConnection - Failed to remove consumer:
> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
> >> java.lang.RuntimeException
> >> o.IOException: An attempt by a client to checkout a Connection has timed
> >> out.
> >> java.lang.RuntimeException: java.io.IOException: An attempt by a client
> >> to
> >> checkout a Connection has timed out.
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> at
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> at
> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> Caused by: java.io.IOException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at
> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> at
> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> ... 16 more
> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> ... 20 more
> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
> >> out
> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> ... 22 more
> >> INFO TransportConnector - Connector openwire Stopped
> >>
> >>
> >> bsnyder wrote:
> >> >
> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
> >> wrote:
> >> >>
> >> >> Thanks for the response Bruce. :)
> >> >>
> >> >> To my knowledge we aren't using a connection pooler. We're just
> >> >> configuring
> >> >> the activemq xml and letting activeMQ do its thing. One of my
> >> colleagues
> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
> >> similar
> >> >> to
> >> >> a connection pooler or not.
> >> >
> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
> >> > connections available for use by a Java application. IIRC, SNAC stands
> >> > for SQL Native Client which has nothing to do with a JDBC connection
> >> > pooler.
> >> >
> >> >> In our scenario during a Sql Server failover it may take 15 minutes.
> >> We
> >> >> aren't using a high availability cluster. Does a connection pooler
> >> give
> >> >> the
> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
> >> >
> >> > One of the options in a good JDBC connection pooler is the ability to
> >> > test JDBC connections as it hands them out to the Java application. If
> >> > the connection is stale for any reason the JDBC connection pooler will
> >> > evict that connection from the pool and try the next one until it
> >> > reaches one that is alive. To my knowledge, some JDBC connection
> >> > poolers provide a the ability to configure some attributes like number
> >> > of retries, etc. E.g., here is some information about what c3p0
> >> > offers:
> >> >
> >> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
> >> >
> >> > The one big caveat I will mention is that longer outage periods such
> >> > as the 15 minute outage you mention are not typically handled by a
> >> > JDBC connection pooler as they're mainly geared toward handling
> >> > momentary outages. As an example, notice that the default value for
> >> > acquireRetryDelay in c3p0 is only one second:
> >> >
> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
> >> >
> >> > But you could certainly configure this to any value you want. I
> >> > encourage you to test this in your environment before deploying to
> >> > production.
> >> >
> >> > Bruce
> >> > --
> >> > perl -e 'print
> >> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> >> > );'
> >> >
> >> > Apache ActiveMQ - http://activemq.apache.org/
> >> > Apache Camel - http://camel.apache.org/
> >> > Apache ServiceMix - http://servicemix.apache.org/
> >> >
> >> > Blog: http://bruceblog.org/
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source SOA
> > http://FUSESource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com