You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by magellings <ma...@qg.com> on 2009/03/10 19:15:52 UTC
Is there JDBC master/slave retry logic if database goes down?
Hello.
Does anyone know if there is logic in activeMQ to retry connecting to a
database if JDBC master/slave is configured? Perhaps there is some
configuration I'm missing. Right now we log ship from the active Sql Server
database to the passive Sql Server database. During maintenance it would be
nice to fail to the passive database without impacting the ActiveMQ broker
and requiring a restart. When I tested I just stopped the sql server
windows service and the active ActiveMQ broker shut down requiring a manual
startup.
Current JDBC master/slave configuration snippets:
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#mssql-ds"/>
</persistenceAdapter>
...
<bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName"
value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
<property name="url"
value="jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$word"/>
<property name="username" value="activemq"/>
<property name="password" value="pa$$word"/>
</bean>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22440338.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Gary Tully <ga...@gmail.com>.
<bean id='dbLocker" class="org.example.newAndBetter.DataBaseLocker" />
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#mssql-ds"
dataBaseLocker="#dbLocker"/>
</persistenceAdapter>
You have got to implement the DataBaseLocker interface in code though or
extend the current default to make it behave as you need.
2009/4/9 magellings <ma...@qg.com>
>
> This implies that I'd need to modify activemq source though doesn't it?
>
> Do you have an example of how it could be done through configuration?
>
>
> Gary Tully wrote:
> >
> > in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
> > set a locker instance.
> > see:
> >
> http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)<http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker%28org.apache.activemq.store.jdbc.DatabaseLocker%29>
> >
> > Otherwise, yea, as you describe, derive and inject using spring via the
> > XML
> > configuration.
> >
> > hope this helps,
> > Gary.
> > 2009/4/7 magellings <ma...@qg.com>
> >
> >>
> >> How do you exactly provide your own implementation?
> >>
> >> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use
> >> the
> >> DefaultDatabaseLocker:
> >>
> >> protected DatabaseLocker createDatabaseLocker() throws IOException {
> >> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
> >> }
> >>
> >> Can I provide my own implemenation declaratively through configuration
> >> provided I write a class deriving from JDBCPersistenceAdaptor of which
> >> overrides the createDatabaseLocker class?
> >>
> >>
> >> Gary Tully wrote:
> >> >
> >> > One thing to note: you can provide your own implementation of the
> >> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> >> > keepAlive to be aware of your database backup policy.
> >> >
> >> > If you go down this route and conceive a better default or optional
> >> > strategy, feel free to submit it as a patch.
> >> >
> >> >
> >> >
> >> > 2009/3/11 magellings <ma...@qg.com>
> >> >
> >> >>
> >> >> Well it was butt-simple to get working like the c3p0 documentation
> >> said.
> >> >> I
> >> >> upped the delay to 20s, unfortunately the master broker still failed.
> >> >>
> >> >> I'm kind of wondering how the slave would respond. My guess is it
> >> would
> >> >> probably time out too.
> >> >>
> >> >> Any thoughts?
> >> >>
> >> >> See further below for the errors.
> >> >>
> >> >>
> >> >>
> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
> >> >> xbean:activemq.xml
> >> >> ACTIVEMQ_HOME:
> >> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> >> ACTIVEMQ_BASE:
> >> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> >> Loading message broker from: xbean:activemq.xml
> >> >> INFO DefaultCamelContext - JMX enabled. Using
> >> >> InstrumentationLifecycleStrategy.
> >> >> INFO MLog - MLog clients using log4j
> >> logging.
> >> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2
> >> [built
> >> >> 21-May-2007 15:04:56; debug? true; trace: 10]
> >> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
> >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> >> acquireRetryAttemp
> >> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> >> checkoutTimeout -
> >> >> > 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName
> >> >> ->
> >> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
> >> >> 2rvy1
> >> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
> >> false,
> >> >> description -> null, driverClass ->
> >> >> com.microsoft.sqlserver.jdbc.SQLServerDri
> >> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions
> >> ->
> >> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> idleConnectionTestPerio
> >> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> >> maxAdministrativeTas
> >> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> >> 200,
> >> >> maxStatementsPerCon
> >> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null,
> >> >> properties -> {user=**
> >> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin
> ->
> >> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
> >> ->
> >> >> 0,
> >> >> use
> >> >> sTraditionalReflectiveProxies -> false ]
> >> >> INFO BrokerService - Using Persistence Adapter:
> >> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
> >> >> acquireIncrement
> >> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
> >> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
> >> >> breakAfterAcquireFailure ->
> >> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName ->
> >> null,
> >> >> connectionTesterClassName ->
> >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> >> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
> >> >> driverClass -> com.microsoft.sql
> >> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
> >> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> >> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$wo
> >> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0,
> maxIdleTime
> >> ->
> >> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100,
> >> maxStatements
> >> >> ->
> >> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
> >> numHelperThreads
> >> >> ->
> >> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null
> >> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
> >> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
> >> >> unreturnedCon
> >> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
> >> >> INFO JDBCPersistenceAdapter - Database driver recognized:
> >> >> [microsoft_sql_server_2005_jdbc_driver]
> >> >> INFO DefaultDatabaseLocker - Attempting to acquire the
> >> >> exclusive
> >> >> lock to become the Master broker
> >> >> INFO DefaultDatabaseLocker - Becoming the master on
> >> dataSource:
> >> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> >> acquireRe
> >> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
> >> >> false,
> >> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> >> checkout
> >> >> Timeout -> 30000, connectionCustomizerClassName -> null,
> >> >> connectionTesterClassName ->
> >> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
> >> >> -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> debugUnreturnedConnectionStackTraces
> >> >> ->
> >> >> false, description -> null, driverClass ->
> >> >> com.microsoft.sqlserver.jdbc.SQL
> >> >> ServerDriver, factoryClassLocation -> null,
> >> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
> >> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> >> maxAdminist
> >> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> >> 200,
> >> >> maxStateme
> >> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> >> null,
> >> >> properties ->
> >> >> {user=******, password=******}, propertyCycle -> 0,
> >> >> testConnectionOnCheckin
> >> >> -> false, testConnectionOnCheckout -> false,
> >> unreturnedConnectionTimeout
> >> >> -> 0, usesTraditionalReflectiveProxies -> false ]
> >> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message
> >> Broker
> >> >> (master) is starting
> >> >> INFO BrokerService - For help or more information
> >> >> please
> >> >> see: http://activemq.apache.org/
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> tcp://QG076619:61616
> >> >> INFO TransportConnector - Connector openwire Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> ssl://QG076619:61617
> >> >> INFO TransportConnector - Connector ssl Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> stomp://QG076619:61613
> >> >> INFO TransportConnector - Connector stomp Started
> >> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> >> xmpp://QG076619:61222
> >> >> INFO TransportConnector - Connector xmpp Started
> >> >> INFO BrokerService - ActiveMQ JMS Message Broker
> >> >> (master,
> >> >> ID:QG076619-3632-1236812198396-0:0) started
> >> >> INFO log - Logging to
> >> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
> >> >> org.mortbay.log.Slf4jLog
> >> >> INFO log - jetty-6.1.9
> >> >> INFO WebConsoleStarter - ActiveMQ WebConsole
> >> initialized.
> >> >> INFO /admin - Initializing Spring
> >> >> FrameworkServlet
> >> >> 'dispatcher'
> >> >> INFO log - ActiveMQ Console at
> >> >> http://0.0.0.0:8162/admin
> >> >> INFO log - ActiveMQ Web Demos at
> >> >> http://0.0.0.0:8162/demo
> >> >> INFO log - RESTful file access
> application
> >> at
> >> >> http://0.0.0.0:8162/fileserver
> >> >> INFO log - Started
> >> >> SelectChannelConnector@0.0.0.0:8162
> >> >> WARN BrokerRegistry - Broker localhost not started
> so
> >> >> using
> >> >> master instead
> >> >> INFO TransportConnector - Connector vm://localhost
> >> Started
> >> >>
> >> >>
> >> >> Then BOOM! Although behavior is different. Now it takes longer to
> >> stop.
> >> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
> >> >> shutdown."
> >> >> If you look closely you can see activeMQ is timing out.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ERROR DefaultDatabaseLocker - Failed to update database
> lock:
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> pe
> >> >> et write error
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> >> peer:
> >> >> socket write error
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown
> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown
> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
> >> >> Source)
> >> >> at
> >> com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
> >> >> Source)
> >> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
> >> Source)
> >> >> at
> >> >>
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
> >> >> Source)
> >> >> at
> >> >>
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> >> at
> >> >>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> >> at
> >> >>
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> >> at
> >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> INFO GooGooStatementCache - Problem with checked-in
> >> Statement,
> >> >> discarding.
> >> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
> >> >> closed.
> >> >> at
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
> >> >> Source)
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
> >> >> Source)
> >> >> at
> >> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
> >> >> Source)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> >> at
> >> >>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> >> at
> >> >>
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> >> at
> >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >> at
> >> >>
> >> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >> at java.lang.Thread.run(Thread.java:619)
> >> >> INFO JDBCPersistenceAdapter - No longer able to keep the
> >> >> exclusive
> >> >> lock so giving up being a master
> >> >> INFO BrokerService - ActiveMQ Message Broker
> >> (master,
> >> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> WARN ThreadPoolAsynchronousRunner -
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> >> -- APPARENT DEADLOCK!!! Crea
> >> >> rgency threads for unassigned pending tasks!
> >> >> WARN ThreadPoolAsynchronousRunner -
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> >> -- APPARENT DEADLOCK!!! Comp
> >> >> tus:
> >> >> Managed Threads: 3
> >> >> Active Threads: 3
> >> >> Active Tasks:
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
> >> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
> >> >> Pending Tasks:
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
> >> >>
> >> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
> >> >>
> >> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
> >> >> Pool thread stack traces:
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
> >> >> java.lang.Thread.sleep(Native Method)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >> >>
> >> >>
> >> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >> >>
> >> >>
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> INFO TransportConnection - The connection to '/
> >> 10.0.1.5:3333'
> >> >> is
> >> >> taking a long time to shutdown.
> >> >> ERROR QueueStorePrefetch - Failed to get message count
> >> >> java.io.IOException: An attempt by a client to checkout a Connection
> >> has
> >> >> timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> >> at
> >> >>
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> >> Caused by: java.sql.SQLException: An attempt by a client to checkout
> a
> >> >> Connection has timed out.
> >> >> at
> >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> >> at
> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> >> ... 20 more
> >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
> >> timed
> >> >> out
> >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> >> ... 22 more
> >> >> WARN TransportConnection - Failed to remove consumer:
> >> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
> >> >> java.lang.RuntimeException
> >> >> o.IOException: An attempt by a client to checkout a Connection has
> >> timed
> >> >> out.
> >> >> java.lang.RuntimeException: java.io.IOException: An attempt by a
> >> client
> >> >> to
> >> >> checkout a Connection has timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> >> at
> >> >>
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> >> Caused by: java.io.IOException: An attempt by a client to checkout a
> >> >> Connection has timed out.
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> >> ... 16 more
> >> >> Caused by: java.sql.SQLException: An attempt by a client to checkout
> a
> >> >> Connection has timed out.
> >> >> at
> >> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> >> at
> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> >> at
> >> >>
> >> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> >> ... 20 more
> >> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
> >> timed
> >> >> out
> >> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> >> at
> >> >>
> >> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> >> ... 22 more
> >> >> INFO TransportConnector - Connector openwire Stopped
> >> >>
> >> >>
> >> >> bsnyder wrote:
> >> >> >
> >> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
> >> >> wrote:
> >> >> >>
> >> >> >> Thanks for the response Bruce. :)
> >> >> >>
> >> >> >> To my knowledge we aren't using a connection pooler. We're just
> >> >> >> configuring
> >> >> >> the activemq xml and letting activeMQ do its thing. One of my
> >> >> colleagues
> >> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
> >> >> similar
> >> >> >> to
> >> >> >> a connection pooler or not.
> >> >> >
> >> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
> >> >> > connections available for use by a Java application. IIRC, SNAC
> >> stands
> >> >> > for SQL Native Client which has nothing to do with a JDBC
> connection
> >> >> > pooler.
> >> >> >
> >> >> >> In our scenario during a Sql Server failover it may take 15
> >> minutes.
> >> >> We
> >> >> >> aren't using a high availability cluster. Does a connection
> pooler
> >> >> give
> >> >> >> the
> >> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
> >> >> >
> >> >> > One of the options in a good JDBC connection pooler is the ability
> >> to
> >> >> > test JDBC connections as it hands them out to the Java application.
> >> If
> >> >> > the connection is stale for any reason the JDBC connection pooler
> >> will
> >> >> > evict that connection from the pool and try the next one until it
> >> >> > reaches one that is alive. To my knowledge, some JDBC connection
> >> >> > poolers provide a the ability to configure some attributes like
> >> number
> >> >> > of retries, etc. E.g., here is some information about what c3p0
> >> >> > offers:
> >> >> >
> >> >> >
> http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
> >> >> >
> >> >> > The one big caveat I will mention is that longer outage periods
> such
> >> >> > as the 15 minute outage you mention are not typically handled by a
> >> >> > JDBC connection pooler as they're mainly geared toward handling
> >> >> > momentary outages. As an example, notice that the default value for
> >> >> > acquireRetryDelay in c3p0 is only one second:
> >> >> >
> >> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
> >> >> >
> >> >> > But you could certainly configure this to any value you want. I
> >> >> > encourage you to test this in your environment before deploying to
> >> >> > production.
> >> >> >
> >> >> > Bruce
> >> >> > --
> >> >> > perl -e 'print
> >> >> >
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> >> >> > );'
> >> >> >
> >> >> > Apache ActiveMQ - http://activemq.apache.org/
> >> >> > Apache Camel - http://camel.apache.org/
> >> >> > Apache ServiceMix - http://servicemix.apache.org/
> >> >> >
> >> >> > Blog: http://bruceblog.org/
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
> >> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > http://blog.garytully.com
> >> >
> >> > Open Source SOA
> >> > http://FUSESource.com
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source SOA
> > http://FUSESource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22973315.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by magellings <ma...@qg.com>.
This implies that I'd need to modify activemq source though doesn't it?
Do you have an example of how it could be done through configuration?
Gary Tully wrote:
>
> in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
> set a locker instance.
> see:
> http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)
>
> Otherwise, yea, as you describe, derive and inject using spring via the
> XML
> configuration.
>
> hope this helps,
> Gary.
> 2009/4/7 magellings <ma...@qg.com>
>
>>
>> How do you exactly provide your own implementation?
>>
>> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use
>> the
>> DefaultDatabaseLocker:
>>
>> protected DatabaseLocker createDatabaseLocker() throws IOException {
>> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
>> }
>>
>> Can I provide my own implemenation declaratively through configuration
>> provided I write a class deriving from JDBCPersistenceAdaptor of which
>> overrides the createDatabaseLocker class?
>>
>>
>> Gary Tully wrote:
>> >
>> > One thing to note: you can provide your own implementation of the
>> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
>> > keepAlive to be aware of your database backup policy.
>> >
>> > If you go down this route and conceive a better default or optional
>> > strategy, feel free to submit it as a patch.
>> >
>> >
>> >
>> > 2009/3/11 magellings <ma...@qg.com>
>> >
>> >>
>> >> Well it was butt-simple to get working like the c3p0 documentation
>> said.
>> >> I
>> >> upped the delay to 20s, unfortunately the master broker still failed.
>> >>
>> >> I'm kind of wondering how the slave would respond. My guess is it
>> would
>> >> probably time out too.
>> >>
>> >> Any thoughts?
>> >>
>> >> See further below for the errors.
>> >>
>> >>
>> >>
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
>> >> xbean:activemq.xml
>> >> ACTIVEMQ_HOME:
>> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> >> ACTIVEMQ_BASE:
>> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> >> Loading message broker from: xbean:activemq.xml
>> >> INFO DefaultCamelContext - JMX enabled. Using
>> >> InstrumentationLifecycleStrategy.
>> >> INFO MLog - MLog clients using log4j
>> logging.
>> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2
>> [built
>> >> 21-May-2007 15:04:56; debug? true; trace: 10]
>> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
>> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> >> acquireRetryAttemp
>> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
>> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> >> checkoutTimeout -
>> >> > 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName
>> >> ->
>> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
>> >> 2rvy1
>> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
>> false,
>> >> description -> null, driverClass ->
>> >> com.microsoft.sqlserver.jdbc.SQLServerDri
>> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions
>> ->
>> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> idleConnectionTestPerio
>> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$word,
>> >> maxAdministrativeTas
>> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> >> 200,
>> >> maxStatementsPerCon
>> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null,
>> >> properties -> {user=**
>> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
>> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
>> ->
>> >> 0,
>> >> use
>> >> sTraditionalReflectiveProxies -> false ]
>> >> INFO BrokerService - Using Persistence Adapter:
>> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
>> >> acquireIncrement
>> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
>> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
>> >> breakAfterAcquireFailure ->
>> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName ->
>> null,
>> >> connectionTesterClassName ->
>> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
>> >> driverClass -> com.microsoft.sql
>> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
>> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> >> 2rvy117z1wqpy2hxzkebp|18105e8,
>> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$wo
>> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime
>> ->
>> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100,
>> maxStatements
>> >> ->
>> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
>> numHelperThreads
>> >> ->
>> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null
>> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
>> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
>> >> unreturnedCon
>> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
>> >> INFO JDBCPersistenceAdapter - Database driver recognized:
>> >> [microsoft_sql_server_2005_jdbc_driver]
>> >> INFO DefaultDatabaseLocker - Attempting to acquire the
>> >> exclusive
>> >> lock to become the Master broker
>> >> INFO DefaultDatabaseLocker - Becoming the master on
>> dataSource:
>> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> >> acquireRe
>> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
>> >> false,
>> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> >> checkout
>> >> Timeout -> 30000, connectionCustomizerClassName -> null,
>> >> connectionTesterClassName ->
>> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
>> >> -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> debugUnreturnedConnectionStackTraces
>> >> ->
>> >> false, description -> null, driverClass ->
>> >> com.microsoft.sqlserver.jdbc.SQL
>> >> ServerDriver, factoryClassLocation -> null,
>> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
>> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> >> jdbc:sqlserver://QG076619.qg.com:1433
>> >> ;databaseName=activedb;user=sa;password=pa$$word,
>> >> maxAdminist
>> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> >> 200,
>> >> maxStateme
>> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
>> null,
>> >> properties ->
>> >> {user=******, password=******}, propertyCycle -> 0,
>> >> testConnectionOnCheckin
>> >> -> false, testConnectionOnCheckout -> false,
>> unreturnedConnectionTimeout
>> >> -> 0, usesTraditionalReflectiveProxies -> false ]
>> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message
>> Broker
>> >> (master) is starting
>> >> INFO BrokerService - For help or more information
>> >> please
>> >> see: http://activemq.apache.org/
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> tcp://QG076619:61616
>> >> INFO TransportConnector - Connector openwire Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> ssl://QG076619:61617
>> >> INFO TransportConnector - Connector ssl Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> stomp://QG076619:61613
>> >> INFO TransportConnector - Connector stomp Started
>> >> INFO TransportServerThreadSupport - Listening for connections at:
>> >> xmpp://QG076619:61222
>> >> INFO TransportConnector - Connector xmpp Started
>> >> INFO BrokerService - ActiveMQ JMS Message Broker
>> >> (master,
>> >> ID:QG076619-3632-1236812198396-0:0) started
>> >> INFO log - Logging to
>> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
>> >> org.mortbay.log.Slf4jLog
>> >> INFO log - jetty-6.1.9
>> >> INFO WebConsoleStarter - ActiveMQ WebConsole
>> initialized.
>> >> INFO /admin - Initializing Spring
>> >> FrameworkServlet
>> >> 'dispatcher'
>> >> INFO log - ActiveMQ Console at
>> >> http://0.0.0.0:8162/admin
>> >> INFO log - ActiveMQ Web Demos at
>> >> http://0.0.0.0:8162/demo
>> >> INFO log - RESTful file access application
>> at
>> >> http://0.0.0.0:8162/fileserver
>> >> INFO log - Started
>> >> SelectChannelConnector@0.0.0.0:8162
>> >> WARN BrokerRegistry - Broker localhost not started so
>> >> using
>> >> master instead
>> >> INFO TransportConnector - Connector vm://localhost
>> Started
>> >>
>> >>
>> >> Then BOOM! Although behavior is different. Now it takes longer to
>> stop.
>> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
>> >> shutdown."
>> >> If you look closely you can see activeMQ is timing out.
>> >>
>> >>
>> >>
>> >>
>> >> ERROR DefaultDatabaseLocker - Failed to update database lock:
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> pe
>> >> et write error
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> >> peer:
>> >> socket write error
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown
>> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
>> >> Source)
>> >> at
>> com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
>> >> Source)
>> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
>> Source)
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
>> >> Source)
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> >> at
>> >>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >> at
>> >>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> >> at
>> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> at java.lang.Thread.run(Thread.java:619)
>> >> INFO GooGooStatementCache - Problem with checked-in
>> Statement,
>> >> discarding.
>> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
>> >> closed.
>> >> at
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
>> >> Source)
>> >> at
>> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
>> >> Source)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> >> at
>> >>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >> at
>> >>
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> >> at
>> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >> at
>> >>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >> at java.lang.Thread.run(Thread.java:619)
>> >> INFO JDBCPersistenceAdapter - No longer able to keep the
>> >> exclusive
>> >> lock so giving up being a master
>> >> INFO BrokerService - ActiveMQ Message Broker
>> (master,
>> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> WARN ThreadPoolAsynchronousRunner -
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> >> -- APPARENT DEADLOCK!!! Crea
>> >> rgency threads for unassigned pending tasks!
>> >> WARN ThreadPoolAsynchronousRunner -
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> >> -- APPARENT DEADLOCK!!! Comp
>> >> tus:
>> >> Managed Threads: 3
>> >> Active Threads: 3
>> >> Active Tasks:
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
>> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
>> >> Pending Tasks:
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
>> >>
>> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
>> >>
>> >>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
>> >> Pool thread stack traces:
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
>> >> java.lang.Thread.sleep(Native Method)
>> >>
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>> >>
>> >>
>> >>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>> >>
>> >>
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> INFO TransportConnection - The connection to '/
>> 10.0.1.5:3333'
>> >> is
>> >> taking a long time to shutdown.
>> >> ERROR QueueStorePrefetch - Failed to get message count
>> >> java.io.IOException: An attempt by a client to checkout a Connection
>> has
>> >> timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> >> at
>> >>
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> >> ... 20 more
>> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
>> timed
>> >> out
>> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> >> ResourcePool@d964af -- timeout at awaitAvailable()
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> >> ... 22 more
>> >> WARN TransportConnection - Failed to remove consumer:
>> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
>> >> java.lang.RuntimeException
>> >> o.IOException: An attempt by a client to checkout a Connection has
>> timed
>> >> out.
>> >> java.lang.RuntimeException: java.io.IOException: An attempt by a
>> client
>> >> to
>> >> checkout a Connection has timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> >> at
>> >>
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> >> Caused by: java.io.IOException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> >>
>> >>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> >> at
>> >>
>> >>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> >> ... 16 more
>> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> >> Connection has timed out.
>> >> at
>> com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> >> at
>> >>
>> >>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> >> ... 20 more
>> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client
>> timed
>> >> out
>> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> >> ResourcePool@d964af -- timeout at awaitAvailable()
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> >> at
>> >>
>> >>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> >> at
>> >>
>> >>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> >> ... 22 more
>> >> INFO TransportConnector - Connector openwire Stopped
>> >>
>> >>
>> >> bsnyder wrote:
>> >> >
>> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
>> >> wrote:
>> >> >>
>> >> >> Thanks for the response Bruce. :)
>> >> >>
>> >> >> To my knowledge we aren't using a connection pooler. We're just
>> >> >> configuring
>> >> >> the activemq xml and letting activeMQ do its thing. One of my
>> >> colleagues
>> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
>> >> similar
>> >> >> to
>> >> >> a connection pooler or not.
>> >> >
>> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
>> >> > connections available for use by a Java application. IIRC, SNAC
>> stands
>> >> > for SQL Native Client which has nothing to do with a JDBC connection
>> >> > pooler.
>> >> >
>> >> >> In our scenario during a Sql Server failover it may take 15
>> minutes.
>> >> We
>> >> >> aren't using a high availability cluster. Does a connection pooler
>> >> give
>> >> >> the
>> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
>> >> >
>> >> > One of the options in a good JDBC connection pooler is the ability
>> to
>> >> > test JDBC connections as it hands them out to the Java application.
>> If
>> >> > the connection is stale for any reason the JDBC connection pooler
>> will
>> >> > evict that connection from the pool and try the next one until it
>> >> > reaches one that is alive. To my knowledge, some JDBC connection
>> >> > poolers provide a the ability to configure some attributes like
>> number
>> >> > of retries, etc. E.g., here is some information about what c3p0
>> >> > offers:
>> >> >
>> >> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
>> >> >
>> >> > The one big caveat I will mention is that longer outage periods such
>> >> > as the 15 minute outage you mention are not typically handled by a
>> >> > JDBC connection pooler as they're mainly geared toward handling
>> >> > momentary outages. As an example, notice that the default value for
>> >> > acquireRetryDelay in c3p0 is only one second:
>> >> >
>> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
>> >> >
>> >> > But you could certainly configure this to any value you want. I
>> >> > encourage you to test this in your environment before deploying to
>> >> > production.
>> >> >
>> >> > Bruce
>> >> > --
>> >> > perl -e 'print
>> >> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> >> > );'
>> >> >
>> >> > Apache ActiveMQ - http://activemq.apache.org/
>> >> > Apache Camel - http://camel.apache.org/
>> >> > Apache ServiceMix - http://servicemix.apache.org/
>> >> >
>> >> > Blog: http://bruceblog.org/
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>> > --
>> > http://blog.garytully.com
>> >
>> > Open Source SOA
>> > http://FUSESource.com
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22973315.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Gary Tully <ga...@gmail.com>.
in 5.2 there is a setter on the JDBCPersistenceAdaptor that allows you to
set a locker instance.
see:
http://activemq.apache.org/maven/5.2.0/activemq-core/apidocs/org/apache/activemq/store/jdbc/JDBCPersistenceAdapter.html#setDatabaseLocker(org.apache.activemq.store.jdbc.DatabaseLocker)
Otherwise, yea, as you describe, derive and inject using spring via the XML
configuration.
hope this helps,
Gary.
2009/4/7 magellings <ma...@qg.com>
>
> How do you exactly provide your own implementation?
>
> On the JDBCPersistenceAdaptor the createDatabaseLocker will always use the
> DefaultDatabaseLocker:
>
> protected DatabaseLocker createDatabaseLocker() throws IOException {
> return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
> }
>
> Can I provide my own implemenation declaratively through configuration
> provided I write a class deriving from JDBCPersistenceAdaptor of which
> overrides the createDatabaseLocker class?
>
>
> Gary Tully wrote:
> >
> > One thing to note: you can provide your own implementation of the
> > org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> > keepAlive to be aware of your database backup policy.
> >
> > If you go down this route and conceive a better default or optional
> > strategy, feel free to submit it as a patch.
> >
> >
> >
> > 2009/3/11 magellings <ma...@qg.com>
> >
> >>
> >> Well it was butt-simple to get working like the c3p0 documentation said.
> >> I
> >> upped the delay to 20s, unfortunately the master broker still failed.
> >>
> >> I'm kind of wondering how the slave would respond. My guess is it would
> >> probably time out too.
> >>
> >> Any thoughts?
> >>
> >> See further below for the errors.
> >>
> >>
> >>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
> >> xbean:activemq.xml
> >> ACTIVEMQ_HOME:
> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> ACTIVEMQ_BASE:
> >> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> >> Loading message broker from: xbean:activemq.xml
> >> INFO DefaultCamelContext - JMX enabled. Using
> >> InstrumentationLifecycleStrategy.
> >> INFO MLog - MLog clients using log4j logging.
> >> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
> >> 21-May-2007 15:04:56; debug? true; trace: 10]
> >> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> acquireRetryAttemp
> >> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> checkoutTimeout -
> >> > 30000, connectionCustomizerClassName -> null,
> connectionTesterClassName
> >> ->
> >> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
> >> 2rvy1
> >> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
> >> description -> null, driverClass ->
> >> com.microsoft.sqlserver.jdbc.SQLServerDri
> >> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
> >> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> idleConnectionTestPerio
> >> d -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> maxAdministrativeTas
> >> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> >> 200,
> >> maxStatementsPerCon
> >> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> >> properties -> {user=**
> >> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
> >> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout ->
> >> 0,
> >> use
> >> sTraditionalReflectiveProxies -> false ]
> >> INFO BrokerService - Using Persistence Adapter:
> >> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
> >> acquireIncrement
> >> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
> >> autoCommitOnClose -> false, automaticTestTable -> con_test,
> >> breakAfterAcquireFailure ->
> >> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName ->
> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> >> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> debugUnreturnedConnectionStackTraces -> false, description -> null,
> >> driverClass -> com.microsoft.sql
> >> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> 2rvy117z1wqpy2hxzkebp|18105e8,
> >> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$wo
> >> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime
> ->
> >> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
> >> ->
> >> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10,
> numHelperThreads
> >> ->
> >> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery ->
> null
> >> , properties -> {user=******, password=******}, propertyCycle -> 0,
> >> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
> >> unreturnedCon
> >> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
> >> INFO JDBCPersistenceAdapter - Database driver recognized:
> >> [microsoft_sql_server_2005_jdbc_driver]
> >> INFO DefaultDatabaseLocker - Attempting to acquire the
> >> exclusive
> >> lock to become the Master broker
> >> INFO DefaultDatabaseLocker - Becoming the master on
> dataSource:
> >> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> >> acquireRe
> >> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
> >> false,
> >> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> >> checkout
> >> Timeout -> 30000, connectionCustomizerClassName -> null,
> >> connectionTesterClassName ->
> >> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
> >> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces
> >> ->
> >> false, description -> null, driverClass ->
> >> com.microsoft.sqlserver.jdbc.SQL
> >> ServerDriver, factoryClassLocation -> null,
> >> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> >> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
> >> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> >> jdbc:sqlserver://QG076619.qg.com:1433
> >> ;databaseName=activedb;user=sa;password=pa$$word,
> >> maxAdminist
> >> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> >> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> >> 200,
> >> maxStateme
> >> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> >> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> >> properties ->
> >> {user=******, password=******}, propertyCycle -> 0,
> >> testConnectionOnCheckin
> >> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
> >> -> 0, usesTraditionalReflectiveProxies -> false ]
> >> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
> >> (master) is starting
> >> INFO BrokerService - For help or more information
> >> please
> >> see: http://activemq.apache.org/
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> tcp://QG076619:61616
> >> INFO TransportConnector - Connector openwire Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> ssl://QG076619:61617
> >> INFO TransportConnector - Connector ssl Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> stomp://QG076619:61613
> >> INFO TransportConnector - Connector stomp Started
> >> INFO TransportServerThreadSupport - Listening for connections at:
> >> xmpp://QG076619:61222
> >> INFO TransportConnector - Connector xmpp Started
> >> INFO BrokerService - ActiveMQ JMS Message Broker
> >> (master,
> >> ID:QG076619-3632-1236812198396-0:0) started
> >> INFO log - Logging to
> >> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
> >> org.mortbay.log.Slf4jLog
> >> INFO log - jetty-6.1.9
> >> INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
> >> INFO /admin - Initializing Spring
> >> FrameworkServlet
> >> 'dispatcher'
> >> INFO log - ActiveMQ Console at
> >> http://0.0.0.0:8162/admin
> >> INFO log - ActiveMQ Web Demos at
> >> http://0.0.0.0:8162/demo
> >> INFO log - RESTful file access application
> at
> >> http://0.0.0.0:8162/fileserver
> >> INFO log - Started
> >> SelectChannelConnector@0.0.0.0:8162
> >> WARN BrokerRegistry - Broker localhost not started so
> >> using
> >> master instead
> >> INFO TransportConnector - Connector vm://localhost Started
> >>
> >>
> >> Then BOOM! Although behavior is different. Now it takes longer to
> stop.
> >> :) "The connection to '/10.0.1.5:3333' is taking a long time to
> >> shutdown."
> >> If you look closely you can see activeMQ is timing out.
> >>
> >>
> >>
> >>
> >> ERROR DefaultDatabaseLocker - Failed to update database lock:
> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
> >> et write error
> >> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
> >> peer:
> >> socket write error
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
> >> Source)
> >> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown
> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
> >> Source)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> at
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> at
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >> INFO GooGooStatementCache - Problem with checked-in
> Statement,
> >> discarding.
> >> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
> >> closed.
> >> at
> >>
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
> >> Source)
> >> at
> >> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
> >> Source)
> >> at
> >>
> >>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
> >> Source)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> >> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >> at
> >>
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> >> at
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> >> at
> >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> at
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> at java.lang.Thread.run(Thread.java:619)
> >> INFO JDBCPersistenceAdapter - No longer able to keep the
> >> exclusive
> >> lock so giving up being a master
> >> INFO BrokerService - ActiveMQ Message Broker (master,
> >> ID:QG076619-3332-1236811661934-0:0) is shutting down
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> WARN ThreadPoolAsynchronousRunner -
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> -- APPARENT DEADLOCK!!! Crea
> >> rgency threads for unassigned pending tasks!
> >> WARN ThreadPoolAsynchronousRunner -
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> >> -- APPARENT DEADLOCK!!! Comp
> >> tus:
> >> Managed Threads: 3
> >> Active Threads: 3
> >> Active Tasks:
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
> >> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
> >> Pending Tasks:
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
> >>
> >> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
> >>
> >>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
> >> Pool thread stack traces:
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
> >> java.lang.Thread.sleep(Native Method)
> >>
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
> >>
> >>
> >>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
> >>
> >>
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> INFO TransportConnection - The connection to '/
> 10.0.1.5:3333'
> >> is
> >> taking a long time to shutdown.
> >> ERROR QueueStorePrefetch - Failed to get message count
> >> java.io.IOException: An attempt by a client to checkout a Connection has
> >> timed out.
> >> at
> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> at
> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> at
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> at
> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> ... 20 more
> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
> >> out
> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> ... 22 more
> >> WARN TransportConnection - Failed to remove consumer:
> >> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
> >> java.lang.RuntimeException
> >> o.IOException: An attempt by a client to checkout a Connection has timed
> >> out.
> >> java.lang.RuntimeException: java.io.IOException: An attempt by a client
> >> to
> >> checkout a Connection has timed out.
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> >> at
> >>
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> >> at
> >>
> >>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> >> at
> >>
> >>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> >> at
> >>
> >>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> >> at
> >>
> >>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> >> Caused by: java.io.IOException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at
> >>
> >>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> >> at
> >>
> >>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> >> at
> >>
> >>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> >> ... 16 more
> >> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> >> Connection has timed out.
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> >> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> >> at
> >>
> >>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> >> ... 20 more
> >> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
> >> out
> >> while waiting to acquire a resource from com.mchange.v2.resourcepo
> >> ResourcePool@d964af -- timeout at awaitAvailable()
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> >> at
> >>
> >>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> >> at
> >>
> >>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> >> ... 22 more
> >> INFO TransportConnector - Connector openwire Stopped
> >>
> >>
> >> bsnyder wrote:
> >> >
> >> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
> >> wrote:
> >> >>
> >> >> Thanks for the response Bruce. :)
> >> >>
> >> >> To my knowledge we aren't using a connection pooler. We're just
> >> >> configuring
> >> >> the activemq xml and letting activeMQ do its thing. One of my
> >> colleagues
> >> >> mentioned SNAC since we are using Sql Server, not sure if that is
> >> similar
> >> >> to
> >> >> a connection pooler or not.
> >> >
> >> > The job of a JDBC connection pooler is to keep a pool of JDBC
> >> > connections available for use by a Java application. IIRC, SNAC stands
> >> > for SQL Native Client which has nothing to do with a JDBC connection
> >> > pooler.
> >> >
> >> >> In our scenario during a Sql Server failover it may take 15 minutes.
> >> We
> >> >> aren't using a high availability cluster. Does a connection pooler
> >> give
> >> >> the
> >> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
> >> >
> >> > One of the options in a good JDBC connection pooler is the ability to
> >> > test JDBC connections as it hands them out to the Java application. If
> >> > the connection is stale for any reason the JDBC connection pooler will
> >> > evict that connection from the pool and try the next one until it
> >> > reaches one that is alive. To my knowledge, some JDBC connection
> >> > poolers provide a the ability to configure some attributes like number
> >> > of retries, etc. E.g., here is some information about what c3p0
> >> > offers:
> >> >
> >> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
> >> >
> >> > The one big caveat I will mention is that longer outage periods such
> >> > as the 15 minute outage you mention are not typically handled by a
> >> > JDBC connection pooler as they're mainly geared toward handling
> >> > momentary outages. As an example, notice that the default value for
> >> > acquireRetryDelay in c3p0 is only one second:
> >> >
> >> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
> >> >
> >> > But you could certainly configure this to any value you want. I
> >> > encourage you to test this in your environment before deploying to
> >> > production.
> >> >
> >> > Bruce
> >> > --
> >> > perl -e 'print
> >> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> >> > );'
> >> >
> >> > Apache ActiveMQ - http://activemq.apache.org/
> >> > Apache Camel - http://camel.apache.org/
> >> > Apache ServiceMix - http://servicemix.apache.org/
> >> >
> >> > Blog: http://bruceblog.org/
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > http://blog.garytully.com
> >
> > Open Source SOA
> > http://FUSESource.com
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by magellings <ma...@qg.com>.
How do you exactly provide your own implementation?
On the JDBCPersistenceAdaptor the createDatabaseLocker will always use the
DefaultDatabaseLocker:
protected DatabaseLocker createDatabaseLocker() throws IOException {
return new DefaultDatabaseLocker(getLockDataSource(), getStatements());
}
Can I provide my own implemenation declaratively through configuration
provided I write a class deriving from JDBCPersistenceAdaptor of which
overrides the createDatabaseLocker class?
Gary Tully wrote:
>
> One thing to note: you can provide your own implementation of the
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
> keepAlive to be aware of your database backup policy.
>
> If you go down this route and conceive a better default or optional
> strategy, feel free to submit it as a patch.
>
>
>
> 2009/3/11 magellings <ma...@qg.com>
>
>>
>> Well it was butt-simple to get working like the c3p0 documentation said.
>> I
>> upped the delay to 20s, unfortunately the master broker still failed.
>>
>> I'm kind of wondering how the slave would respond. My guess is it would
>> probably time out too.
>>
>> Any thoughts?
>>
>> See further below for the errors.
>>
>>
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
>> xbean:activemq.xml
>> ACTIVEMQ_HOME:
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> ACTIVEMQ_BASE:
>> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
>> Loading message broker from: xbean:activemq.xml
>> INFO DefaultCamelContext - JMX enabled. Using
>> InstrumentationLifecycleStrategy.
>> INFO MLog - MLog clients using log4j logging.
>> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
>> 21-May-2007 15:04:56; debug? true; trace: 10]
>> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
>> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> acquireRetryAttemp
>> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
>> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> checkoutTimeout -
>> > 30000, connectionCustomizerClassName -> null, connectionTesterClassName
>> ->
>> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
>> 2rvy1
>> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
>> description -> null, driverClass ->
>> com.microsoft.sqlserver.jdbc.SQLServerDri
>> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
>> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> idleConnectionTestPerio
>> d -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$word,
>> maxAdministrativeTas
>> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
>> 200,
>> maxStatementsPerCon
>> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
>> properties -> {user=**
>> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
>> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout ->
>> 0,
>> use
>> sTraditionalReflectiveProxies -> false ]
>> INFO BrokerService - Using Persistence Adapter:
>> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
>> acquireIncrement
>> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
>> autoCommitOnClose -> false, automaticTestTable -> con_test,
>> breakAfterAcquireFailure ->
>> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName ->
>> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
>> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
>> debugUnreturnedConnectionStackTraces -> false, description -> null,
>> driverClass -> com.microsoft.sql
>> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
>> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> 2rvy117z1wqpy2hxzkebp|18105e8,
>> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$wo
>> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime ->
>> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements
>> ->
>> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10, numHelperThreads
>> ->
>> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null
>> , properties -> {user=******, password=******}, propertyCycle -> 0,
>> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
>> unreturnedCon
>> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
>> INFO JDBCPersistenceAdapter - Database driver recognized:
>> [microsoft_sql_server_2005_jdbc_driver]
>> INFO DefaultDatabaseLocker - Attempting to acquire the
>> exclusive
>> lock to become the Master broker
>> INFO DefaultDatabaseLocker - Becoming the master on dataSource:
>> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
>> acquireRe
>> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose ->
>> false,
>> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
>> checkout
>> Timeout -> 30000, connectionCustomizerClassName -> null,
>> connectionTesterClassName ->
>> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
>> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces
>> ->
>> false, description -> null, driverClass ->
>> com.microsoft.sqlserver.jdbc.SQL
>> ServerDriver, factoryClassLocation -> null,
>> forceIgnoreUnresolvedTransactions -> false, identityToken ->
>> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
>> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
>> jdbc:sqlserver://QG076619.qg.com:1433
>> ;databaseName=activedb;user=sa;password=pa$$word,
>> maxAdminist
>> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
>> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
>> 200,
>> maxStateme
>> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
>> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
>> properties ->
>> {user=******, password=******}, propertyCycle -> 0,
>> testConnectionOnCheckin
>> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
>> -> 0, usesTraditionalReflectiveProxies -> false ]
>> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
>> (master) is starting
>> INFO BrokerService - For help or more information
>> please
>> see: http://activemq.apache.org/
>> INFO TransportServerThreadSupport - Listening for connections at:
>> tcp://QG076619:61616
>> INFO TransportConnector - Connector openwire Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> ssl://QG076619:61617
>> INFO TransportConnector - Connector ssl Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> stomp://QG076619:61613
>> INFO TransportConnector - Connector stomp Started
>> INFO TransportServerThreadSupport - Listening for connections at:
>> xmpp://QG076619:61222
>> INFO TransportConnector - Connector xmpp Started
>> INFO BrokerService - ActiveMQ JMS Message Broker
>> (master,
>> ID:QG076619-3632-1236812198396-0:0) started
>> INFO log - Logging to
>> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
>> org.mortbay.log.Slf4jLog
>> INFO log - jetty-6.1.9
>> INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
>> INFO /admin - Initializing Spring
>> FrameworkServlet
>> 'dispatcher'
>> INFO log - ActiveMQ Console at
>> http://0.0.0.0:8162/admin
>> INFO log - ActiveMQ Web Demos at
>> http://0.0.0.0:8162/demo
>> INFO log - RESTful file access application at
>> http://0.0.0.0:8162/fileserver
>> INFO log - Started
>> SelectChannelConnector@0.0.0.0:8162
>> WARN BrokerRegistry - Broker localhost not started so
>> using
>> master instead
>> INFO TransportConnector - Connector vm://localhost Started
>>
>>
>> Then BOOM! Although behavior is different. Now it takes longer to stop.
>> :) "The connection to '/10.0.1.5:3333' is taking a long time to
>> shutdown."
>> If you look closely you can see activeMQ is timing out.
>>
>>
>>
>>
>> ERROR DefaultDatabaseLocker - Failed to update database lock:
>> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
>> et write error
>> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by
>> peer:
>> socket write error
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
>> Source)
>> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
>> Source)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
>> at
>>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> INFO GooGooStatementCache - Problem with checked-in Statement,
>> discarding.
>> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is
>> closed.
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
>> Source)
>> at
>> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown
>> Source)
>> at
>>
>> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
>> Source)
>> at
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
>> at
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
>> at
>>
>> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
>> at
>>
>> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> at java.lang.Thread.run(Thread.java:619)
>> INFO JDBCPersistenceAdapter - No longer able to keep the
>> exclusive
>> lock so giving up being a master
>> INFO BrokerService - ActiveMQ Message Broker (master,
>> ID:QG076619-3332-1236811661934-0:0) is shutting down
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> WARN ThreadPoolAsynchronousRunner -
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> -- APPARENT DEADLOCK!!! Crea
>> rgency threads for unassigned pending tasks!
>> WARN ThreadPoolAsynchronousRunner -
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
>> -- APPARENT DEADLOCK!!! Comp
>> tus:
>> Managed Threads: 3
>> Active Threads: 3
>> Active Tasks:
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
>> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
>> Pending Tasks:
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
>>
>> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
>> Pool thread stack traces:
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
>> java.lang.Thread.sleep(Native Method)
>>
>>
>> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>>
>>
>> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>>
>>
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> INFO TransportConnection - The connection to '/10.0.1.5:3333'
>> is
>> taking a long time to shutdown.
>> ERROR QueueStorePrefetch - Failed to get message count
>> java.io.IOException: An attempt by a client to checkout a Connection has
>> timed out.
>> at
>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> at
>>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> at
>>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> at
>>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> at
>>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> at
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> at
>>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> at
>>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> at
>>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> at
>>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> Connection has timed out.
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> at
>>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> ... 20 more
>> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
>> out
>> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> ResourcePool@d964af -- timeout at awaitAvailable()
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> ... 22 more
>> WARN TransportConnection - Failed to remove consumer:
>> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
>> java.lang.RuntimeException
>> o.IOException: An attempt by a client to checkout a Connection has timed
>> out.
>> java.lang.RuntimeException: java.io.IOException: An attempt by a client
>> to
>> checkout a Connection has timed out.
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
>> at
>>
>> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
>> at
>>
>> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
>> at
>> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
>> at
>>
>> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
>> at
>>
>> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
>> at
>>
>> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
>> at
>>
>> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
>> at
>>
>> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
>> at
>>
>> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
>> at
>>
>> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
>> Caused by: java.io.IOException: An attempt by a client to checkout a
>> Connection has timed out.
>> at
>>
>> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
>> at
>>
>> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
>> at
>>
>> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
>> at
>>
>> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
>> at
>>
>> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
>> ... 16 more
>> Caused by: java.sql.SQLException: An attempt by a client to checkout a
>> Connection has timed out.
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
>> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
>> at
>>
>> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
>> at
>>
>> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
>> ... 20 more
>> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed
>> out
>> while waiting to acquire a resource from com.mchange.v2.resourcepo
>> ResourcePool@d964af -- timeout at awaitAvailable()
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
>> at
>>
>> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
>> at
>>
>> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
>> ... 22 more
>> INFO TransportConnector - Connector openwire Stopped
>>
>>
>> bsnyder wrote:
>> >
>> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
>> wrote:
>> >>
>> >> Thanks for the response Bruce. :)
>> >>
>> >> To my knowledge we aren't using a connection pooler. We're just
>> >> configuring
>> >> the activemq xml and letting activeMQ do its thing. One of my
>> colleagues
>> >> mentioned SNAC since we are using Sql Server, not sure if that is
>> similar
>> >> to
>> >> a connection pooler or not.
>> >
>> > The job of a JDBC connection pooler is to keep a pool of JDBC
>> > connections available for use by a Java application. IIRC, SNAC stands
>> > for SQL Native Client which has nothing to do with a JDBC connection
>> > pooler.
>> >
>> >> In our scenario during a Sql Server failover it may take 15 minutes.
>> We
>> >> aren't using a high availability cluster. Does a connection pooler
>> give
>> >> the
>> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
>> >
>> > One of the options in a good JDBC connection pooler is the ability to
>> > test JDBC connections as it hands them out to the Java application. If
>> > the connection is stale for any reason the JDBC connection pooler will
>> > evict that connection from the pool and try the next one until it
>> > reaches one that is alive. To my knowledge, some JDBC connection
>> > poolers provide a the ability to configure some attributes like number
>> > of retries, etc. E.g., here is some information about what c3p0
>> > offers:
>> >
>> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
>> >
>> > The one big caveat I will mention is that longer outage periods such
>> > as the 15 minute outage you mention are not typically handled by a
>> > JDBC connection pooler as they're mainly geared toward handling
>> > momentary outages. As an example, notice that the default value for
>> > acquireRetryDelay in c3p0 is only one second:
>> >
>> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
>> >
>> > But you could certainly configure this to any value you want. I
>> > encourage you to test this in your environment before deploying to
>> > production.
>> >
>> > Bruce
>> > --
>> > perl -e 'print
>> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> > );'
>> >
>> > Apache ActiveMQ - http://activemq.apache.org/
>> > Apache Camel - http://camel.apache.org/
>> > Apache ServiceMix - http://servicemix.apache.org/
>> >
>> > Blog: http://bruceblog.org/
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> http://blog.garytully.com
>
> Open Source SOA
> http://FUSESource.com
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22939332.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Gary Tully <ga...@gmail.com>.
One thing to note: you can provide your own implementation of the
org.apache.activemq.store.jdbc.DefaultDatabaseLocker and implement
keepAlive to be aware of your database backup policy.
If you go down this route and conceive a better default or optional
strategy, feel free to submit it as a patch.
2009/3/11 magellings <ma...@qg.com>
>
> Well it was butt-simple to get working like the c3p0 documentation said. I
> upped the delay to 20s, unfortunately the master broker still failed.
>
> I'm kind of wondering how the slave would respond. My guess is it would
> probably time out too.
>
> Any thoughts?
>
> See further below for the errors.
>
>
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
> xbean:activemq.xml
> ACTIVEMQ_HOME:
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> ACTIVEMQ_BASE:
> C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
> Loading message broker from: xbean:activemq.xml
> INFO DefaultCamelContext - JMX enabled. Using
> InstrumentationLifecycleStrategy.
> INFO MLog - MLog clients using log4j logging.
> INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
> 21-May-2007 15:04:56; debug? true; trace: 10]
> INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> acquireRetryAttemp
> ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
> checkoutTimeout -
> > 30000, connectionCustomizerClassName -> null, connectionTesterClassName
> ->
> > com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2rvy1
> 17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
> description -> null, driverClass ->
> com.microsoft.sqlserver.jdbc.SQLServerDri
> ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
> false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
> idleConnectionTestPerio
> d -> 30, initialPoolSize -> 10, jdbcUrl ->
> jdbc:sqlserver://QG076619.qg.com:1433
> ;databaseName=activedb;user=sa;password=pa$$word,
> maxAdministrativeTas
> kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> 200,
> maxStatementsPerCon
> nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> properties -> {user=**
> ****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0,
> use
> sTraditionalReflectiveProxies -> false ]
> INFO BrokerService - Using Persistence Adapter:
> JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
> acquireIncrement
> -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
> autoCommitOnClose -> false, automaticTestTable -> con_test,
> breakAfterAcquireFailure ->
> false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
> connectionTesterClassName ->
> com.mchange.v2.c3p0.impl.DefaultConnectionTester,
> dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
> debugUnreturnedConnectionStackTraces -> false, description -> null,
> driverClass -> com.microsoft.sql
> server.jdbc.SQLServerDriver, factoryClassLocation -> null,
> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> 2rvy117z1wqpy2hxzkebp|18105e8,
> idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> jdbc:sqlserver://QG076619.qg.com:1433
> ;databaseName=activedb;user=sa;password=pa$$wo
> rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime ->
> 30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> 200, maxStatementsPerConnection -> 0, minPoolSize -> 10, numHelperThreads
> ->
> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null
> , properties -> {user=******, password=******}, propertyCycle -> 0,
> testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
> unreturnedCon
> nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
> INFO JDBCPersistenceAdapter - Database driver recognized:
> [microsoft_sql_server_2005_jdbc_driver]
> INFO DefaultDatabaseLocker - Attempting to acquire the exclusive
> lock to become the Master broker
> INFO DefaultDatabaseLocker - Becoming the master on dataSource:
> com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
> acquireRe
> tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
> automaticTestTable -> con_test, breakAfterAcquireFailure -> false, checkout
> Timeout -> 30000, connectionCustomizerClassName -> null,
> connectionTesterClassName ->
> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
> -> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
> false, description -> null, driverClass -> com.microsoft.sqlserver.jdbc.SQL
> ServerDriver, factoryClassLocation -> null,
> forceIgnoreUnresolvedTransactions -> false, identityToken ->
> 2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
> TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
> jdbc:sqlserver://QG076619.qg.com:1433
> ;databaseName=activedb;user=sa;password=pa$$word,
> maxAdminist
> rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
> maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
> 200,
> maxStateme
> ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
> numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
> properties ->
> {user=******, password=******}, propertyCycle -> 0,
> testConnectionOnCheckin
> -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
> -> 0, usesTraditionalReflectiveProxies -> false ]
> INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
> (master) is starting
> INFO BrokerService - For help or more information please
> see: http://activemq.apache.org/
> INFO TransportServerThreadSupport - Listening for connections at:
> tcp://QG076619:61616
> INFO TransportConnector - Connector openwire Started
> INFO TransportServerThreadSupport - Listening for connections at:
> ssl://QG076619:61617
> INFO TransportConnector - Connector ssl Started
> INFO TransportServerThreadSupport - Listening for connections at:
> stomp://QG076619:61613
> INFO TransportConnector - Connector stomp Started
> INFO TransportServerThreadSupport - Listening for connections at:
> xmpp://QG076619:61222
> INFO TransportConnector - Connector xmpp Started
> INFO BrokerService - ActiveMQ JMS Message Broker (master,
> ID:QG076619-3632-1236812198396-0:0) started
> INFO log - Logging to
> org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> INFO log - jetty-6.1.9
> INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
> INFO /admin - Initializing Spring FrameworkServlet
> 'dispatcher'
> INFO log - ActiveMQ Console at
> http://0.0.0.0:8162/admin
> INFO log - ActiveMQ Web Demos at
> http://0.0.0.0:8162/demo
> INFO log - RESTful file access application at
> http://0.0.0.0:8162/fileserver
> INFO log - Started
> SelectChannelConnector@0.0.0.0:8162
> WARN BrokerRegistry - Broker localhost not started so
> using
> master instead
> INFO TransportConnector - Connector vm://localhost Started
>
>
> Then BOOM! Although behavior is different. Now it takes longer to stop.
> :) "The connection to '/10.0.1.5:3333' is taking a long time to
> shutdown."
> If you look closely you can see activeMQ is timing out.
>
>
>
>
> ERROR DefaultDatabaseLocker - Failed to update database lock:
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
> et write error
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by peer:
> socket write error
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown Source)
> at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
> at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
> at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
> Source)
> at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown Source)
> at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
> Source)
> at
>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
> Source)
> at
>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
> Source)
> at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
> Source)
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
> Source)
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
> Source)
> at
>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
> Source)
> at
>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
> at
>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
> at
>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> at
>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> INFO GooGooStatementCache - Problem with checked-in Statement,
> discarding.
> com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
> at
> com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
> Source)
> at
> com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown
> Source)
> at
> com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown Source)
> at
>
> com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
> Source)
> at
>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
> at
>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
> at
>
> com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
> at
>
> com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
> at
>
> org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
> at
>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
> at
>
> org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> INFO JDBCPersistenceAdapter - No longer able to keep the exclusive
> lock so giving up being a master
> INFO BrokerService - ActiveMQ Message Broker (master,
> ID:QG076619-3332-1236811661934-0:0) is shutting down
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> WARN ThreadPoolAsynchronousRunner -
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> -- APPARENT DEADLOCK!!! Crea
> rgency threads for unassigned pending tasks!
> WARN ThreadPoolAsynchronousRunner -
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
> -- APPARENT DEADLOCK!!! Comp
> tus:
> Managed Threads: 3
> Active Threads: 3
> Active Tasks:
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
> (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
> Pending Tasks:
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
>
> com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
>
> com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
> Pool thread stack traces:
>
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
> java.lang.Thread.sleep(Native Method)
>
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
> java.lang.Thread.sleep(Native Method)
>
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>
>
> Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
> java.lang.Thread.sleep(Native Method)
>
>
> com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
>
>
> com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
>
>
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> INFO TransportConnection - The connection to '/10.0.1.5:3333'
> is
> taking a long time to shutdown.
> ERROR QueueStorePrefetch - Failed to get message count
> java.io.IOException: An attempt by a client to checkout a Connection has
> timed out.
> at
>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> at
>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> at
>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> at
>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> at
>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> at
>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> at
>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> at
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> at
>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> at
>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> at
>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> at
>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> at
>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> Connection has timed out.
> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> at
>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> at
>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> at
>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> ... 20 more
> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out
> while waiting to acquire a resource from com.mchange.v2.resourcepo
> ResourcePool@d964af -- timeout at awaitAvailable()
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> at
>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> ... 22 more
> WARN TransportConnection - Failed to remove consumer:
> 9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason:
> java.lang.RuntimeException
> o.IOException: An attempt by a client to checkout a Connection has timed
> out.
> java.lang.RuntimeException: java.io.IOException: An attempt by a client to
> checkout a Connection has timed out.
> at
>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
> at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
> at
>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
> at
> org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
> at
>
> org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
> at
>
> org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
> at
>
> org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
> at
>
> org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
> at
>
> org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
> at
>
> org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
> at
>
> org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
> Caused by: java.io.IOException: An attempt by a client to checkout a
> Connection has timed out.
> at
>
> org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
> at
>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
> at
>
> org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
> at
>
> org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
> at
>
> org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
> at
>
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
> ... 16 more
> Caused by: java.sql.SQLException: An attempt by a client to checkout a
> Connection has timed out.
> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
> at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
> at
>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
> at
>
> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
> at
>
> org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
> ... 20 more
> Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out
> while waiting to acquire a resource from com.mchange.v2.resourcepo
> ResourcePool@d964af -- timeout at awaitAvailable()
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
> at
>
> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
> at
>
> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
> ... 22 more
> INFO TransportConnector - Connector openwire Stopped
>
>
> bsnyder wrote:
> >
> > On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com>
> wrote:
> >>
> >> Thanks for the response Bruce. :)
> >>
> >> To my knowledge we aren't using a connection pooler. We're just
> >> configuring
> >> the activemq xml and letting activeMQ do its thing. One of my
> colleagues
> >> mentioned SNAC since we are using Sql Server, not sure if that is
> similar
> >> to
> >> a connection pooler or not.
> >
> > The job of a JDBC connection pooler is to keep a pool of JDBC
> > connections available for use by a Java application. IIRC, SNAC stands
> > for SQL Native Client which has nothing to do with a JDBC connection
> > pooler.
> >
> >> In our scenario during a Sql Server failover it may take 15 minutes. We
> >> aren't using a high availability cluster. Does a connection pooler give
> >> the
> >> capability of dealing with this so that ActiveMQ doesn't shutdown?
> >
> > One of the options in a good JDBC connection pooler is the ability to
> > test JDBC connections as it hands them out to the Java application. If
> > the connection is stale for any reason the JDBC connection pooler will
> > evict that connection from the pool and try the next one until it
> > reaches one that is alive. To my knowledge, some JDBC connection
> > poolers provide a the ability to configure some attributes like number
> > of retries, etc. E.g., here is some information about what c3p0
> > offers:
> >
> > http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
> >
> > The one big caveat I will mention is that longer outage periods such
> > as the 15 minute outage you mention are not typically handled by a
> > JDBC connection pooler as they're mainly geared toward handling
> > momentary outages. As an example, notice that the default value for
> > acquireRetryDelay in c3p0 is only one second:
> >
> > http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
> >
> > But you could certainly configure this to any value you want. I
> > encourage you to test this in your environment before deploying to
> > production.
> >
> > Bruce
> > --
> > perl -e 'print
> > unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> > );'
> >
> > Apache ActiveMQ - http://activemq.apache.org/
> > Apache Camel - http://camel.apache.org/
> > Apache ServiceMix - http://servicemix.apache.org/
> >
> > Blog: http://bruceblog.org/
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
http://blog.garytully.com
Open Source SOA
http://FUSESource.com
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by magellings <ma...@qg.com>.
Well it was butt-simple to get working like the c3p0 documentation said. I
upped the delay to 20s, unfortunately the master broker still failed.
I'm kind of wondering how the slave would respond. My guess is it would
probably time out too.
Any thoughts?
See further below for the errors.
C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin>activemq.bat
xbean:activemq.xml
ACTIVEMQ_HOME:
C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
ACTIVEMQ_BASE:
C:\WIP\ActiveMQ\apache-activemq-5.2.0-bin\apache-activemq-5.2.0\bin\..
Loading message broker from: xbean:activemq.xml
INFO DefaultCamelContext - JMX enabled. Using
InstrumentationLifecycleStrategy.
INFO MLog - MLog clients using log4j logging.
INFO C3P0Registry - Initializing c3p0-0.9.1.2 [built
21-May-2007 15:04:56; debug? true; trace: 10]
INFO AbstractPoolBackedDataSource - Initializing c3p0 pool...
com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3,
acquireRetryAttemp
ts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
automaticTestTable -> con_test, breakAfterAcquireFailure -> false,
checkoutTimeout -
> 30000, connectionCustomizerClassName -> null, connectionTesterClassName ->
> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 2rvy1
17z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces -> false,
description -> null, driverClass ->
com.microsoft.sqlserver.jdbc.SQLServerDri
ver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
false, identityToken -> 2rvy117z1wqpy2hxzkebp|18105e8,
idleConnectionTestPerio
d -> 30, initialPoolSize -> 10, jdbcUrl ->
jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$word,
maxAdministrativeTas
kTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> 200,
maxStatementsPerCon
nection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
properties -> {user=**
****, password=******}, propertyCycle -> 0, testConnectionOnCheckin ->
false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0,
use
sTraditionalReflectiveProxies -> false ]
INFO BrokerService - Using Persistence Adapter:
JDBCPersistenceAdaptor(com.mchange.v2.c3p0.ComboPooledDataSource [
acquireIncrement
-> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 20000,
autoCommitOnClose -> false, automaticTestTable -> con_test,
breakAfterAcquireFailure ->
false, checkoutTimeout -> 30000, connectionCustomizerClassName -> null,
connectionTesterClassName ->
com.mchange.v2.c3p0.impl.DefaultConnectionTester,
dataSourceName -> 2rvy117z1wqpy2hxzkebp|18105e8,
debugUnreturnedConnectionStackTraces -> false, description -> null,
driverClass -> com.microsoft.sql
server.jdbc.SQLServerDriver, factoryClassLocation -> null,
forceIgnoreUnresolvedTransactions -> false, identityToken ->
2rvy117z1wqpy2hxzkebp|18105e8,
idleConnectionTestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$wo
rd, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime ->
30, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements ->
200, maxStatementsPerConnection -> 0, minPoolSize -> 10, numHelperThreads ->
3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null
, properties -> {user=******, password=******}, propertyCycle -> 0,
testConnectionOnCheckin -> false, testConnectionOnCheckout -> false,
unreturnedCon
nectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ])
INFO JDBCPersistenceAdapter - Database driver recognized:
[microsoft_sql_server_2005_jdbc_driver]
INFO DefaultDatabaseLocker - Attempting to acquire the exclusive
lock to become the Master broker
INFO DefaultDatabaseLocker - Becoming the master on dataSource:
com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRe
tryAttempts -> 30, acquireRetryDelay -> 20000, autoCommitOnClose -> false,
automaticTestTable -> con_test, breakAfterAcquireFailure -> false, checkout
Timeout -> 30000, connectionCustomizerClassName -> null,
connectionTesterClassName ->
com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName
-> 2rvy117z1wqpy2hxzkebp|18105e8, debugUnreturnedConnectionStackTraces ->
false, description -> null, driverClass -> com.microsoft.sqlserver.jdbc.SQL
ServerDriver, factoryClassLocation -> null,
forceIgnoreUnresolvedTransactions -> false, identityToken ->
2rvy117z1wqpy2hxzkebp|18105e8, idleConnection
TestPeriod -> 30, initialPoolSize -> 10, jdbcUrl ->
jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$word,
maxAdminist
rativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 30,
maxIdleTimeExcessConnections -> 0, maxPoolSize -> 100, maxStatements -> 200,
maxStateme
ntsPerConnection -> 0, minPoolSize -> 10, numHelperThreads -> 3,
numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null,
properties ->
{user=******, password=******}, propertyCycle -> 0, testConnectionOnCheckin
-> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
-> 0, usesTraditionalReflectiveProxies -> false ]
INFO BrokerService - ActiveMQ 5.2.0 JMS Message Broker
(master) is starting
INFO BrokerService - For help or more information please
see: http://activemq.apache.org/
INFO TransportServerThreadSupport - Listening for connections at:
tcp://QG076619:61616
INFO TransportConnector - Connector openwire Started
INFO TransportServerThreadSupport - Listening for connections at:
ssl://QG076619:61617
INFO TransportConnector - Connector ssl Started
INFO TransportServerThreadSupport - Listening for connections at:
stomp://QG076619:61613
INFO TransportConnector - Connector stomp Started
INFO TransportServerThreadSupport - Listening for connections at:
xmpp://QG076619:61222
INFO TransportConnector - Connector xmpp Started
INFO BrokerService - ActiveMQ JMS Message Broker (master,
ID:QG076619-3632-1236812198396-0:0) started
INFO log - Logging to
org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
INFO log - jetty-6.1.9
INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
INFO /admin - Initializing Spring FrameworkServlet
'dispatcher'
INFO log - ActiveMQ Console at
http://0.0.0.0:8162/admin
INFO log - ActiveMQ Web Demos at
http://0.0.0.0:8162/demo
INFO log - RESTful file access application at
http://0.0.0.0:8162/fileserver
INFO log - Started
SelectChannelConnector@0.0.0.0:8162
WARN BrokerRegistry - Broker localhost not started so using
master instead
INFO TransportConnector - Connector vm://localhost Started
Then BOOM! Although behavior is different. Now it takes longer to stop.
:) "The connection to '/10.0.1.5:3333' is taking a long time to shutdown."
If you look closely you can see activeMQ is timing out.
ERROR DefaultDatabaseLocker - Failed to update database lock:
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by pe
et write error
com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset by peer:
socket write error
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSChannel.write(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSWriter.flush(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSWriter.writePacket(Unknown
Source)
at com.microsoft.sqlserver.jdbc.TDSWriter.endMessage(Unknown Source)
at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown
Source)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown
Source)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
at
org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:118)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO GooGooStatementCache - Problem with checked-in Statement,
discarding.
com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
at
com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(Unknown
Source)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(Unknown Source)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.checkClosed(Unknown Source)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.clearParameters(Unknown
Source)
at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.refreshStatement(GooGooStatementCache.java:604)
at
com.mchange.v2.c3p0.stmt.GooGooStatementCache.checkinStatement(GooGooStatementCache.java:236)
at
com.mchange.v2.c3p0.impl.NewPooledConnection.checkinStatement(NewPooledConnection.java:282)
at
com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.close(NewProxyPreparedStatement.java:1807)
at
org.apache.activemq.store.jdbc.DefaultDatabaseLocker.keepAlive(DefaultDatabaseLocker.java:127)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.databaseLockKeepAlive(JDBCPersistenceAdapter.java:481)
at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter$1.run(JDBCPersistenceAdapter.java:183)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
INFO JDBCPersistenceAdapter - No longer able to keep the exclusive
lock so giving up being a master
INFO BrokerService - ActiveMQ Message Broker (master,
ID:QG076619-3332-1236811661934-0:0) is shutting down
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
WARN ThreadPoolAsynchronousRunner -
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
-- APPARENT DEADLOCK!!! Crea
rgency threads for unassigned pending tasks!
WARN ThreadPoolAsynchronousRunner -
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@1b00766
-- APPARENT DEADLOCK!!! Comp
tus:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@81933a
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e90097
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@125ee49
(com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThrea
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@13f212a
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@19a6087
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@37504d
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@178feba
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@9b86ea
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@b6a561
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@123961b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@153f141
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@405f6
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@235e9d
com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTask@189cab
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d5b945
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@767fb3
com.mchange.v2.c3p0.stmt.GooGooStatementCache$1StatementCloseTask@1b9e7fc
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.lang.Thread.sleep(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.lang.Thread.sleep(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.lang.Thread.sleep(Native Method)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
INFO TransportConnection - The connection to '/10.0.1.5:3333' is
taking a long time to shutdown.
ERROR QueueStorePrefetch - Failed to get message count
java.io.IOException: An attempt by a client to checkout a Connection has
timed out.
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
at
org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
at
org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
at
org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
at
org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
at
org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
at
org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
Caused by: java.sql.SQLException: An attempt by a client to checkout a
Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
... 20 more
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out
while waiting to acquire a resource from com.mchange.v2.resourcepo
ResourcePool@d964af -- timeout at awaitAvailable()
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 22 more
WARN TransportConnection - Failed to remove consumer:
9cf41a09-ad04-42c0-af9c-efcedb5e61a6:1:1. Reason: java.lang.RuntimeException
o.IOException: An attempt by a client to checkout a Connection has timed
out.
java.lang.RuntimeException: java.io.IOException: An attempt by a client to
checkout a Connection has timed out.
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:66)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.gc(AbstractStoreCursor.java:212)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.gc(StoreQueueCursor.java:257)
at
org.apache.activemq.broker.region.Queue.removeSubscription(Queue.java:308)
at
org.apache.activemq.broker.region.AbstractRegion.removeConsumer(AbstractRegion.java:324)
at
org.apache.activemq.broker.region.RegionBroker.removeConsumer(RegionBroker.java:392)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:114)
at
org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:121)
at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:565)
at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:600)
at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:705)
at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1000)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:74)
at
org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:925)
Caused by: java.io.IOException: An attempt by a client to checkout a
Connection has timed out.
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doGetMessageCount(DefaultJDBCAdapter.java:687)
at
org.apache.activemq.store.jdbc.JDBCMessageStore.getMessageCount(JDBCMessageStore.java:208)
at
org.apache.activemq.store.ProxyMessageStore.getMessageCount(ProxyMessageStore.java:79)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.getStoreSize(QueueStorePrefetch.java:63)
... 16 more
Caused by: java.sql.SQLException: An attempt by a client to checkout a
Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:65)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:527)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
... 20 more
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out
while waiting to acquire a resource from com.mchange.v2.resourcepo
ResourcePool@d964af -- timeout at awaitAvailable()
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1317)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
... 22 more
INFO TransportConnector - Connector openwire Stopped
bsnyder wrote:
>
> On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com> wrote:
>>
>> Thanks for the response Bruce. :)
>>
>> To my knowledge we aren't using a connection pooler. We're just
>> configuring
>> the activemq xml and letting activeMQ do its thing. One of my colleagues
>> mentioned SNAC since we are using Sql Server, not sure if that is similar
>> to
>> a connection pooler or not.
>
> The job of a JDBC connection pooler is to keep a pool of JDBC
> connections available for use by a Java application. IIRC, SNAC stands
> for SQL Native Client which has nothing to do with a JDBC connection
> pooler.
>
>> In our scenario during a Sql Server failover it may take 15 minutes. We
>> aren't using a high availability cluster. Does a connection pooler give
>> the
>> capability of dealing with this so that ActiveMQ doesn't shutdown?
>
> One of the options in a good JDBC connection pooler is the ability to
> test JDBC connections as it hands them out to the Java application. If
> the connection is stale for any reason the JDBC connection pooler will
> evict that connection from the pool and try the next one until it
> reaches one that is alive. To my knowledge, some JDBC connection
> poolers provide a the ability to configure some attributes like number
> of retries, etc. E.g., here is some information about what c3p0
> offers:
>
> http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
>
> The one big caveat I will mention is that longer outage periods such
> as the 15 minute outage you mention are not typically handled by a
> JDBC connection pooler as they're mainly geared toward handling
> momentary outages. As an example, notice that the default value for
> acquireRetryDelay in c3p0 is only one second:
>
> http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
>
> But you could certainly configure this to any value you want. I
> encourage you to test this in your environment before deploying to
> production.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.apache.org/
> Apache Camel - http://camel.apache.org/
> Apache ServiceMix - http://servicemix.apache.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22466448.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, Mar 10, 2009 at 4:40 PM, magellings <ma...@qg.com> wrote:
>
> Thanks for the response Bruce. :)
>
> To my knowledge we aren't using a connection pooler. We're just configuring
> the activemq xml and letting activeMQ do its thing. One of my colleagues
> mentioned SNAC since we are using Sql Server, not sure if that is similar to
> a connection pooler or not.
The job of a JDBC connection pooler is to keep a pool of JDBC
connections available for use by a Java application. IIRC, SNAC stands
for SQL Native Client which has nothing to do with a JDBC connection
pooler.
> In our scenario during a Sql Server failover it may take 15 minutes. We
> aren't using a high availability cluster. Does a connection pooler give the
> capability of dealing with this so that ActiveMQ doesn't shutdown?
One of the options in a good JDBC connection pooler is the ability to
test JDBC connections as it hands them out to the Java application. If
the connection is stale for any reason the JDBC connection pooler will
evict that connection from the pool and try the next one until it
reaches one that is alive. To my knowledge, some JDBC connection
poolers provide a the ability to configure some attributes like number
of retries, etc. E.g., here is some information about what c3p0
offers:
http://www.mchange.com/projects/c3p0/index.html#configuring_recovery
The one big caveat I will mention is that longer outage periods such
as the 15 minute outage you mention are not typically handled by a
JDBC connection pooler as they're mainly geared toward handling
momentary outages. As an example, notice that the default value for
acquireRetryDelay in c3p0 is only one second:
http://www.mchange.com/projects/c3p0/index.html#acquireRetryDelay
But you could certainly configure this to any value you want. I
encourage you to test this in your environment before deploying to
production.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/
Blog: http://bruceblog.org/
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by magellings <ma...@qg.com>.
Thanks for the response Bruce. :)
To my knowledge we aren't using a connection pooler. We're just configuring
the activemq xml and letting activeMQ do its thing. One of my colleagues
mentioned SNAC since we are using Sql Server, not sure if that is similar to
a connection pooler or not.
In our scenario during a Sql Server failover it may take 15 minutes. We
aren't using a high availability cluster. Does a connection pooler give the
capability of dealing with this so that ActiveMQ doesn't shutdown?
bsnyder wrote:
>
> On Tue, Mar 10, 2009 at 12:15 PM, magellings <ma...@qg.com> wrote:
>>
>> Hello.
>>
>> Does anyone know if there is logic in activeMQ to retry connecting to a
>> database if JDBC master/slave is configured? Perhaps there is some
>> configuration I'm missing. Right now we log ship from the active Sql
>> Server
>> database to the passive Sql Server database. During maintenance it would
>> be
>> nice to fail to the passive database without impacting the ActiveMQ
>> broker
>> and requiring a restart. When I tested I just stopped the sql server
>> windows service and the active ActiveMQ broker shut down requiring a
>> manual
>> startup.
>>
>> Current JDBC master/slave configuration snippets:
>>
>>
>> <persistenceAdapter>
>> <jdbcPersistenceAdapter dataSource="#mssql-ds"/>
>> </persistenceAdapter>
>>
>> ...
>>
>> <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource"
>> destroy-method="close">
>> <property name="driverClassName"
>> value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
>> <property name="url"
>> value="jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$word"/>
>> <property name="username" value="activemq"/>
>> <property name="password" value="pa$$word"/>
>> </bean>
>
> This is commonly the job of the connection pooler that you're using. A
> good connection pooler will test the connections when they're handed
> out from the pool to make sure that they're still in good shape.
>
> I know that c3p0 (http://c3p0.sf.net/) handles this via the
> c3p0.testConnectionOnCheckout attribute in its configuration. There
> are other connection poolers that handle this as well including the
> new Tomcat JDBC pooler
> (http://tomcat.apache.org/dev/dist/jdbc-pool/1.0-beta/v1.0.15-beta/)
> which is a rewrite of Commons DBCP (because DBCP is horribly
> unmaintained and broken).
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.apache.org/
> Apache Camel - http://camel.apache.org/
> Apache ServiceMix - http://servicemix.apache.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/Is-there-JDBC-master-slave-retry-logic-if-database-goes-down--tp22440338p22445136.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Is there JDBC master/slave retry logic if database goes down?
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, Mar 10, 2009 at 12:15 PM, magellings <ma...@qg.com> wrote:
>
> Hello.
>
> Does anyone know if there is logic in activeMQ to retry connecting to a
> database if JDBC master/slave is configured? Perhaps there is some
> configuration I'm missing. Right now we log ship from the active Sql Server
> database to the passive Sql Server database. During maintenance it would be
> nice to fail to the passive database without impacting the ActiveMQ broker
> and requiring a restart. When I tested I just stopped the sql server
> windows service and the active ActiveMQ broker shut down requiring a manual
> startup.
>
> Current JDBC master/slave configuration snippets:
>
>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataSource="#mssql-ds"/>
> </persistenceAdapter>
>
> ...
>
> <bean id="mssql-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName"
> value="com.microsoft.sqlserver.jdbc.SQLServerDriver"/>
> <property name="url"
> value="jdbc:sqlserver://QG076619.qg.com:1433;databaseName=activedb;user=sa;password=pa$$word"/>
> <property name="username" value="activemq"/>
> <property name="password" value="pa$$word"/>
> </bean>
This is commonly the job of the connection pooler that you're using. A
good connection pooler will test the connections when they're handed
out from the pool to make sure that they're still in good shape.
I know that c3p0 (http://c3p0.sf.net/) handles this via the
c3p0.testConnectionOnCheckout attribute in its configuration. There
are other connection poolers that handle this as well including the
new Tomcat JDBC pooler
(http://tomcat.apache.org/dev/dist/jdbc-pool/1.0-beta/v1.0.15-beta/)
which is a rewrite of Commons DBCP (because DBCP is horribly
unmaintained and broken).
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.apache.org/
Apache Camel - http://camel.apache.org/
Apache ServiceMix - http://servicemix.apache.org/
Blog: http://bruceblog.org/