You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Veaceslav Doina (JIRA)" <ji...@apache.org> on 2018/11/13 12:23:00 UTC
[jira] [Commented] (AMQ-5238) Add JDBC Persistence for Scheduler
[ https://issues.apache.org/jira/browse/AMQ-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685119#comment-16685119 ]
Veaceslav Doina commented on AMQ-5238:
--------------------------------------
Hello,
Tried to configure HA using JDBC and Amazon EFS for scheduled messages.
Today it worked fine. On the restart, new master also acquire lock on shared storage for scheduled messages:
----
[root@broker-1 centos]# cat /opt/activemq/data/activemq.log
2018-11-13 12:10:18,185 | INFO | Attempting to acquire the exclusive lock to become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:10:18,198 | INFO | Becoming the master on dataSource: org.apache.commons.dbcp2.BasicDataSource@4a8355dd | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:10:18,200 | INFO | PListStore:[/activemq-data/broker-1/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-13 12:10:18,358 | INFO | Scheduler Store version 1 loaded | org.apache.activemq.store.kahadb.scheduler.JobSchedulerKahaDBMetaData | main
2018-11-13 12:10:18,469 | INFO | Recovering from the scheduled job journal @3:2315632 | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-13 12:10:18,474 | INFO | Recovery replayed 1 operations from the journal in 0.005 seconds. | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-13 12:10:18,520 | INFO | JobSchedulerStore: /efs/scheduler started. | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-13 12:10:18,597 | INFO | Apache ActiveMQ 5.15.7 (broker-1, ID:broker-1-38641-1542111018366-1:1) is starting | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,689 | INFO | Listening for connections at: tcp://broker-1:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,691 | INFO | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,694 | INFO | Listening for connections at: amqp://broker-1:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,696 | INFO | Connector amqp started | org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,699 | INFO | Listening for connections at: stomp://broker-1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,701 | INFO | Connector stomp started | org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,704 | INFO | Listening for connections at: mqtt://broker-1:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,705 | INFO | Connector mqtt started | org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,712 | INFO | Starting Jetty server | org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-13 12:10:18,742 | INFO | Creating Jetty connector | org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-13 12:10:18,810 | WARN | ServletContext@o.e.j.s.ServletContextHandler@4a9f80d3\{/,null,STARTING} has uncovered http methods for path: / | org.eclipse.jetty.security.SecurityHandler | main
2018-11-13 12:10:18,845 | INFO | Listening for connections at ws://broker-1:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.ws.WSTransportServer | main
2018-11-13 12:10:18,850 | INFO | Connector ws started | org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,851 | INFO | Apache ActiveMQ 5.15.7 (broker-1, ID:broker-1-38641-1542111018366-1:1) started | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,852 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,854 | WARN | Store limit is 102400 mb (current store usage is 0 mb). The data directory: / only has 13135 mb of usable space. - resetting to maximum available disk space: 13135 mb | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,855 | WARN | Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: / only has 13135 mb of usable space. - resetting to maximum available disk space: 13135 mb | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:19,338 | INFO | No Spring WebApplicationInitializer types detected on classpath | /admin | main
2018-11-13 12:10:19,568 | INFO | ActiveMQ WebConsole available at http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2018-11-13 12:10:19,568 | INFO | ActiveMQ Jolokia REST API available at http://0.0.0.0:8161/api/jolokia/ | org.apache.activemq.web.WebConsoleStarter | main
2018-11-13 12:10:19,617 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2018-11-13 12:10:19,831 | INFO | No Spring WebApplicationInitializer types detected on classpath | /api | main
2018-11-13 12:10:19,906 | INFO | jolokia-agent: Using policy access restrictor classpath:/jolokia-access.xml | /api | main
[root@broker-2 centos]# cat /opt/activemq/data/activemq.log
2018-11-13 12:10:17,854 | INFO | Apache ActiveMQ 5.15.7 (broker-2, ID:broker-2-41696-1542110982743-1:1) is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:17,857 | INFO | Connector openwire stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,858 | INFO | socketQueue interrupted - stopping | org.apache.activemq.transport.tcp.TcpTransportServer | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2018-11-13 12:10:17,859 | INFO | Could not accept connection during shutdown : {} | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)[:1.8.0_192]
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)[:1.8.0_192]
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)[:1.8.0_192]
at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:494)[activemq-client-5.15.7.jar:5.15.7]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_192]
2018-11-13 12:10:17,862 | INFO | Connector amqp stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,863 | INFO | Connector stomp stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,871 | INFO | Connector mqtt stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,878 | INFO | Connector ws stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:18,105 | INFO | JobSchedulerStore: /efs/scheduler stopped. | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | ActiveMQ ShutdownHook
2018-11-13 12:10:18,115 | INFO | PListStore:[/activemq-data/broker-2/tmp_storage] stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ ShutdownHook
2018-11-13 12:10:18,123 | INFO | Apache ActiveMQ 5.15.7 (broker-2, ID:broker-2-41696-1542110982743-1:1) uptime 39.918 seconds | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:18,124 | INFO | Apache ActiveMQ 5.15.7 (broker-2, ID:broker-2-41696-1542110982743-1:1) is shutdown | org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:18,124 | INFO | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Tue Nov 13 12:09:37 UTC 2018]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | ActiveMQ ShutdownHook
2018-11-13 12:10:18,131 | INFO | Destroying Spring FrameworkServlet 'dispatcher' | /admin | ActiveMQ ShutdownHook
2018-11-13 12:10:19,693 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Tue Nov 13 12:10:19 UTC 2018]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-13 12:10:20,687 | INFO | JobScheduler using directory: /efs/scheduler | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:20,712 | INFO | Using Persistence Adapter: JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:21,287 | INFO | Database adapter driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-13 12:11:12,437 | INFO | Database lock driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-13 12:11:12,439 | INFO | Attempting to acquire the exclusive lock to become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:12:03,469 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:13:04,485 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:14:05,500 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:15:06,514 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:16:07,530 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:17:08,544 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
----
Yesterday have some strange behavior with such configuration, when both brokers tried to get lock without success and as a resul both dead:
----
[root@broker-1 centos]# cat /opt/activemq/data/activemq.log
2018-11-12 16:40:50,550 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Mon Nov 12 16:40:50 UTC 2018]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-12 16:40:51,641 | INFO | JobScheduler using directory: /efs/scheduler | org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:51,664 | INFO | Using Persistence Adapter: JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) | org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:52,374 | INFO | Database adapter driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:41:43,495 | INFO | Database lock driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:41:43,497 | INFO | Attempting to acquire the exclusive lock to become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:42:34,522 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:43:35,535 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:44:36,549 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:45:37,562 | INFO | Failed to acquire lock. Sleeping for 10000 milli(s) before trying again... | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
[root@broker-2 opt]# cat /opt/activemq/data/activemq.log
2018-11-12 16:40:02,638 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Mon Nov 12 16:40:02 UTC 2018]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-12 16:40:03,637 | INFO | JobScheduler using directory: /efs/scheduler | org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:03,657 | INFO | Using Persistence Adapter: JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) | org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:04,299 | INFO | Database adapter driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:40:04,414 | INFO | Database lock driver override not found for : [mysql_connector_j]. Will use default implementation. | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:40:04,416 | INFO | Attempting to acquire the exclusive lock to become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:40:04,433 | INFO | Becoming the master on dataSource: org.apache.commons.dbcp2.BasicDataSource@4a8355dd | org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:40:04,436 | INFO | PListStore:[/activemq-data/broker-2/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-12 16:40:04,450 | INFO | Database /efs/scheduler/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired | org.apache.activemq.store.SharedFileLocker | main
----
> Add JDBC Persistence for Scheduler
> ----------------------------------
>
> Key: AMQ-5238
> URL: https://issues.apache.org/jira/browse/AMQ-5238
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Job Scheduler
> Affects Versions: 5.9.0
> Environment: JBoss A-MQ 6.0
> Reporter: Jason Shepherd
> Priority: Minor
>
> AMQ-3024 added support for non-KahaDB persistence of scheduled jobs, but a JDBC alternative has yet to be added.
> If we're using JDBC persistence adapter, we want all persistence for the broker to be done for JDBC. Ideally this would also allows us to view historical statistics of the scheduler as per AMQ-5192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)