You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ramesh Venkitaswaran <ve...@gmail.com> on 2020/08/07 08:07:52 UTC
Activemq 5.15 startup error
Hello
We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. It's
a very old system. However last night we started to get these errors and
the broker did not come up. Any ideas on how to fix this would be
appreciated. I can provide further log messages as needed, if sufficient
direction is provided. Thank you!
Here are the logs
2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
is starting |
org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,732 | INFO | Listening for connections at:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
.maxFrameSize=104857600 |
org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,733 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,823 | INFO | Listening for connections at:
ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
at
org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)[:1.8.0_232]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
at
org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
2020-08-07 02:44:46,669 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date [Fri
Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated
LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,10.184.11.9:2181,
10.184.11.7:2181//activemq/ap
ps/mkt/lpf/prod-usc1a/BROKER01P] | org.apache.activemq.broker.BrokerService
| main
2020-08-07 02:44:51,492 | INFO | Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,494 | INFO | Client
environment:host.name=mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,496 | INFO | Client environment:java.version=1.8.0_232
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle
Corporation | org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,500 | INFO | Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,504 | INFO | Client
environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
|
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,505 | INFO | Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
| org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,506 | INFO | Client
environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,509 | INFO | Client
environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,510 | INFO | Client
environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,514 | INFO | Initiating client connection,
connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
sessionTimeout=2000 watcher=org.apache.activem
q.leveldb.replicated.groups.ZKClient@36060e |
org.apache.zookeeper.ZooKeeper | main
2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state
change dispatcher thread
2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section
named 'Client' was found in specified JAAS
configuration file:
'/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue
connection to Zookeeper server without SASL authentication, if Zookeeper
server a
llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
10.184.11.9:2181)
2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
main-SendThread(10.184.11.9:2181)
2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed |
org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
2020-08-07 02:44:51,606 | INFO | Socket connection established to
10.184.11.9/10.184.11.9:2181, initiating session |
org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
11.9:2181)
2020-08-07 02:44:51,622 | INFO | Session establishment complete on server
10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated
timeout = 4000 | org.apache.
zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
2020-08-07 02:44:51,946 | INFO | Promoted to master |
org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
2020-08-07 02:44:52,110 | WARN | listeners are taking too long to process
the events |
org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
main-EventThread
2020-08-07 02:44:52,111 | WARN | listeners are taking too long to process
the events |
org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
main-EventThread
2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
BrokerService[BROKER01P] Task-1
2020-08-07 02:44:54,349 | INFO | Master started:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
| org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
BrokerService[BROKER01P] Task-2
2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
is starting | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,732 | INFO | Listening for connections at:
tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,733 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,823 | INFO | Listening for connections at:
ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
| org.apache.activemq.transport.TransportServerThreadSupport | main
2020-08-07 02:44:54,825 | INFO | Connector ssl started |
org.apache.activemq.broker.TransportConnector | main
2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
started | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,827 | INFO | For help or more information please see:
http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store
usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb only
has 42345 mb of usable space. - resetting to maximum available disk space:
42345 mb | org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
(current store usage is 25900 mb). The data directory:
/var/lib/activemq/data only has 42345 mb of usable space. - resetting to
maximum available disk space: 42345 mb |
org.apache.activemq.broker.BrokerService | main
2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due to
exception, java.io.EOFException: File
'/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 |
org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
handler.
java.io.EOFException: File
'/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
at org.apache.activemq.console.Main.main(Main.java:115)
Re: Activemq 5.15 startup error
Posted by Ramesh Venkitaswaran <ve...@gmail.com>.
Thanks everyone. I think this is what I would need to convince the team to
really get out of this. I appreciate the time you put into this and guiding
me appropriately.
Thanks again.
On Fri, Aug 7, 2020 at 10:24 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> In terms of exporting I am not sure. I personally have never looked at
> LevelDB. In terms of the error itself it seems like some of the files got
> corrupt so not sure if it's possible to recover. If you want to export the
> data you would probably need to write your own export tool to parse the
> LevelDB structure. Or if you manage to get the broker running again and
> want to migrate you could write a client to move data to a new broker.
>
> Tim is right on the reasoning for deprecation and upcoming removal.
> LevelDB has been deprecated for 4 years now and as far as I'm aware no one
> still around really knows anything about it and no one is going to answer
> any questions on it. There was no community support or knowledge (it was
> written in Scala which was one major issue) and the people who wrote it
> have long stopped contributing to the project so it didn't make sense to
> continue supporting it if no one was around to do so.
>
>
> On Fri, Aug 7, 2020 at 11:20 AM Tim Bain <tb...@alumni.duke.edu> wrote:
>
> > LevelDB support was removed in large part due to the absence of
> developers
> > willing to maintain the code and volunteers willing to provide support
> for
> > it on this mailing list. In the 5+ years I've been involved in this
> mailing
> > list, I don't recall ever seeing a LevelDB question answered (and I've
> > definitely seen them asked).
> >
> > When Chris says that LevelDB is unsupported, he means that it's
> > UNSUPPORTED. If you're not prepared to provide your own support for
> > whatever problems you encounter (leveraging the open-source codebase),
> you
> > shouldn't be using it, and should switch to KahaDB (or Artemis, as you
> > said, but KahaDB gives you a short-term solution for minimal effort). I'm
> > sorry to be the bearer of bad news, especially since you might not have
> > been aware of LevelDB's status until now, but you're on your own for
> > LevelDB issues.
> >
> > Good luck.
> >
> > Tim
> >
> > On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran <
> venkitaswaran@gmail.com
> > >
> > wrote:
> >
> > > Any ideas? Or, is there a way to export these messages so that I can
> > > recreate the broker data directory and import them?
> > >
> > > On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran <
> > > venkitaswaran@gmail.com>
> > > wrote:
> > >
> > > > Understood, that LevelDB is deprecated, and we are trying to get this
> > > team
> > > > to move to Artemis. However we still need this broker to run while
> the
> > > > migration is going on.
> > > >
> > > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon <
> > > > christopher.l.shannon@gmail.com> wrote:
> > > >
> > > >> LevelDB support has been deprecated for many years now and is going
> to
> > > be
> > > >> removed in the next version of ActiveMQ (5.17.0) so I would suggest
> > > trying
> > > >> to migrate over to KahaDB.
> > > >>
> > > >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <
> > > >> venkitaswaran@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hello
> > > >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level
> > DB.
> > > >> It's
> > > >> > a very old system. However last night we started to get these
> errors
> > > and
> > > >> > the broker did not come up. Any ideas on how to fix this would be
> > > >> > appreciated. I can provide further log messages as needed, if
> > > sufficient
> > > >> > direction is provided. Thank you!
> > > >> >
> > > >> > Here are the logs
> > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0
> (BROKER01P,
> > > >> >
> > > >> >
> > > >>
> > >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > > >> > is starting |
> > > >> > org.apache.activemq.broker.BrokerService | main
> > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
> > > >> > .maxFrameSize=104857600 |
> > > >> > org.apache.activemq.transport.TransportServerThreadSupport | main
> > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > > >> > org.apache.activemq.broker.TransportConnector | main
> > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > >> > Method)[:1.8.0_232]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
> > > >> > at
> > > java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
> > > >> > at
> > > >> >
> > > >>
> > >
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
> > > >> > 2020-08-07 02:44:46,669 | INFO | Refreshing
> > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup
> > date
> > > >> [Fri
> > > >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
> > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> > > >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter:
> > > Replicated
> > > >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,
> > > >> 10.184.11.9:2181,
> > > >> > 10.184.11.7:2181//activemq/ap
> > > >> > ps/mkt/lpf/prod-usc1a/BROKER01P] |
> > > >> org.apache.activemq.broker.BrokerService
> > > >> > | main
> > > >> > 2020-08-07 02:44:51,492 | INFO | Client
> > > >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> > 09:09
> > > >> GMT
> > > >> > | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,494 | INFO | Client
> > > >> > environment:host.name
> > > >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
> > > >> > | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,496 | INFO | Client
> > > >> environment:java.version=1.8.0_232
> > > >> > | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,499 | INFO | Client
> > > environment:java.vendor=Oracle
> > > >> > Corporation | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,500 | INFO | Client
> > > >> >
> > > >> >
> > > >>
> > >
> >
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
> > > >> > | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,504 | INFO | Client
> > > >> >
> > > >> >
> > > >>
> > >
> >
> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
> > > >> > |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,505 | INFO | Client
> > > >> >
> > > >> >
> > > >>
> > >
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > >> > | org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,506 | INFO | Client
> > > >> >
> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,507 | INFO | Client
> > > environment:java.compiler=<NA>
> > > >> |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name
> =Linux
> > |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,508 | INFO | Client
> environment:os.arch=amd64 |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,509 | INFO | Client
> > > >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name
> > > =activemq
> > > >> |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,510 | INFO | Client
> > > >> > environment:user.home=/home/activemq |
> > org.apache.zookeeper.ZooKeeper
> > > |
> > > >> > main
> > > >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
> > > >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
> > > >> > sessionTimeout=2000 watcher=org.apache.activem
> > > >> > q.leveldb.replicated.groups.ZKClient@36060e |
> > > >> > org.apache.zookeeper.ZooKeeper | main
> > > >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
> > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper
> > > state
> > > >> > change dispatcher thread
> > > >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
> > > >> > javax.security.auth.login.LoginException: No JAAS configuration
> > > section
> > > >> > named 'Client' was found in specified JAAS
> > > >> > configuration file:
> > > >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will
> > > continue
> > > >> > connection to Zookeeper server without SASL authentication, if
> > > Zookeeper
> > > >> > server a
> > > >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
> > > >> > 10.184.11.9:2181)
> > > >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to
> > server
> > > >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
> > > >> > main-SendThread(10.184.11.9:2181)
> > > >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state:
> > AuthFailed
> > > |
> > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient |
> > > >> main-EventThread
> > > >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to
> > > >> > 10.184.11.9/10.184.11.9:2181, initiating session |
> > > >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
> > > >> > 11.9:2181)
> > > >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete
> on
> > > >> server
> > > >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001,
> > > negotiated
> > > >> > timeout = 4000 | org.apache.
> > > >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
> > > >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master |
> > > >> > org.apache.activemq.leveldb.replicated.MasterElector |
> > > main-EventThread
> > > >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to
> > > >> process
> > > >> > the events |
> > > >> >
> org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport
> > |
> > > >> > main-EventThread
> > > >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to
> > > >> process
> > > >> > the events |
> > > >> >
> org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport
> > |
> > > >> > main-EventThread
> > > >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
> > > >> > implementation. | org.apache.activemq.leveldb.LevelDBClient |
> > ActiveMQ
> > > >> > BrokerService[BROKER01P] Task-1
> > > >> > 2020-08-07 02:44:54,349 | INFO | Master started:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
> > > >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
> > > >> > BrokerService[BROKER01P] Task-2
> > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0
> (BROKER01P,
> > > >> >
> > > >> >
> > > >>
> > >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > > >> > is starting | org.apache.activemq.broker.BrokerService | main
> > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> > > >> > | org.apache.activemq.transport.TransportServerThreadSupport |
> main
> > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > > >> > org.apache.activemq.broker.TransportConnector | main
> > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> > > >> >
> > > >> >
> > > >>
> > >
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
> > > >> > | org.apache.activemq.transport.TransportServerThreadSupport |
> main
> > > >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
> > > >> > org.apache.activemq.broker.TransportConnector | main
> > > >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0
> (BROKER01P,
> > > >> >
> > > >> >
> > > >>
> > >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > > >> > started | org.apache.activemq.broker.BrokerService | main
> > > >> > 2020-08-07 02:44:54,827 | INFO | For help or more information
> > please
> > > >> see:
> > > >> > http://activemq.apache.org |
> > > org.apache.activemq.broker.BrokerService |
> > > >> > main
> > > >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb
> (current
> > > >> store
> > > >> > usage is 25900 mb). The data directory:
> > /var/lib/activemq/data/leveldb
> > > >> only
> > > >> > has 42345 mb of usable space. - resetting to maximum available
> disk
> > > >> space:
> > > >> > 42345 mb | org.apache.activemq.broker.BrokerService | main
> > > >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200
> mb
> > > >> > (current store usage is 25900 mb). The data directory:
> > > >> > /var/lib/activemq/data only has 42345 mb of usable space. -
> > resetting
> > > to
> > > >> > maximum available disk space: 42345 mb |
> > > >> > org.apache.activemq.broker.BrokerService | main
> > > >> > 2020-08-07 02:44:55,677 | INFO | Stopping
> BrokerService[BROKER01P]
> > > due
> > > >> to
> > > >> > exception, java.io.EOFException: File
> > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> > > 273453365
> > > >> |
> > > >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB
> > > IOException
> > > >> > handler.
> > > >> > java.io.EOFException: File
> > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> > > 273453365
> > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > >> > at
> > > >> >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> > at java.lang.reflect.Method.invoke(Method.java:498)
> > > >> > at
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> > > >> > at org.apache.activemq.console.Main.main(Main.java:115)
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Activemq 5.15 startup error
Posted by Christopher Shannon <ch...@gmail.com>.
In terms of exporting I am not sure. I personally have never looked at
LevelDB. In terms of the error itself it seems like some of the files got
corrupt so not sure if it's possible to recover. If you want to export the
data you would probably need to write your own export tool to parse the
LevelDB structure. Or if you manage to get the broker running again and
want to migrate you could write a client to move data to a new broker.
Tim is right on the reasoning for deprecation and upcoming removal.
LevelDB has been deprecated for 4 years now and as far as I'm aware no one
still around really knows anything about it and no one is going to answer
any questions on it. There was no community support or knowledge (it was
written in Scala which was one major issue) and the people who wrote it
have long stopped contributing to the project so it didn't make sense to
continue supporting it if no one was around to do so.
On Fri, Aug 7, 2020 at 11:20 AM Tim Bain <tb...@alumni.duke.edu> wrote:
> LevelDB support was removed in large part due to the absence of developers
> willing to maintain the code and volunteers willing to provide support for
> it on this mailing list. In the 5+ years I've been involved in this mailing
> list, I don't recall ever seeing a LevelDB question answered (and I've
> definitely seen them asked).
>
> When Chris says that LevelDB is unsupported, he means that it's
> UNSUPPORTED. If you're not prepared to provide your own support for
> whatever problems you encounter (leveraging the open-source codebase), you
> shouldn't be using it, and should switch to KahaDB (or Artemis, as you
> said, but KahaDB gives you a short-term solution for minimal effort). I'm
> sorry to be the bearer of bad news, especially since you might not have
> been aware of LevelDB's status until now, but you're on your own for
> LevelDB issues.
>
> Good luck.
>
> Tim
>
> On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran <venkitaswaran@gmail.com
> >
> wrote:
>
> > Any ideas? Or, is there a way to export these messages so that I can
> > recreate the broker data directory and import them?
> >
> > On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran <
> > venkitaswaran@gmail.com>
> > wrote:
> >
> > > Understood, that LevelDB is deprecated, and we are trying to get this
> > team
> > > to move to Artemis. However we still need this broker to run while the
> > > migration is going on.
> > >
> > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon <
> > > christopher.l.shannon@gmail.com> wrote:
> > >
> > >> LevelDB support has been deprecated for many years now and is going to
> > be
> > >> removed in the next version of ActiveMQ (5.17.0) so I would suggest
> > trying
> > >> to migrate over to KahaDB.
> > >>
> > >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <
> > >> venkitaswaran@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello
> > >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level
> DB.
> > >> It's
> > >> > a very old system. However last night we started to get these errors
> > and
> > >> > the broker did not come up. Any ideas on how to fix this would be
> > >> > appreciated. I can provide further log messages as needed, if
> > sufficient
> > >> > direction is provided. Thank you!
> > >> >
> > >> > Here are the logs
> > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> > >> >
> > >> >
> > >>
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > >> > is starting |
> > >> > org.apache.activemq.broker.BrokerService | main
> > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> > >> >
> > >> >
> > >>
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
> > >> > .maxFrameSize=104857600 |
> > >> > org.apache.activemq.transport.TransportServerThreadSupport | main
> > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > >> > org.apache.activemq.broker.TransportConnector | main
> > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> > >> >
> > >> >
> > >>
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > >> > Method)[:1.8.0_232]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
> > >> > at
> > java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
> > >> > at
> > >> >
> > >> >
> > >>
> >
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
> > >> > at
> > >> >
> > >>
> > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
> > >> > 2020-08-07 02:44:46,669 | INFO | Refreshing
> > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup
> date
> > >> [Fri
> > >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
> > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> > >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter:
> > Replicated
> > >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,
> > >> 10.184.11.9:2181,
> > >> > 10.184.11.7:2181//activemq/ap
> > >> > ps/mkt/lpf/prod-usc1a/BROKER01P] |
> > >> org.apache.activemq.broker.BrokerService
> > >> > | main
> > >> > 2020-08-07 02:44:51,492 | INFO | Client
> > >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09
> > >> GMT
> > >> > | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,494 | INFO | Client
> > >> > environment:host.name
> > >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
> > >> > | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,496 | INFO | Client
> > >> environment:java.version=1.8.0_232
> > >> > | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,499 | INFO | Client
> > environment:java.vendor=Oracle
> > >> > Corporation | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,500 | INFO | Client
> > >> >
> > >> >
> > >>
> >
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
> > >> > | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,504 | INFO | Client
> > >> >
> > >> >
> > >>
> >
> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
> > >> > |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,505 | INFO | Client
> > >> >
> > >> >
> > >>
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > >> > | org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,506 | INFO | Client
> > >> >
> environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,507 | INFO | Client
> > environment:java.compiler=<NA>
> > >> |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux
> |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,509 | INFO | Client
> > >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name
> > =activemq
> > >> |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,510 | INFO | Client
> > >> > environment:user.home=/home/activemq |
> org.apache.zookeeper.ZooKeeper
> > |
> > >> > main
> > >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
> > >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
> > >> > sessionTimeout=2000 watcher=org.apache.activem
> > >> > q.leveldb.replicated.groups.ZKClient@36060e |
> > >> > org.apache.zookeeper.ZooKeeper | main
> > >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
> > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper
> > state
> > >> > change dispatcher thread
> > >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
> > >> > javax.security.auth.login.LoginException: No JAAS configuration
> > section
> > >> > named 'Client' was found in specified JAAS
> > >> > configuration file:
> > >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will
> > continue
> > >> > connection to Zookeeper server without SASL authentication, if
> > Zookeeper
> > >> > server a
> > >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
> > >> > 10.184.11.9:2181)
> > >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to
> server
> > >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
> > >> > main-SendThread(10.184.11.9:2181)
> > >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state:
> AuthFailed
> > |
> > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient |
> > >> main-EventThread
> > >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to
> > >> > 10.184.11.9/10.184.11.9:2181, initiating session |
> > >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
> > >> > 11.9:2181)
> > >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on
> > >> server
> > >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001,
> > negotiated
> > >> > timeout = 4000 | org.apache.
> > >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
> > >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master |
> > >> > org.apache.activemq.leveldb.replicated.MasterElector |
> > main-EventThread
> > >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to
> > >> process
> > >> > the events |
> > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport
> |
> > >> > main-EventThread
> > >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to
> > >> process
> > >> > the events |
> > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport
> |
> > >> > main-EventThread
> > >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
> > >> > implementation. | org.apache.activemq.leveldb.LevelDBClient |
> ActiveMQ
> > >> > BrokerService[BROKER01P] Task-1
> > >> > 2020-08-07 02:44:54,349 | INFO | Master started:
> > >> >
> > >> >
> > >>
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
> > >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
> > >> > BrokerService[BROKER01P] Task-2
> > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> > >> >
> > >> >
> > >>
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > >> > is starting | org.apache.activemq.broker.BrokerService | main
> > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> > >> >
> > >> >
> > >>
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > >> > org.apache.activemq.broker.TransportConnector | main
> > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> > >> >
> > >> >
> > >>
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
> > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> > >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
> > >> > org.apache.activemq.broker.TransportConnector | main
> > >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> > >> >
> > >> >
> > >>
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > >> > started | org.apache.activemq.broker.BrokerService | main
> > >> > 2020-08-07 02:44:54,827 | INFO | For help or more information
> please
> > >> see:
> > >> > http://activemq.apache.org |
> > org.apache.activemq.broker.BrokerService |
> > >> > main
> > >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current
> > >> store
> > >> > usage is 25900 mb). The data directory:
> /var/lib/activemq/data/leveldb
> > >> only
> > >> > has 42345 mb of usable space. - resetting to maximum available disk
> > >> space:
> > >> > 42345 mb | org.apache.activemq.broker.BrokerService | main
> > >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
> > >> > (current store usage is 25900 mb). The data directory:
> > >> > /var/lib/activemq/data only has 42345 mb of usable space. -
> resetting
> > to
> > >> > maximum available disk space: 42345 mb |
> > >> > org.apache.activemq.broker.BrokerService | main
> > >> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P]
> > due
> > >> to
> > >> > exception, java.io.EOFException: File
> > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> > 273453365
> > >> |
> > >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB
> > IOException
> > >> > handler.
> > >> > java.io.EOFException: File
> > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> > 273453365
> > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >> > at
> > >> >
> > >> >
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> > at java.lang.reflect.Method.invoke(Method.java:498)
> > >> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> > >> > at org.apache.activemq.console.Main.main(Main.java:115)
> > >> >
> > >>
> > >
> >
>
Re: Activemq 5.15 startup error
Posted by Tim Bain <tb...@alumni.duke.edu>.
LevelDB support was removed in large part due to the absence of developers
willing to maintain the code and volunteers willing to provide support for
it on this mailing list. In the 5+ years I've been involved in this mailing
list, I don't recall ever seeing a LevelDB question answered (and I've
definitely seen them asked).
When Chris says that LevelDB is unsupported, he means that it's
UNSUPPORTED. If you're not prepared to provide your own support for
whatever problems you encounter (leveraging the open-source codebase), you
shouldn't be using it, and should switch to KahaDB (or Artemis, as you
said, but KahaDB gives you a short-term solution for minimal effort). I'm
sorry to be the bearer of bad news, especially since you might not have
been aware of LevelDB's status until now, but you're on your own for
LevelDB issues.
Good luck.
Tim
On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran <ve...@gmail.com>
wrote:
> Any ideas? Or, is there a way to export these messages so that I can
> recreate the broker data directory and import them?
>
> On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran <
> venkitaswaran@gmail.com>
> wrote:
>
> > Understood, that LevelDB is deprecated, and we are trying to get this
> team
> > to move to Artemis. However we still need this broker to run while the
> > migration is going on.
> >
> > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon <
> > christopher.l.shannon@gmail.com> wrote:
> >
> >> LevelDB support has been deprecated for many years now and is going to
> be
> >> removed in the next version of ActiveMQ (5.17.0) so I would suggest
> trying
> >> to migrate over to KahaDB.
> >>
> >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <
> >> venkitaswaran@gmail.com>
> >> wrote:
> >>
> >> > Hello
> >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB.
> >> It's
> >> > a very old system. However last night we started to get these errors
> and
> >> > the broker did not come up. Any ideas on how to fix this would be
> >> > appreciated. I can provide further log messages as needed, if
> sufficient
> >> > direction is provided. Thank you!
> >> >
> >> > Here are the logs
> >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >> >
> >> >
> >>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> >> > is starting |
> >> > org.apache.activemq.broker.BrokerService | main
> >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> >> >
> >> >
> >>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
> >> > .maxFrameSize=104857600 |
> >> > org.apache.activemq.transport.TransportServerThreadSupport | main
> >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> >> > org.apache.activemq.broker.TransportConnector | main
> >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> >> >
> >> >
> >>
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > Method)[:1.8.0_232]
> >> > at
> >> >
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
> >> > at
> >> >
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
> >> > at
> java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
> >> > at
> >> >
> >> >
> >>
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
> >> > at
> >> >
> >>
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
> >> > 2020-08-07 02:44:46,669 | INFO | Refreshing
> >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date
> >> [Fri
> >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
> >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter:
> Replicated
> >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,
> >> 10.184.11.9:2181,
> >> > 10.184.11.7:2181//activemq/ap
> >> > ps/mkt/lpf/prod-usc1a/BROKER01P] |
> >> org.apache.activemq.broker.BrokerService
> >> > | main
> >> > 2020-08-07 02:44:51,492 | INFO | Client
> >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> >> GMT
> >> > | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,494 | INFO | Client
> >> > environment:host.name
> >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
> >> > | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,496 | INFO | Client
> >> environment:java.version=1.8.0_232
> >> > | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,499 | INFO | Client
> environment:java.vendor=Oracle
> >> > Corporation | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,500 | INFO | Client
> >> >
> >> >
> >>
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
> >> > | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,504 | INFO | Client
> >> >
> >> >
> >>
> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
> >> > |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,505 | INFO | Client
> >> >
> >> >
> >>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> >> > | org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,506 | INFO | Client
> >> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,507 | INFO | Client
> environment:java.compiler=<NA>
> >> |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,509 | INFO | Client
> >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name
> =activemq
> >> |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,510 | INFO | Client
> >> > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper
> |
> >> > main
> >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
> >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
> >> > sessionTimeout=2000 watcher=org.apache.activem
> >> > q.leveldb.replicated.groups.ZKClient@36060e |
> >> > org.apache.zookeeper.ZooKeeper | main
> >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
> >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper
> state
> >> > change dispatcher thread
> >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
> >> > javax.security.auth.login.LoginException: No JAAS configuration
> section
> >> > named 'Client' was found in specified JAAS
> >> > configuration file:
> >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will
> continue
> >> > connection to Zookeeper server without SASL authentication, if
> Zookeeper
> >> > server a
> >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
> >> > 10.184.11.9:2181)
> >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
> >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
> >> > main-SendThread(10.184.11.9:2181)
> >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed
> |
> >> > org.apache.activemq.leveldb.replicated.groups.ZKClient |
> >> main-EventThread
> >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to
> >> > 10.184.11.9/10.184.11.9:2181, initiating session |
> >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
> >> > 11.9:2181)
> >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on
> >> server
> >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001,
> negotiated
> >> > timeout = 4000 | org.apache.
> >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
> >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master |
> >> > org.apache.activemq.leveldb.replicated.MasterElector |
> main-EventThread
> >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to
> >> process
> >> > the events |
> >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> >> > main-EventThread
> >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to
> >> process
> >> > the events |
> >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> >> > main-EventThread
> >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
> >> > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
> >> > BrokerService[BROKER01P] Task-1
> >> > 2020-08-07 02:44:54,349 | INFO | Master started:
> >> >
> >> >
> >>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
> >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
> >> > BrokerService[BROKER01P] Task-2
> >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >> >
> >> >
> >>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> >> > is starting | org.apache.activemq.broker.BrokerService | main
> >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> >> >
> >> >
> >>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> >> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> >> > org.apache.activemq.broker.TransportConnector | main
> >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> >> >
> >> >
> >>
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
> >> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
> >> > org.apache.activemq.broker.TransportConnector | main
> >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >> >
> >> >
> >>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> >> > started | org.apache.activemq.broker.BrokerService | main
> >> > 2020-08-07 02:44:54,827 | INFO | For help or more information please
> >> see:
> >> > http://activemq.apache.org |
> org.apache.activemq.broker.BrokerService |
> >> > main
> >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current
> >> store
> >> > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb
> >> only
> >> > has 42345 mb of usable space. - resetting to maximum available disk
> >> space:
> >> > 42345 mb | org.apache.activemq.broker.BrokerService | main
> >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
> >> > (current store usage is 25900 mb). The data directory:
> >> > /var/lib/activemq/data only has 42345 mb of usable space. - resetting
> to
> >> > maximum available disk space: 42345 mb |
> >> > org.apache.activemq.broker.BrokerService | main
> >> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P]
> due
> >> to
> >> > exception, java.io.EOFException: File
> >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> 273453365
> >> |
> >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB
> IOException
> >> > handler.
> >> > java.io.EOFException: File
> >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset:
> 273453365
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at
> >> >
> >> >
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> > at
> >> >
> >> >
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> > at java.lang.reflect.Method.invoke(Method.java:498)
> >> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> >> > at org.apache.activemq.console.Main.main(Main.java:115)
> >> >
> >>
> >
>
Re: Activemq 5.15 startup error
Posted by Ramesh Venkitaswaran <ve...@gmail.com>.
Any ideas? Or, is there a way to export these messages so that I can
recreate the broker data directory and import them?
On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran <ve...@gmail.com>
wrote:
> Understood, that LevelDB is deprecated, and we are trying to get this team
> to move to Artemis. However we still need this broker to run while the
> migration is going on.
>
> On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon <
> christopher.l.shannon@gmail.com> wrote:
>
>> LevelDB support has been deprecated for many years now and is going to be
>> removed in the next version of ActiveMQ (5.17.0) so I would suggest trying
>> to migrate over to KahaDB.
>>
>> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <
>> venkitaswaran@gmail.com>
>> wrote:
>>
>> > Hello
>> > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB.
>> It's
>> > a very old system. However last night we started to get these errors and
>> > the broker did not come up. Any ideas on how to fix this would be
>> > appreciated. I can provide further log messages as needed, if sufficient
>> > direction is provided. Thank you!
>> >
>> > Here are the logs
>> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>> >
>> >
>> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
>> > is starting |
>> > org.apache.activemq.broker.BrokerService | main
>> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
>> >
>> >
>> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
>> > .maxFrameSize=104857600 |
>> > org.apache.activemq.transport.TransportServerThreadSupport | main
>> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
>> > org.apache.activemq.broker.TransportConnector | main
>> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
>> >
>> >
>> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
>> > at
>> >
>> >
>> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
>> > at
>> >
>> >
>> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
>> > at
>> >
>> >
>> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
>> > at
>> >
>> >
>> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
>> > at
>> >
>> >
>> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
>> > at
>> >
>> >
>> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)[:1.8.0_232]
>> > at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
>> > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
>> > at
>> >
>> >
>> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
>> > at
>> >
>> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
>> > 2020-08-07 02:44:46,669 | INFO | Refreshing
>> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date
>> [Fri
>> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
>> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
>> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated
>> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,
>> 10.184.11.9:2181,
>> > 10.184.11.7:2181//activemq/ap
>> > ps/mkt/lpf/prod-usc1a/BROKER01P] |
>> org.apache.activemq.broker.BrokerService
>> > | main
>> > 2020-08-07 02:44:51,492 | INFO | Client
>> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
>> GMT
>> > | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,494 | INFO | Client
>> > environment:host.name
>> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
>> > | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,496 | INFO | Client
>> environment:java.version=1.8.0_232
>> > | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle
>> > Corporation | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,500 | INFO | Client
>> >
>> >
>> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
>> > | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,504 | INFO | Client
>> >
>> >
>> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
>> > |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,505 | INFO | Client
>> >
>> >
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> > | org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,506 | INFO | Client
>> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA>
>> |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,509 | INFO | Client
>> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq
>> |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,510 | INFO | Client
>> > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper |
>> > main
>> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
>> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
>> > sessionTimeout=2000 watcher=org.apache.activem
>> > q.leveldb.replicated.groups.ZKClient@36060e |
>> > org.apache.zookeeper.ZooKeeper | main
>> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
>> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state
>> > change dispatcher thread
>> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
>> > javax.security.auth.login.LoginException: No JAAS configuration section
>> > named 'Client' was found in specified JAAS
>> > configuration file:
>> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue
>> > connection to Zookeeper server without SASL authentication, if Zookeeper
>> > server a
>> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
>> > 10.184.11.9:2181)
>> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
>> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
>> > main-SendThread(10.184.11.9:2181)
>> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed |
>> > org.apache.activemq.leveldb.replicated.groups.ZKClient |
>> main-EventThread
>> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to
>> > 10.184.11.9/10.184.11.9:2181, initiating session |
>> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
>> > 11.9:2181)
>> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on
>> server
>> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated
>> > timeout = 4000 | org.apache.
>> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
>> > 2020-08-07 02:44:51,946 | INFO | Promoted to master |
>> > org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
>> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to
>> process
>> > the events |
>> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
>> > main-EventThread
>> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to
>> process
>> > the events |
>> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
>> > main-EventThread
>> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
>> > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
>> > BrokerService[BROKER01P] Task-1
>> > 2020-08-07 02:44:54,349 | INFO | Master started:
>> >
>> >
>> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
>> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
>> > BrokerService[BROKER01P] Task-2
>> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>> >
>> >
>> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
>> > is starting | org.apache.activemq.broker.BrokerService | main
>> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
>> >
>> >
>> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
>> > | org.apache.activemq.transport.TransportServerThreadSupport | main
>> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
>> > org.apache.activemq.broker.TransportConnector | main
>> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
>> >
>> >
>> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
>> > | org.apache.activemq.transport.TransportServerThreadSupport | main
>> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
>> > org.apache.activemq.broker.TransportConnector | main
>> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>> >
>> >
>> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
>> > started | org.apache.activemq.broker.BrokerService | main
>> > 2020-08-07 02:44:54,827 | INFO | For help or more information please
>> see:
>> > http://activemq.apache.org | org.apache.activemq.broker.BrokerService |
>> > main
>> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current
>> store
>> > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb
>> only
>> > has 42345 mb of usable space. - resetting to maximum available disk
>> space:
>> > 42345 mb | org.apache.activemq.broker.BrokerService | main
>> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
>> > (current store usage is 25900 mb). The data directory:
>> > /var/lib/activemq/data only has 42345 mb of usable space. - resetting to
>> > maximum available disk space: 42345 mb |
>> > org.apache.activemq.broker.BrokerService | main
>> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due
>> to
>> > exception, java.io.EOFException: File
>> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
>> |
>> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
>> > handler.
>> > java.io.EOFException: File
>> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:498)
>> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
>> > at org.apache.activemq.console.Main.main(Main.java:115)
>> >
>>
>
Re: Activemq 5.15 startup error
Posted by Ramesh Venkitaswaran <ve...@gmail.com>.
Understood, that LevelDB is deprecated, and we are trying to get this team
to move to Artemis. However we still need this broker to run while the
migration is going on.
On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> LevelDB support has been deprecated for many years now and is going to be
> removed in the next version of ActiveMQ (5.17.0) so I would suggest trying
> to migrate over to KahaDB.
>
> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <
> venkitaswaran@gmail.com>
> wrote:
>
> > Hello
> > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB.
> It's
> > a very old system. However last night we started to get these errors and
> > the broker did not come up. Any ideas on how to fix this would be
> > appreciated. I can provide further log messages as needed, if sufficient
> > direction is provided. Thank you!
> >
> > Here are the logs
> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > is starting |
> > org.apache.activemq.broker.BrokerService | main
> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
> > .maxFrameSize=104857600 |
> > org.apache.activemq.transport.TransportServerThreadSupport | main
> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > org.apache.activemq.broker.TransportConnector | main
> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> >
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
> > at
> >
> >
> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
> > at
> >
> >
> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
> > at
> >
> >
> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
> > at
> >
> >
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
> > at
> >
> >
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
> > at
> >
> >
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)[:1.8.0_232]
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
> > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
> > at
> >
> >
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
> > at
> > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
> > 2020-08-07 02:44:46,669 | INFO | Refreshing
> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date
> [Fri
> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated
> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,
> 10.184.11.9:2181,
> > 10.184.11.7:2181//activemq/ap
> > ps/mkt/lpf/prod-usc1a/BROKER01P] |
> org.apache.activemq.broker.BrokerService
> > | main
> > 2020-08-07 02:44:51,492 | INFO | Client
> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09
> GMT
> > | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,494 | INFO | Client
> > environment:host.name
> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
> > | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,496 | INFO | Client
> environment:java.version=1.8.0_232
> > | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle
> > Corporation | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,500 | INFO | Client
> >
> >
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
> > | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,504 | INFO | Client
> >
> >
> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
> > |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,505 | INFO | Client
> >
> >
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > | org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,506 | INFO | Client
> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,509 | INFO | Client
> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq
> |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,510 | INFO | Client
> > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper |
> > main
> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
> > sessionTimeout=2000 watcher=org.apache.activem
> > q.leveldb.replicated.groups.ZKClient@36060e |
> > org.apache.zookeeper.ZooKeeper | main
> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state
> > change dispatcher thread
> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
> > javax.security.auth.login.LoginException: No JAAS configuration section
> > named 'Client' was found in specified JAAS
> > configuration file:
> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue
> > connection to Zookeeper server without SASL authentication, if Zookeeper
> > server a
> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
> > 10.184.11.9:2181)
> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
> > main-SendThread(10.184.11.9:2181)
> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed |
> > org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to
> > 10.184.11.9/10.184.11.9:2181, initiating session |
> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
> > 11.9:2181)
> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on
> server
> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated
> > timeout = 4000 | org.apache.
> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
> > 2020-08-07 02:44:51,946 | INFO | Promoted to master |
> > org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to
> process
> > the events |
> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> > main-EventThread
> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to
> process
> > the events |
> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> > main-EventThread
> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
> > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
> > BrokerService[BROKER01P] Task-1
> > 2020-08-07 02:44:54,349 | INFO | Master started:
> >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
> > BrokerService[BROKER01P] Task-2
> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > is starting | org.apache.activemq.broker.BrokerService | main
> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
> >
> >
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> > org.apache.activemq.broker.TransportConnector | main
> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
> >
> >
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
> > | org.apache.activemq.transport.TransportServerThreadSupport | main
> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
> > org.apache.activemq.broker.TransportConnector | main
> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
> >
> >
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> > started | org.apache.activemq.broker.BrokerService | main
> > 2020-08-07 02:44:54,827 | INFO | For help or more information please
> see:
> > http://activemq.apache.org | org.apache.activemq.broker.BrokerService |
> > main
> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store
> > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb
> only
> > has 42345 mb of usable space. - resetting to maximum available disk
> space:
> > 42345 mb | org.apache.activemq.broker.BrokerService | main
> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
> > (current store usage is 25900 mb). The data directory:
> > /var/lib/activemq/data only has 42345 mb of usable space. - resetting to
> > maximum available disk space: 42345 mb |
> > org.apache.activemq.broker.BrokerService | main
> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due
> to
> > exception, java.io.EOFException: File
> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 |
> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> > handler.
> > java.io.EOFException: File
> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> > at org.apache.activemq.console.Main.main(Main.java:115)
> >
>
Re: Activemq 5.15 startup error
Posted by Christopher Shannon <ch...@gmail.com>.
LevelDB support has been deprecated for many years now and is going to be
removed in the next version of ActiveMQ (5.17.0) so I would suggest trying
to migrate over to KahaDB.
On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran <ve...@gmail.com>
wrote:
> Hello
> We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. It's
> a very old system. However last night we started to get these errors and
> the broker did not come up. Any ideas on how to fix this would be
> appreciated. I can provide further log messages as needed, if sufficient
> direction is provided. Thank you!
>
> Here are the logs
> 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> is starting |
> org.apache.activemq.broker.BrokerService | main
> 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat
> .maxFrameSize=104857600 |
> org.apache.activemq.transport.TransportServerThreadSupport | main
> 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> org.apache.activemq.broker.TransportConnector | main
> 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
>
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat
> at
>
> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0]
> at
>
> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0]
> at
>
> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0]
> at
>
> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0]
> at
>
> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0]
> at
>
> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)[:1.8.0_232]
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232]
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232]
> at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232]
> at
>
> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0]
> at
> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0]
> 2020-08-07 02:44:46,669 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date [Fri
> Aug 07 02:44:46 CDT 2020]; root of context hierarchy |
> org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
> 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated
> LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181,10.184.11.9:2181,
> 10.184.11.7:2181//activemq/ap
> ps/mkt/lpf/prod-usc1a/BROKER01P] | org.apache.activemq.broker.BrokerService
> | main
> 2020-08-07 02:44:51,492 | INFO | Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,494 | INFO | Client
> environment:host.name
> =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal
> | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,496 | INFO | Client environment:java.version=1.8.0_232
> | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle
> Corporation | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,500 | INFO | Client
>
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
> | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,504 | INFO | Client
>
> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar
> |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,505 | INFO | Client
>
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> | org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,506 | INFO | Client
> environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,509 | INFO | Client
> environment:os.version=3.10.0-1062.4.1.el7.x86_64 |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,510 | INFO | Client
> environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper |
> main
> 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,514 | INFO | Initiating client connection,
> connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181
> sessionTimeout=2000 watcher=org.apache.activem
> q.leveldb.replicated.groups.ZKClient@36060e |
> org.apache.zookeeper.ZooKeeper | main
> 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher |
> org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state
> change dispatcher thread
> 2020-08-07 02:44:51,573 | WARN | SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS
> configuration file:
> '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server a
> llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread(
> 10.184.11.9:2181)
> 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server
> 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn |
> main-SendThread(10.184.11.9:2181)
> 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed |
> org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread
> 2020-08-07 02:44:51,606 | INFO | Socket connection established to
> 10.184.11.9/10.184.11.9:2181, initiating session |
> org.apache.zookeeper.ClientCnxn | main-SendThread(10.184.
> 11.9:2181)
> 2020-08-07 02:44:51,622 | INFO | Session establishment complete on server
> 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated
> timeout = 4000 | org.apache.
> zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181)
> 2020-08-07 02:44:51,946 | INFO | Promoted to master |
> org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread
> 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to process
> the events |
> org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> main-EventThread
> 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to process
> the events |
> org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport |
> main-EventThread
> 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB
> implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ
> BrokerService[BROKER01P] Task-1
> 2020-08-07 02:44:54,349 | INFO | Master started:
>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218
> | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ
> BrokerService[BROKER01P] Task-2
> 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> is starting | org.apache.activemq.broker.BrokerService | main
> 2020-08-07 02:44:54,732 | INFO | Listening for connections at:
>
> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2020-08-07 02:44:54,733 | INFO | Connector openwire started |
> org.apache.activemq.broker.TransportConnector | main
> 2020-08-07 02:44:54,823 | INFO | Listening for connections at:
>
> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true
> | org.apache.activemq.transport.TransportServerThreadSupport | main
> 2020-08-07 02:44:54,825 | INFO | Connector ssl started |
> org.apache.activemq.broker.TransportConnector | main
> 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P,
>
> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1)
> started | org.apache.activemq.broker.BrokerService | main
> 2020-08-07 02:44:54,827 | INFO | For help or more information please see:
> http://activemq.apache.org | org.apache.activemq.broker.BrokerService |
> main
> 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store
> usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb only
> has 42345 mb of usable space. - resetting to maximum available disk space:
> 42345 mb | org.apache.activemq.broker.BrokerService | main
> 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb
> (current store usage is 25900 mb). The data directory:
> /var/lib/activemq/data only has 42345 mb of usable space. - resetting to
> maximum available disk space: 42345 mb |
> org.apache.activemq.broker.BrokerService | main
> 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due to
> exception, java.io.EOFException: File
> '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 |
> org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException
> handler.
> java.io.EOFException: File
> '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
> at org.apache.activemq.console.Main.main(Main.java:115)
>