You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Clebert Suconic <cl...@gmail.com> on 2018/08/03 22:22:09 UTC
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
https://github.com/apache/activemq-artemis/pull/2214
I will make it into 2.6.x as soon as the PR tests are finished.
If you could try a snapshot of master or 2.6.x (after I merge it of
course). Let me know if you need help buildling it.
On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com> wrote:
> Michael for the win!
> @Clebert im not opposed to coding up a test case it is just that i am
> swamped at the moment so im glad to hear that Michael already has a bead on
> this
>
> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
> imp* on a clean 2.6.2. the new system didnt have Address Settings and
> Security Settings that had previously been added via the jolokia api. i
> should look into the export options more to see if i just failed to provide
> a command line argument. fortunately in my scenario it was pretty trivial
> to recreate those settings quickly. so i did that. the import created all
> the addresses, routes, queues, and messages that i needed. it went pretty
> well. so my cluster is now on 2.6.2
>
> i plan to rework our usage of AddressSetting to not create a new entry for
> every Address we create but instead only create them for the few exceptions
> to our default configuration
>
> then i can start working on running the broker in CF with the cluster in
> more of a Master-Master mode.
> thanks again for all your help. feel free to respond if there is any more
> data you need from me.
>
> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
> while handshaking has occurred. " but thats another issue i will figure
> out. *
>
> my 30 minute startup times are now down to 30 seconds max. more like 10-15
> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
>
> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
> <mi...@me.com.invalid> wrote:
>
>> Clebert im able to recreate this and also worked out in code why this
>> happens. Ill try catch you on irc tomorrow or friday.
>>
>> Sent from my Samsung Galaxy smartphone.
>> -------- Original message --------From: Clebert Suconic <
>> clebert.suconic@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To:
>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes
>> to boot & failover
>> Yes.. it should include it.
>>
>>
>> Do you have an example of the API you used with 2.1?
>>
>>
>> I can't write a compatibility test this week.. but if you provide me
>> the example I will do it early next week.
>>
>>
>> In a perfect world, if you did it .. it would be great :)
>>
>>
>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
>> wrote:
>> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
>> > tried a few months ago. unfortunately there was not. it may have failed
>> > much faster than 2.5.0 i dont recall the timing specifics but the error
>> is
>> > the same
>> >
>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
>> > Failure in initialisation: java.lang.NegativeArraySizeException
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at
>> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > [rt.jar:1.8.0_171]
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > [rt.jar:1.8.0_171]
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > [rt.jar:1.8.0_171]
>> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> > [artemis-boot.jar:2.6.2]
>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> > [artemis-boot.jar:2.6.2]
>> >
>> >
>> > right there in the AddressSettings.decode
>> >
>> >
>> > I am going to start looking into a journal export and then a journal
>> import
>> > and see what happens there.
>> > are programmatically created addresses / queues stored in the data export
>> > along with the address and security settings?
>> >
>> >
>> >
>> >
>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> > wrote:
>> >
>> >> @dan take a look at the compatibility tests in master.. We're help to
>> >> help if you don't understand anything.. it's using ClassLoaders and
>> >> Groovy so you can produce stuff with a combination of clients and
>> >> servers on 2.1.0 and consume on the current version.
>> >>
>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
>> >> <cl...@gmail.com> wrote:
>> >> > @Dan: what about this:
>> >> >
>> >> > you provide us a code path that could generate the incompatibility in
>> >> 2.1.0...
>> >> >
>> >> > an ultimate deal would be if you produced a compatibility test with
>> >> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
>> >> > and then consume the messages on master (or 2.6.x)...
>> >> >
>> >> >
>> >> > if you generate the compatibility test showing the issue, we can try
>> >> fixing it.
>> >> >
>> >> >
>> >> > What about this idea? that would help you migrate into 2.6.x or
>> master.
>> >> >
>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
>> >> > <cl...@gmail.com> wrote:
>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangford@gmail.com
>> >
>> >> wrote:
>> >> >>> Thank you that was very helpful. we actually do have an address
>> >> settings
>> >> >>> entry for each queue. there could be a better pattern for us. but
>> >> currently
>> >> >>> our automated system for creating queues creates an address setting
>> at
>> >> the
>> >> >>> same time. i will look into improved patterns.
>> >> >>>
>> >> >>> as far as upgrading goes. i agree we really want to upgrade. until i
>> >> can
>> >> >>> find a work around for the config-delete-queues deserialization bug
>> >> >>> introduced in 2.2.0 i brought up back in April we will not be able
>> to
>> >> >>> easily move.
>> >> >>
>> >> >> You should be able to move to 2.6.2. if you're not able to I would
>> >> >> like to know where it failed.
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> thanks again for all the help
>> >> >>>
>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
>> >> clebert.suconic@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> If you do not want to upgrade for any reason export the journal.
>> >> Cleani
>> >> >>>> uo. Edit the text and remove the garbage (you will see) manually.
>> >> Delete
>> >> >>>> all data and te import.
>> >> >>>>
>> >> >>>> (Make a backup to be safe of course)
>> >> >>>>
>> >> >>>>
>> >> >>>> But I still recommend the upgrade.
>> >> >>>>
>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>> >> >>>> clebert.suconic@gmail.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> > The address setting is the garbage I was talking about. Upgrade
>> to
>> >> the
>> >> >>>> > latest broker and there will be a cleanup done at the load
>> before it
>> >> >>>> > starts.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > I highly recommend upgrade.
>> >> >>>> >
>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
>> >> jbertram@apache.org>
>> >> >>>> > wrote:
>> >> >>>> >
>> >> >>>> >> Analyzing thread dumps like this is pretty simple. I generally
>> >> just
>> >> >>>> >> scroll
>> >> >>>> >> through and look for long stack-traces with lots of calls from
>> >> >>>> >> org.apache.activemq.artemis. In your case every single thread
>> >> dump has
>> >> >>>> a
>> >> >>>> >> thread doing something like this:
>> >> >>>> >>
>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
>> >> runnable
>> >> >>>> >> [0x00007f9031675000]
>> >> >>>> >> java.lang.Thread.State: RUNNABLE
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>> >> >>>> >> - locked <0x0000000080a8fce8> (a
>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>> >> >>>> >> - locked <0x0000000080a8fce8> (a
>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> >> >>>> >> - locked <0x000000008098be80> (a
>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
>> >> >>>> >> at
>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>> >> >>>> >> at
>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>> >> >>>> >> at
>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>> >> >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >> >>>> >> at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> >>>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>> >> >>>> >> at
>> >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> >> >>>> >> at
>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> >> >>>> >>
>> >> >>>> >> In every case it's the "main" thread which isn't surprising as
>> >> that is
>> >> >>>> the
>> >> >>>> >> thread responsible for starting the broker. Also, you can
>> pretty
>> >> >>>> clearly
>> >> >>>> >> see in the trace that this is the thread starting the broker,
>> it's
>> >> >>>> loading
>> >> >>>> >> the journals, & restoring stored configuration (either address
>> >> settings
>> >> >>>> or
>> >> >>>> >> security settings). I've seen high broker start times when
>> there
>> >> are
>> >> >>>> lots
>> >> >>>> >> and lots of addresses and lots of and lots of settings. Do
>> either
>> >> (or
>> >> >>>> >> both) of these situations apply to you?
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >> Justin
>> >> >>>> >>
>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
>> >> danlangford@gmail.com>
>> >> >>>> >> wrote:
>> >> >>>> >>
>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>> >> >>>> >> config-delete-queues
>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
>> >> squashed
>> >> >>>> in
>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
>> >> load) but
>> >> >>>> we
>> >> >>>> >> > discussed it back in April. I should go confirm that bug on
>> 2.6
>> >> and
>> >> >>>> >> make a
>> >> >>>> >> > jira for that
>> >> >>>> >> >
>> >> >>>> >> > Thanks
>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>> >> >>>> >> clebert.suconic@gmail.com
>> >> >>>> >> > >
>> >> >>>> >> > wrote:
>> >> >>>> >> >
>> >> >>>> >> > > There is an issue I remember where the journal would have
>> some
>> >> dirt
>> >> >>>> >> that
>> >> >>>> >> > > was fixed on 2.3/0.
>> >> >>>> >> > >
>> >> >>>> >> > > I would ipgrade to 2.6.2.
>> >> >>>> >> > >
>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
>> >> danlangford@gmail.com
>> >> >>>> >
>> >> >>>> >> > > wrote:
>> >> >>>> >> > >
>> >> >>>> >> > > > would you be willing to help me translate these thread
>> dumps?
>> >> >>>> >> > > >
>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i
>> will
>> >> also
>> >> >>>> >> share
>> >> >>>> >> > > > the fasthread.io links for each file. (i was struggling
>> >> getting
>> >> >>>> >> > > > fastthread to do a combo report with the threads in the
>> >> correct
>> >> >>>> >> order)
>> >> >>>> >> > > >
>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>> >> >>>> >> > > >
>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
>> >> than at
>> >> >>>> the
>> >> >>>> >> > > other
>> >> >>>> >> > > > sample times. but i dont know what to look for in here to
>> >> >>>> determine
>> >> >>>> >> > what
>> >> >>>> >> > > > the broker is actually during during this time.
>> >> >>>> >> > > >
>> >> >>>> >> > > > thanks
>> >> >>>> >> > > >
>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>> >> >>>> jbertram@apache.org
>> >> >>>> >> >
>> >> >>>> >> > > > wrote:
>> >> >>>> >> > > >
>> >> >>>> >> > > >> The first place I would start is grabbing thread dumps
>> >> every so
>> >> >>>> >> often
>> >> >>>> >> > to
>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
>> >> minutes.
>> >> >>>> >> > > >>
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> Justin
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>> >> >>>> >> danlangford@gmail.com>
>> >> >>>> >> > > >> wrote:
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> > so my server startup times and failover times are
>> growing
>> >> >>>> pretty
>> >> >>>> >> > big.
>> >> >>>> >> > > >> but i
>> >> >>>> >> > > >> > dont really know where to start looking.
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
>> >> stamps:
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis.
>> >> >>>> >> > integration.bootstrap]
>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:31,818 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221000:
>> >> >>>> >> > > >> > live Message Broker is starting with configuration
>> Broker
>> >> >>>> >> > > Configuration
>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
>> >> >>>> >> > journal,bindingsDirectory=./
>> >> >>>> >> > > >> >
>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,462 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221012:
>> >> >>>> >> > > >> > Using AIO Journal
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,493 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221057:
>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
>> >> size
>> >> >>>> >> (-Xmx).
>> >> >>>> >> > > >> being
>> >> >>>> >> > > >> > defined as 1,073,741,824
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,555 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
>> protocol
>> >> >>>> support
>> >> >>>> >> > for:
>> >> >>>> >> > > >> CORE
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,555 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
>> >> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: AMQP
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
>> Adding
>> >> >>>> >> protocol
>> >> >>>> >> > > >> support
>> >> >>>> >> > > >> > for: HORNETQ
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
>> >> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: MQTT
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
>> Adding
>> >> >>>> >> protocol
>> >> >>>> >> > > >> support
>> >> >>>> >> > > >> > for: OPENWIRE
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>> >> >>>> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: STOMP
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,963 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,963 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
>> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,983 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>> >> >>>> >> > > AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,983 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,984 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,985 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue example in
>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,985 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue
>> in
>> >> >>>> >> > > AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,986 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
>> >> to be
>> >> >>>> >> > looking
>> >> >>>> >> > > at
>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
>> >> looking
>> >> >>>> >> at
>> >> >>>> >> > the
>> >> >>>> >> > > >> > number of messages persisted on these queues? where
>> would
>> >> you
>> >> >>>> >> look
>> >> >>>> >> > to
>> >> >>>> >> > > >> > determine why the startup times are so long?
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
>> off of
>> >> >>>> 2.1.0
>> >> >>>> >> due
>> >> >>>> >> > > to
>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
>> think
>> >> i have
>> >> >>>> >> > > another
>> >> >>>> >> > > >> > thread on here (that i need to update) regarding that
>> >> upgrade
>> >> >>>> >> issue.
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >>
>> >> >>>> >> > > > --
>> >> >>>> >> > > Clebert Suconic
>> >> >>>> >> > >
>> >> >>>> >> >
>> >> >>>> >>
>> >> >>>> > --
>> >> >>>> > Clebert Suconic
>> >> >>>> >
>> >> >>>> --
>> >> >>>> Clebert Suconic
>> >> >>>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Clebert Suconic
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Clebert Suconic
>> >>
>> >>
>> >>
>> >> --
>> >> Clebert Suconic
>> >>
>>
>>
>>
>> --
>> Clebert Suconic
>>
--
Clebert Suconic
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Posted by Dan Langford <da...@gmail.com>.
thank you very much for your work on this. i really appreciate it.
On Thu, Aug 23, 2018 at 12:26 PM Clebert Suconic <cl...@gmail.com>
wrote:
> > what is the timeline on a 2.6.3 or 2.7.0 release?
> >
>
>
> I was supposed to release it next week based on my HEADS up. I'm
> working on some extensive QA and dev at the moment.. but I should
> make it by next week.
>
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Posted by Clebert Suconic <cl...@gmail.com>.
> what is the timeline on a 2.6.3 or 2.7.0 release?
>
I was supposed to release it next week based on my HEADS up. I'm
working on some extensive QA and dev at the moment.. but I should
make it by next week.
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Posted by Dan Langford <da...@gmail.com>.
i validated that the fix did in fact fix our upgrade path. i made a new
2.1.0 broker. i added some address settings via JMX. i attempted an upgrade
to 2.6.2 and watched it fail. i pulled the code that you linked above. i
packaged a release and then i upgraded to the released code and the server
started just fine.
thank you so much!!
what is the timeline on a 2.6.3 or 2.7.0 release?
On Fri, Aug 3, 2018 at 4:22 PM Clebert Suconic <cl...@gmail.com>
wrote:
> I'm considering doing a 2.6.3 very soon.. it will include the fix..
> you should be able to move from 2.1.0 . let me know if you have any
> other issues.
>
> On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > https://github.com/apache/activemq-artemis/pull/2214
> >
> >
> > I will make it into 2.6.x as soon as the PR tests are finished.
> >
> >
> > If you could try a snapshot of master or 2.6.x (after I merge it of
> > course). Let me know if you need help buildling it.
> >
> > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com>
> wrote:
> >> Michael for the win!
> >> @Clebert im not opposed to coding up a test case it is just that i am
> >> swamped at the moment so im glad to hear that Michael already has a
> bead on
> >> this
> >>
> >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
> >> imp* on a clean 2.6.2. the new system didnt have Address Settings and
> >> Security Settings that had previously been added via the jolokia api. i
> >> should look into the export options more to see if i just failed to
> provide
> >> a command line argument. fortunately in my scenario it was pretty
> trivial
> >> to recreate those settings quickly. so i did that. the import created
> all
> >> the addresses, routes, queues, and messages that i needed. it went
> pretty
> >> well. so my cluster is now on 2.6.2
> >>
> >> i plan to rework our usage of AddressSetting to not create a new entry
> for
> >> every Address we create but instead only create them for the few
> exceptions
> >> to our default configuration
> >>
> >> then i can start working on running the broker in CF with the cluster in
> >> more of a Master-Master mode.
> >> thanks again for all your help. feel free to respond if there is any
> more
> >> data you need from me.
> >>
> >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
> >> while handshaking has occurred. " but thats another issue i will figure
> >> out. *
> >>
> >> my 30 minute startup times are now down to 30 seconds max. more like
> 10-15
> >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
> >>
> >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
> >> <mi...@me.com.invalid> wrote:
> >>
> >>> Clebert im able to recreate this and also worked out in code why this
> >>> happens. Ill try catch you on irc tomorrow or friday.
> >>>
> >>> Sent from my Samsung Galaxy smartphone.
> >>> -------- Original message --------From: Clebert Suconic <
> >>> clebert.suconic@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To:
> >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+
> minutes
> >>> to boot & failover
> >>> Yes.. it should include it.
> >>>
> >>>
> >>> Do you have an example of the API you used with 2.1?
> >>>
> >>>
> >>> I can't write a compatibility test this week.. but if you provide me
> >>> the example I will do it early next week.
> >>>
> >>>
> >>> In a perfect world, if you did it .. it would be great :)
> >>>
> >>>
> >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
> >>> wrote:
> >>> > i tried 2.6.2 this morning to see if that was an improvement from
> 2.5.0 i
> >>> > tried a few months ago. unfortunately there was not. it may have
> failed
> >>> > much faster than 2.5.0 i dont recall the timing specifics but the
> error
> >>> is
> >>> > the same
> >>> >
> >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server]
> AMQ224000:
> >>> > Failure in initialisation: java.lang.NegativeArraySizeException
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > [rt.jar:1.8.0_171]
> >>> > at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_171]
> >>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> > [artemis-boot.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> > [artemis-boot.jar:2.6.2]
> >>> >
> >>> >
> >>> > right there in the AddressSettings.decode
> >>> >
> >>> >
> >>> > I am going to start looking into a journal export and then a journal
> >>> import
> >>> > and see what happens there.
> >>> > are programmatically created addresses / queues stored in the data
> export
> >>> > along with the address and security settings?
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
> >>> clebert.suconic@gmail.com>
> >>> > wrote:
> >>> >
> >>> >> @dan take a look at the compatibility tests in master.. We're help
> to
> >>> >> help if you don't understand anything.. it's using ClassLoaders and
> >>> >> Groovy so you can produce stuff with a combination of clients and
> >>> >> servers on 2.1.0 and consume on the current version.
> >>> >>
> >>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
> >>> >> <cl...@gmail.com> wrote:
> >>> >> > @Dan: what about this:
> >>> >> >
> >>> >> > you provide us a code path that could generate the
> incompatibility in
> >>> >> 2.1.0...
> >>> >> >
> >>> >> > an ultimate deal would be if you produced a compatibility test
> with
> >>> >> > 2.1.0.... you add some code that will generate the journal with
> 2.1.0,
> >>> >> > and then consume the messages on master (or 2.6.x)...
> >>> >> >
> >>> >> >
> >>> >> > if you generate the compatibility test showing the issue, we can
> try
> >>> >> fixing it.
> >>> >> >
> >>> >> >
> >>> >> > What about this idea? that would help you migrate into 2.6.x or
> >>> master.
> >>> >> >
> >>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> >>> >> > <cl...@gmail.com> wrote:
> >>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <
> danlangford@gmail.com
> >>> >
> >>> >> wrote:
> >>> >> >>> Thank you that was very helpful. we actually do have an address
> >>> >> settings
> >>> >> >>> entry for each queue. there could be a better pattern for us.
> but
> >>> >> currently
> >>> >> >>> our automated system for creating queues creates an address
> setting
> >>> at
> >>> >> the
> >>> >> >>> same time. i will look into improved patterns.
> >>> >> >>>
> >>> >> >>> as far as upgrading goes. i agree we really want to upgrade.
> until i
> >>> >> can
> >>> >> >>> find a work around for the config-delete-queues deserialization
> bug
> >>> >> >>> introduced in 2.2.0 i brought up back in April we will not be
> able
> >>> to
> >>> >> >>> easily move.
> >>> >> >>
> >>> >> >> You should be able to move to 2.6.2. if you're not able to I
> would
> >>> >> >> like to know where it failed.
> >>> >> >>
> >>> >> >>
> >>> >> >>>
> >>> >> >>> thanks again for all the help
> >>> >> >>>
> >>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
> >>> >> clebert.suconic@gmail.com>
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>>> If you do not want to upgrade for any reason export the
> journal.
> >>> >> Cleani
> >>> >> >>>> uo. Edit the text and remove the garbage (you will see)
> manually.
> >>> >> Delete
> >>> >> >>>> all data and te import.
> >>> >> >>>>
> >>> >> >>>> (Make a backup to be safe of course)
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>> But I still recommend the upgrade.
> >>> >> >>>>
> >>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
> >>> >> >>>> clebert.suconic@gmail.com>
> >>> >> >>>> wrote:
> >>> >> >>>>
> >>> >> >>>> > The address setting is the garbage I was talking about.
> Upgrade
> >>> to
> >>> >> the
> >>> >> >>>> > latest broker and there will be a cleanup done at the load
> >>> before it
> >>> >> >>>> > starts.
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> > I highly recommend upgrade.
> >>> >> >>>> >
> >>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
> >>> >> jbertram@apache.org>
> >>> >> >>>> > wrote:
> >>> >> >>>> >
> >>> >> >>>> >> Analyzing thread dumps like this is pretty simple. I
> generally
> >>> >> just
> >>> >> >>>> >> scroll
> >>> >> >>>> >> through and look for long stack-traces with lots of calls
> from
> >>> >> >>>> >> org.apache.activemq.artemis. In your case every single
> thread
> >>> >> dump has
> >>> >> >>>> a
> >>> >> >>>> >> thread doing something like this:
> >>> >> >>>> >>
> >>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
> >>> >> runnable
> >>> >> >>>> >> [0x00007f9031675000]
> >>> >> >>>> >> java.lang.Thread.State: RUNNABLE
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >>> >> >>>> >> - locked <0x0000000080a8fce8> (a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >>> >> >>>> >> - locked <0x0000000080a8fce8> (a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> >> >>>> >> - locked <0x000000008098be80> (a
> >>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
> >>> >> >>>> >> at
> >>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >>> >> >>>> >> at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >>> >> >>>> >> at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >>> >> >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>> Method)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> >> >>>> >> at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> >> >>>> >> at java.lang.reflect.Method.invoke(Method.java:498)
> >>> >> >>>> >> at
> >>> >> >>>>
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> >> >>>> >> at
> >>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> >> >>>> >>
> >>> >> >>>> >> In every case it's the "main" thread which isn't surprising
> as
> >>> >> that is
> >>> >> >>>> the
> >>> >> >>>> >> thread responsible for starting the broker. Also, you can
> >>> pretty
> >>> >> >>>> clearly
> >>> >> >>>> >> see in the trace that this is the thread starting the
> broker,
> >>> it's
> >>> >> >>>> loading
> >>> >> >>>> >> the journals, & restoring stored configuration (either
> address
> >>> >> settings
> >>> >> >>>> or
> >>> >> >>>> >> security settings). I've seen high broker start times when
> >>> there
> >>> >> are
> >>> >> >>>> lots
> >>> >> >>>> >> and lots of addresses and lots of and lots of settings. Do
> >>> either
> >>> >> (or
> >>> >> >>>> >> both) of these situations apply to you?
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>> >> Justin
> >>> >> >>>> >>
> >>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
> >>> >> danlangford@gmail.com>
> >>> >> >>>> >> wrote:
> >>> >> >>>> >>
> >>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
> >>> >> >>>> >> config-delete-queues
> >>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug
> was
> >>> >> squashed
> >>> >> >>>> in
> >>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and
> work
> >>> >> load) but
> >>> >> >>>> we
> >>> >> >>>> >> > discussed it back in April. I should go confirm that bug
> on
> >>> 2.6
> >>> >> and
> >>> >> >>>> >> make a
> >>> >> >>>> >> > jira for that
> >>> >> >>>> >> >
> >>> >> >>>> >> > Thanks
> >>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> >>> >> >>>> >> clebert.suconic@gmail.com
> >>> >> >>>> >> > >
> >>> >> >>>> >> > wrote:
> >>> >> >>>> >> >
> >>> >> >>>> >> > > There is an issue I remember where the journal would
> have
> >>> some
> >>> >> dirt
> >>> >> >>>> >> that
> >>> >> >>>> >> > > was fixed on 2.3/0.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > I would ipgrade to 2.6.2.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
> >>> >> danlangford@gmail.com
> >>> >> >>>> >
> >>> >> >>>> >> > > wrote:
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > > would you be willing to help me translate these thread
> >>> dumps?
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them.
> i
> >>> will
> >>> >> also
> >>> >> >>>> >> share
> >>> >> >>>> >> > > > the fasthread.io links for each file. (i was
> struggling
> >>> >> getting
> >>> >> >>>> >> > > > fastthread to do a combo report with the threads in
> the
> >>> >> correct
> >>> >> >>>> >> order)
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> >>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> >>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> >>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> >>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> >>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> >>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets
> bigger
> >>> >> than at
> >>> >> >>>> the
> >>> >> >>>> >> > > other
> >>> >> >>>> >> > > > sample times. but i dont know what to look for in
> here to
> >>> >> >>>> determine
> >>> >> >>>> >> > what
> >>> >> >>>> >> > > > the broker is actually during during this time.
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > thanks
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
> >>> >> >>>> jbertram@apache.org
> >>> >> >>>> >> >
> >>> >> >>>> >> > > > wrote:
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > >> The first place I would start is grabbing thread
> dumps
> >>> >> every so
> >>> >> >>>> >> often
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
> >>> >> minutes.
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> Justin
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> >>> >> >>>> >> danlangford@gmail.com>
> >>> >> >>>> >> > > >> wrote:
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> > so my server startup times and failover times are
> >>> growing
> >>> >> >>>> pretty
> >>> >> >>>> >> > big.
> >>> >> >>>> >> > > >> but i
> >>> >> >>>> >> > > >> > dont really know where to start looking.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
> >>> >> stamps:
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis.
> >>> >> >>>> >> > integration.bootstrap]
> >>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,818 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221000:
> >>> >> >>>> >> > > >> > live Message Broker is starting with configuration
> >>> Broker
> >>> >> >>>> >> > > Configuration
> >>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
> >>> >> >>>> >> > journal,bindingsDirectory=./
> >>> >> >>>> >> > > >> >
> >>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
> >>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,462 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221012:
> >>> >> >>>> >> > > >> > Using AIO Journal
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,493 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221057:
> >>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the
> JVM max
> >>> >> size
> >>> >> >>>> >> (-Xmx).
> >>> >> >>>> >> > > >> being
> >>> >> >>>> >> > > >> > defined as 1,073,741,824
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
> >>> protocol
> >>> >> >>>> support
> >>> >> >>>> >> > for:
> >>> >> >>>> >> > > >> CORE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: AMQP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: HORNETQ
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: MQTT
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: OPENWIRE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol].
> Adding
> >>> >> >>>> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: STOMP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
> >>> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue
> ExpiryQueue in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,984 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue example in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue
> exampleQueue
> >>> in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,986 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i
> need
> >>> >> to be
> >>> >> >>>> >> > looking
> >>> >> >>>> >> > > at
> >>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need
> to be
> >>> >> looking
> >>> >> >>>> >> at
> >>> >> >>>> >> > the
> >>> >> >>>> >> > > >> > number of messages persisted on these queues? where
> >>> would
> >>> >> you
> >>> >> >>>> >> look
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> > determine why the startup times are so long?
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
> >>> off of
> >>> >> >>>> 2.1.0
> >>> >> >>>> >> due
> >>> >> >>>> >> > > to
> >>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
> >>> think
> >>> >> i have
> >>> >> >>>> >> > > another
> >>> >> >>>> >> > > >> > thread on here (that i need to update) regarding
> that
> >>> >> upgrade
> >>> >> >>>> >> issue.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > > --
> >>> >> >>>> >> > > Clebert Suconic
> >>> >> >>>> >> > >
> >>> >> >>>> >> >
> >>> >> >>>> >>
> >>> >> >>>> > --
> >>> >> >>>> > Clebert Suconic
> >>> >> >>>> >
> >>> >> >>>> --
> >>> >> >>>> Clebert Suconic
> >>> >> >>>>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> Clebert Suconic
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Clebert Suconic
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Clebert Suconic
> >>> >>
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>>
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Posted by Clebert Suconic <cl...@gmail.com>.
I'm considering doing a 2.6.3 very soon.. it will include the fix..
you should be able to move from 2.1.0 . let me know if you have any
other issues.
On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic
<cl...@gmail.com> wrote:
> https://github.com/apache/activemq-artemis/pull/2214
>
>
> I will make it into 2.6.x as soon as the PR tests are finished.
>
>
> If you could try a snapshot of master or 2.6.x (after I merge it of
> course). Let me know if you need help buildling it.
>
> On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com> wrote:
>> Michael for the win!
>> @Clebert im not opposed to coding up a test case it is just that i am
>> swamped at the moment so im glad to hear that Michael already has a bead on
>> this
>>
>> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
>> imp* on a clean 2.6.2. the new system didnt have Address Settings and
>> Security Settings that had previously been added via the jolokia api. i
>> should look into the export options more to see if i just failed to provide
>> a command line argument. fortunately in my scenario it was pretty trivial
>> to recreate those settings quickly. so i did that. the import created all
>> the addresses, routes, queues, and messages that i needed. it went pretty
>> well. so my cluster is now on 2.6.2
>>
>> i plan to rework our usage of AddressSetting to not create a new entry for
>> every Address we create but instead only create them for the few exceptions
>> to our default configuration
>>
>> then i can start working on running the broker in CF with the cluster in
>> more of a Master-Master mode.
>> thanks again for all your help. feel free to respond if there is any more
>> data you need from me.
>>
>> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
>> while handshaking has occurred. " but thats another issue i will figure
>> out. *
>>
>> my 30 minute startup times are now down to 30 seconds max. more like 10-15
>> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
>>
>> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
>> <mi...@me.com.invalid> wrote:
>>
>>> Clebert im able to recreate this and also worked out in code why this
>>> happens. Ill try catch you on irc tomorrow or friday.
>>>
>>> Sent from my Samsung Galaxy smartphone.
>>> -------- Original message --------From: Clebert Suconic <
>>> clebert.suconic@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To:
>>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes
>>> to boot & failover
>>> Yes.. it should include it.
>>>
>>>
>>> Do you have an example of the API you used with 2.1?
>>>
>>>
>>> I can't write a compatibility test this week.. but if you provide me
>>> the example I will do it early next week.
>>>
>>>
>>> In a perfect world, if you did it .. it would be great :)
>>>
>>>
>>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
>>> wrote:
>>> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
>>> > tried a few months ago. unfortunately there was not. it may have failed
>>> > much faster than 2.5.0 i dont recall the timing specifics but the error
>>> is
>>> > the same
>>> >
>>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
>>> > Failure in initialisation: java.lang.NegativeArraySizeException
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at
>>> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > [rt.jar:1.8.0_171]
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> > [rt.jar:1.8.0_171]
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > [rt.jar:1.8.0_171]
>>> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
>>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>> > [artemis-boot.jar:2.6.2]
>>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>> > [artemis-boot.jar:2.6.2]
>>> >
>>> >
>>> > right there in the AddressSettings.decode
>>> >
>>> >
>>> > I am going to start looking into a journal export and then a journal
>>> import
>>> > and see what happens there.
>>> > are programmatically created addresses / queues stored in the data export
>>> > along with the address and security settings?
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> > wrote:
>>> >
>>> >> @dan take a look at the compatibility tests in master.. We're help to
>>> >> help if you don't understand anything.. it's using ClassLoaders and
>>> >> Groovy so you can produce stuff with a combination of clients and
>>> >> servers on 2.1.0 and consume on the current version.
>>> >>
>>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
>>> >> <cl...@gmail.com> wrote:
>>> >> > @Dan: what about this:
>>> >> >
>>> >> > you provide us a code path that could generate the incompatibility in
>>> >> 2.1.0...
>>> >> >
>>> >> > an ultimate deal would be if you produced a compatibility test with
>>> >> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
>>> >> > and then consume the messages on master (or 2.6.x)...
>>> >> >
>>> >> >
>>> >> > if you generate the compatibility test showing the issue, we can try
>>> >> fixing it.
>>> >> >
>>> >> >
>>> >> > What about this idea? that would help you migrate into 2.6.x or
>>> master.
>>> >> >
>>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
>>> >> > <cl...@gmail.com> wrote:
>>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangford@gmail.com
>>> >
>>> >> wrote:
>>> >> >>> Thank you that was very helpful. we actually do have an address
>>> >> settings
>>> >> >>> entry for each queue. there could be a better pattern for us. but
>>> >> currently
>>> >> >>> our automated system for creating queues creates an address setting
>>> at
>>> >> the
>>> >> >>> same time. i will look into improved patterns.
>>> >> >>>
>>> >> >>> as far as upgrading goes. i agree we really want to upgrade. until i
>>> >> can
>>> >> >>> find a work around for the config-delete-queues deserialization bug
>>> >> >>> introduced in 2.2.0 i brought up back in April we will not be able
>>> to
>>> >> >>> easily move.
>>> >> >>
>>> >> >> You should be able to move to 2.6.2. if you're not able to I would
>>> >> >> like to know where it failed.
>>> >> >>
>>> >> >>
>>> >> >>>
>>> >> >>> thanks again for all the help
>>> >> >>>
>>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
>>> >> clebert.suconic@gmail.com>
>>> >> >>> wrote:
>>> >> >>>
>>> >> >>>> If you do not want to upgrade for any reason export the journal.
>>> >> Cleani
>>> >> >>>> uo. Edit the text and remove the garbage (you will see) manually.
>>> >> Delete
>>> >> >>>> all data and te import.
>>> >> >>>>
>>> >> >>>> (Make a backup to be safe of course)
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> But I still recommend the upgrade.
>>> >> >>>>
>>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>>> >> >>>> clebert.suconic@gmail.com>
>>> >> >>>> wrote:
>>> >> >>>>
>>> >> >>>> > The address setting is the garbage I was talking about. Upgrade
>>> to
>>> >> the
>>> >> >>>> > latest broker and there will be a cleanup done at the load
>>> before it
>>> >> >>>> > starts.
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > I highly recommend upgrade.
>>> >> >>>> >
>>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
>>> >> jbertram@apache.org>
>>> >> >>>> > wrote:
>>> >> >>>> >
>>> >> >>>> >> Analyzing thread dumps like this is pretty simple. I generally
>>> >> just
>>> >> >>>> >> scroll
>>> >> >>>> >> through and look for long stack-traces with lots of calls from
>>> >> >>>> >> org.apache.activemq.artemis. In your case every single thread
>>> >> dump has
>>> >> >>>> a
>>> >> >>>> >> thread doing something like this:
>>> >> >>>> >>
>>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
>>> >> runnable
>>> >> >>>> >> [0x00007f9031675000]
>>> >> >>>> >> java.lang.Thread.State: RUNNABLE
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>>> >> >>>> >> - locked <0x0000000080a8fce8> (a
>>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>>> >> >>>> >> - locked <0x0000000080a8fce8> (a
>>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>> >> >>>> >> - locked <0x000000008098be80> (a
>>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
>>> >> >>>> >> at
>>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>>> >> >>>> >> at
>>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>>> >> >>>> >> at
>>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>>> >> >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> >> >>>> >> at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> >>>> >> at java.lang.reflect.Method.invoke(Method.java:498)
>>> >> >>>> >> at
>>> >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>> >> >>>> >> at
>>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>> >> >>>> >>
>>> >> >>>> >> In every case it's the "main" thread which isn't surprising as
>>> >> that is
>>> >> >>>> the
>>> >> >>>> >> thread responsible for starting the broker. Also, you can
>>> pretty
>>> >> >>>> clearly
>>> >> >>>> >> see in the trace that this is the thread starting the broker,
>>> it's
>>> >> >>>> loading
>>> >> >>>> >> the journals, & restoring stored configuration (either address
>>> >> settings
>>> >> >>>> or
>>> >> >>>> >> security settings). I've seen high broker start times when
>>> there
>>> >> are
>>> >> >>>> lots
>>> >> >>>> >> and lots of addresses and lots of and lots of settings. Do
>>> either
>>> >> (or
>>> >> >>>> >> both) of these situations apply to you?
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> Justin
>>> >> >>>> >>
>>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
>>> >> danlangford@gmail.com>
>>> >> >>>> >> wrote:
>>> >> >>>> >>
>>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>>> >> >>>> >> config-delete-queues
>>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
>>> >> squashed
>>> >> >>>> in
>>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
>>> >> load) but
>>> >> >>>> we
>>> >> >>>> >> > discussed it back in April. I should go confirm that bug on
>>> 2.6
>>> >> and
>>> >> >>>> >> make a
>>> >> >>>> >> > jira for that
>>> >> >>>> >> >
>>> >> >>>> >> > Thanks
>>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>>> >> >>>> >> clebert.suconic@gmail.com
>>> >> >>>> >> > >
>>> >> >>>> >> > wrote:
>>> >> >>>> >> >
>>> >> >>>> >> > > There is an issue I remember where the journal would have
>>> some
>>> >> dirt
>>> >> >>>> >> that
>>> >> >>>> >> > > was fixed on 2.3/0.
>>> >> >>>> >> > >
>>> >> >>>> >> > > I would ipgrade to 2.6.2.
>>> >> >>>> >> > >
>>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
>>> >> danlangford@gmail.com
>>> >> >>>> >
>>> >> >>>> >> > > wrote:
>>> >> >>>> >> > >
>>> >> >>>> >> > > > would you be willing to help me translate these thread
>>> dumps?
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i
>>> will
>>> >> also
>>> >> >>>> >> share
>>> >> >>>> >> > > > the fasthread.io links for each file. (i was struggling
>>> >> getting
>>> >> >>>> >> > > > fastthread to do a combo report with the threads in the
>>> >> correct
>>> >> >>>> >> order)
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
>>> >> than at
>>> >> >>>> the
>>> >> >>>> >> > > other
>>> >> >>>> >> > > > sample times. but i dont know what to look for in here to
>>> >> >>>> determine
>>> >> >>>> >> > what
>>> >> >>>> >> > > > the broker is actually during during this time.
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > thanks
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>>> >> >>>> jbertram@apache.org
>>> >> >>>> >> >
>>> >> >>>> >> > > > wrote:
>>> >> >>>> >> > > >
>>> >> >>>> >> > > >> The first place I would start is grabbing thread dumps
>>> >> every so
>>> >> >>>> >> often
>>> >> >>>> >> > to
>>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
>>> >> minutes.
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> Justin
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>>> >> >>>> >> danlangford@gmail.com>
>>> >> >>>> >> > > >> wrote:
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> > so my server startup times and failover times are
>>> growing
>>> >> >>>> pretty
>>> >> >>>> >> > big.
>>> >> >>>> >> > > >> but i
>>> >> >>>> >> > > >> > dont really know where to start looking.
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
>>> >> stamps:
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis.
>>> >> >>>> >> > integration.bootstrap]
>>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:31,818 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221000:
>>> >> >>>> >> > > >> > live Message Broker is starting with configuration
>>> Broker
>>> >> >>>> >> > > Configuration
>>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
>>> >> >>>> >> > journal,bindingsDirectory=./
>>> >> >>>> >> > > >> >
>>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
>>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,462 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221012:
>>> >> >>>> >> > > >> > Using AIO Journal
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,493 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221057:
>>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
>>> >> size
>>> >> >>>> >> (-Xmx).
>>> >> >>>> >> > > >> being
>>> >> >>>> >> > > >> > defined as 1,073,741,824
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,555 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
>>> protocol
>>> >> >>>> support
>>> >> >>>> >> > for:
>>> >> >>>> >> > > >> CORE
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,555 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
>>> >> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: AMQP
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
>>> Adding
>>> >> >>>> >> protocol
>>> >> >>>> >> > > >> support
>>> >> >>>> >> > > >> > for: HORNETQ
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
>>> >> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: MQTT
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
>>> Adding
>>> >> >>>> >> protocol
>>> >> >>>> >> > > >> support
>>> >> >>>> >> > > >> > for: OPENWIRE
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>>> >> >>>> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: STOMP
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,963 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,963 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
>>> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,983 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>>> >> >>>> >> > > AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,983 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,984 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,985 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue example in
>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,985 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue
>>> in
>>> >> >>>> >> > > AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,986 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
>>> >> to be
>>> >> >>>> >> > looking
>>> >> >>>> >> > > at
>>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
>>> >> looking
>>> >> >>>> >> at
>>> >> >>>> >> > the
>>> >> >>>> >> > > >> > number of messages persisted on these queues? where
>>> would
>>> >> you
>>> >> >>>> >> look
>>> >> >>>> >> > to
>>> >> >>>> >> > > >> > determine why the startup times are so long?
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
>>> off of
>>> >> >>>> 2.1.0
>>> >> >>>> >> due
>>> >> >>>> >> > > to
>>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
>>> think
>>> >> i have
>>> >> >>>> >> > > another
>>> >> >>>> >> > > >> > thread on here (that i need to update) regarding that
>>> >> upgrade
>>> >> >>>> >> issue.
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > > --
>>> >> >>>> >> > > Clebert Suconic
>>> >> >>>> >> > >
>>> >> >>>> >> >
>>> >> >>>> >>
>>> >> >>>> > --
>>> >> >>>> > Clebert Suconic
>>> >> >>>> >
>>> >> >>>> --
>>> >> >>>> Clebert Suconic
>>> >> >>>>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Clebert Suconic
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Clebert Suconic
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Clebert Suconic
>>> >>
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic