You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by Dan Langford <da...@gmail.com> on 2018/07/06 16:34:30 UTC

[artemis 2.1.0] taking 30+ minutes to boot & failover

so my server startup times and failover times are growing pretty big. but i
dont really know where to start looking.

here is a snippet of some logs to show you the time stamps:

08:11:31,801 INFO  [org.apache.activemq.artemis.integration.bootstrap]
AMQ101000: Starting ActiveMQ Artemis Server

08:11:31,818 INFO  [org.apache.activemq.artemis.core.server] AMQ221000:
live Message Broker is starting with configuration Broker Configuration
(clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging)

08:11:34,462 INFO  [org.apache.activemq.artemis.core.server] AMQ221012:
Using AIO Journal

08:11:34,493 INFO  [org.apache.activemq.artemis.core.server] AMQ221057:
Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being
defined as 1,073,741,824

08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-server]. Adding protocol support for: CORE

08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-amqp-protocol]. Adding protocol support
for: AMQP

08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-hornetq-protocol]. Adding protocol support
for: HORNETQ

08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-mqtt-protocol]. Adding protocol support
for: MQTT

08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-openwire-protocol]. Adding protocol support
for: OPENWIRE

08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
Protocol module found: [artemis-stomp-protocol]. Adding protocol support
for: STOMP

08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
Dead Letter Address configured for queue DLQ in AddressSettings

08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
Expiry Address configured for queue DLQ in AddressSettings

08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
Dead Letter Address configured for queue ExpiryQueue in AddressSettings

08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
Expiry Address configured for queue ExpiryQueue in AddressSettings

08:41:38,984 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
Dead Letter Address configured for queue example in AddressSettings

08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
Expiry Address configured for queue example in AddressSettings

08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
Dead Letter Address configured for queue exampleQueue in AddressSettings

08:41:38,986 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
Expiry Address configured for queue exampleQueue in AddressSettings


and it continues. i have 138 queues. i wonder if i need to be looking at
PAGE configuration or some cache sizes. do i need to be looking at the
number of messages persisted on these queues? where would you look to
determine why the startup times are so long?

also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to
some deseralization changes introduced in 2.2.0. i think i have another
thread on here (that i need to update) regarding that upgrade issue.

thanks for any ideas or insight you have for me

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

thank you very much for your work on this. i really appreciate it.

On Thu, Aug 23, 2018 at 12:26 PM Clebert Suconic <cl...@gmail.com>
wrote:

> > what is the timeline on a 2.6.3 or 2.7.0 release?
> >
>
>
> I was supposed to release it next week based on my HEADS up. I'm
> working on some extensive QA and dev at the moment..  but I should
> make it by next week.
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

> what is the timeline on a 2.6.3 or 2.7.0 release?
>


I was supposed to release it next week based on my HEADS up. I'm
working on some extensive QA and dev at the moment..  but I should
make it by next week.

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

i validated that the fix did in fact fix our upgrade path. i made a new
2.1.0 broker. i added some address settings via JMX. i attempted an upgrade
to 2.6.2 and watched it fail. i pulled the code that you linked above. i
packaged a release and then i upgraded to the released code and the server
started just fine.

thank you so much!!

what is the timeline on a 2.6.3 or 2.7.0 release?

On Fri, Aug 3, 2018 at 4:22 PM Clebert Suconic <cl...@gmail.com>
wrote:

> I'm considering doing a 2.6.3 very soon.. it will include the fix..
> you should be able to move from 2.1.0 . let me know if you have any
> other issues.
>
> On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > https://github.com/apache/activemq-artemis/pull/2214
> >
> >
> > I will make it into 2.6.x as soon as the PR tests are finished.
> >
> >
> > If you could try a snapshot of master or 2.6.x (after I merge it of
> > course). Let me know if you need help buildling it.
> >
> > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com>
> wrote:
> >> Michael for the win!
> >> @Clebert im not opposed to coding up a test case it is just that i am
> >> swamped at the moment so im glad to hear that Michael already has a
> bead on
> >> this
> >>
> >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
> >> imp* on a clean 2.6.2. the new system didnt have Address Settings and
> >> Security Settings that had previously been added via the jolokia api. i
> >> should look into the export options more to see if i just failed to
> provide
> >> a command line argument. fortunately in my scenario it was pretty
> trivial
> >> to recreate those settings quickly. so i did that. the import created
> all
> >> the addresses, routes, queues, and messages that i needed. it went
> pretty
> >> well. so my cluster is now on 2.6.2
> >>
> >> i plan to rework our usage of AddressSetting to not create a new entry
> for
> >> every Address we create but instead only create them for the few
> exceptions
> >> to our default configuration
> >>
> >> then i can start working on running the broker in CF with the cluster in
> >> more of a Master-Master mode.
> >> thanks again for all your help. feel free to respond if there is any
> more
> >> data you need from me.
> >>
> >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
> >> while handshaking has occurred. " but thats another issue i will figure
> >> out. *
> >>
> >> my 30 minute startup times are now down to 30 seconds max. more like
> 10-15
> >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
> >>
> >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
> >> <mi...@me.com.invalid> wrote:
> >>
> >>> Clebert im able to recreate this and also worked out in code why this
> >>> happens. Ill try catch you on irc tomorrow or friday.
> >>>
> >>> Sent from my Samsung Galaxy smartphone.
> >>> -------- Original message --------From: Clebert Suconic <
> >>> clebert.suconic@gmail.com> Date: 25/07/2018  20:20  (GMT+00:00) To:
> >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+
> minutes
> >>> to boot & failover
> >>> Yes.. it should include it.
> >>>
> >>>
> >>> Do you have an example of the API you used with 2.1?
> >>>
> >>>
> >>> I can't write a compatibility test this week.. but if you provide me
> >>> the example I will do it early next week.
> >>>
> >>>
> >>> In a perfect world, if you did it .. it would be great :)
> >>>
> >>>
> >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
> >>> wrote:
> >>> > i tried 2.6.2 this morning to see if that was an improvement from
> 2.5.0 i
> >>> > tried a few months ago. unfortunately there was not. it may have
> failed
> >>> > much faster than 2.5.0 i dont recall the timing specifics but the
> error
> >>> is
> >>> > the same
> >>> >
> >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server]
> AMQ224000:
> >>> > Failure in initialisation: java.lang.NegativeArraySizeException
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
> >>> > [artemis-commons-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
> >>> > [artemis-server-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at
> >>> >
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
> >>> > [artemis-cli-2.6.2.jar:2.6.2]
> >>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> > [rt.jar:1.8.0_171]
> >>> > at
> >>> >
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > [rt.jar:1.8.0_171]
> >>> > at java.lang.reflect.Method.invoke(Method.java:498)
> [rt.jar:1.8.0_171]
> >>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> > [artemis-boot.jar:2.6.2]
> >>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> > [artemis-boot.jar:2.6.2]
> >>> >
> >>> >
> >>> > right there in the AddressSettings.decode
> >>> >
> >>> >
> >>> > I am going to start looking into a journal export and then a journal
> >>> import
> >>> > and see what happens there.
> >>> > are programmatically created addresses / queues stored in the data
> export
> >>> > along with the address and security settings?
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
> >>> clebert.suconic@gmail.com>
> >>> > wrote:
> >>> >
> >>> >> @dan take a look at the compatibility tests in master.. We're help
> to
> >>> >> help if you don't understand anything.. it's using ClassLoaders and
> >>> >> Groovy so you can produce stuff with a combination of clients and
> >>> >> servers on 2.1.0 and consume on the current version.
> >>> >>
> >>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
> >>> >> <cl...@gmail.com> wrote:
> >>> >> > @Dan: what about this:
> >>> >> >
> >>> >> > you provide us a code path that could generate the
> incompatibility in
> >>> >> 2.1.0...
> >>> >> >
> >>> >> > an ultimate deal would be if you produced a compatibility test
> with
> >>> >> > 2.1.0.... you add some code that will generate the journal with
> 2.1.0,
> >>> >> > and then consume the messages on master (or 2.6.x)...
> >>> >> >
> >>> >> >
> >>> >> > if you generate the compatibility test showing the issue, we can
> try
> >>> >> fixing it.
> >>> >> >
> >>> >> >
> >>> >> > What about this idea? that would help you migrate into 2.6.x or
> >>> master.
> >>> >> >
> >>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> >>> >> > <cl...@gmail.com> wrote:
> >>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <
> danlangford@gmail.com
> >>> >
> >>> >> wrote:
> >>> >> >>> Thank you that was very helpful. we actually do have an address
> >>> >> settings
> >>> >> >>> entry for each queue. there could be a better pattern for us.
> but
> >>> >> currently
> >>> >> >>> our automated system for creating queues creates an address
> setting
> >>> at
> >>> >> the
> >>> >> >>> same time. i will look into improved patterns.
> >>> >> >>>
> >>> >> >>> as far as upgrading goes. i agree we really want to upgrade.
> until i
> >>> >> can
> >>> >> >>> find a work around for the config-delete-queues deserialization
> bug
> >>> >> >>> introduced in 2.2.0 i brought up back in April we will not be
> able
> >>> to
> >>> >> >>> easily move.
> >>> >> >>
> >>> >> >> You should be able to move to 2.6.2. if you're not able to I
> would
> >>> >> >> like to know where it failed.
> >>> >> >>
> >>> >> >>
> >>> >> >>>
> >>> >> >>> thanks again for all the help
> >>> >> >>>
> >>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
> >>> >> clebert.suconic@gmail.com>
> >>> >> >>> wrote:
> >>> >> >>>
> >>> >> >>>> If you do not want to upgrade for any reason export the
> journal.
> >>> >> Cleani
> >>> >> >>>> uo.  Edit the text and remove the garbage (you will see)
> manually.
> >>> >>  Delete
> >>> >> >>>> all data and te import.
> >>> >> >>>>
> >>> >> >>>> (Make a backup to be safe of course)
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>> But I still recommend the upgrade.
> >>> >> >>>>
> >>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
> >>> >> >>>> clebert.suconic@gmail.com>
> >>> >> >>>> wrote:
> >>> >> >>>>
> >>> >> >>>> > The address setting is the garbage I was talking about.
> Upgrade
> >>> to
> >>> >> the
> >>> >> >>>> > latest broker and there will be a cleanup done at the load
> >>> before it
> >>> >> >>>> > starts.
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> > I highly recommend upgrade.
> >>> >> >>>> >
> >>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
> >>> >> jbertram@apache.org>
> >>> >> >>>> > wrote:
> >>> >> >>>> >
> >>> >> >>>> >> Analyzing thread dumps like this is pretty simple.  I
> generally
> >>> >> just
> >>> >> >>>> >> scroll
> >>> >> >>>> >> through and look for long stack-traces with lots of calls
> from
> >>> >> >>>> >> org.apache.activemq.artemis.  In your case every single
> thread
> >>> >> dump has
> >>> >> >>>> a
> >>> >> >>>> >> thread doing something like this:
> >>> >> >>>> >>
> >>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
> >>> >> runnable
> >>> >> >>>> >> [0x00007f9031675000]
> >>> >> >>>> >>    java.lang.Thread.State: RUNNABLE
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
> >>> >> >>>> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>> >> >>>> >>     - locked <0x000000008098be80> (a
> >>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >>> >> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>> Method)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>> >> >>>> >>     at
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>>
> >>> >>
> >>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> >> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
> >>> >> >>>> >>     at
> >>> >> >>>>
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>> >> >>>> >>     at
> >>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>> >> >>>> >>
> >>> >> >>>> >> In every case it's the "main" thread which isn't surprising
> as
> >>> >> that is
> >>> >> >>>> the
> >>> >> >>>> >> thread responsible for starting the broker.  Also, you can
> >>> pretty
> >>> >> >>>> clearly
> >>> >> >>>> >> see in the trace that this is the thread starting the
> broker,
> >>> it's
> >>> >> >>>> loading
> >>> >> >>>> >> the journals, & restoring stored configuration (either
> address
> >>> >> settings
> >>> >> >>>> or
> >>> >> >>>> >> security settings).  I've seen high broker start times when
> >>> there
> >>> >> are
> >>> >> >>>> lots
> >>> >> >>>> >> and lots of addresses and lots of and lots of settings.  Do
> >>> either
> >>> >> (or
> >>> >> >>>> >> both) of these situations apply to you?
> >>> >> >>>> >>
> >>> >> >>>> >>
> >>> >> >>>> >> Justin
> >>> >> >>>> >>
> >>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
> >>> >> danlangford@gmail.com>
> >>> >> >>>> >> wrote:
> >>> >> >>>> >>
> >>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
> >>> >> >>>> >> config-delete-queues
> >>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug
> was
> >>> >> squashed
> >>> >> >>>> in
> >>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and
> work
> >>> >> load) but
> >>> >> >>>> we
> >>> >> >>>> >> > discussed it back in April. I should go confirm that bug
> on
> >>> 2.6
> >>> >> and
> >>> >> >>>> >> make a
> >>> >> >>>> >> > jira for that
> >>> >> >>>> >> >
> >>> >> >>>> >> > Thanks
> >>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> >>> >> >>>> >> clebert.suconic@gmail.com
> >>> >> >>>> >> > >
> >>> >> >>>> >> > wrote:
> >>> >> >>>> >> >
> >>> >> >>>> >> > > There is an issue I remember where the journal would
> have
> >>> some
> >>> >> dirt
> >>> >> >>>> >> that
> >>> >> >>>> >> > > was fixed on 2.3/0.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > I would ipgrade to 2.6.2.
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
> >>> >> danlangford@gmail.com
> >>> >> >>>> >
> >>> >> >>>> >> > > wrote:
> >>> >> >>>> >> > >
> >>> >> >>>> >> > > > would you be willing to help me translate these thread
> >>> dumps?
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them.
> i
> >>> will
> >>> >> also
> >>> >> >>>> >> share
> >>> >> >>>> >> > > > the fasthread.io links for each file. (i was
> struggling
> >>> >> getting
> >>> >> >>>> >> > > > fastthread to do a combo report with the threads in
> the
> >>> >> correct
> >>> >> >>>> >> order)
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> >>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> >>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> >>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> >>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> >>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> >>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets
> bigger
> >>> >> than at
> >>> >> >>>> the
> >>> >> >>>> >> > > other
> >>> >> >>>> >> > > > sample times. but i dont know what to look for in
> here to
> >>> >> >>>> determine
> >>> >> >>>> >> > what
> >>> >> >>>> >> > > > the broker is actually during during this time.
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > thanks
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
> >>> >> >>>> jbertram@apache.org
> >>> >> >>>> >> >
> >>> >> >>>> >> > > > wrote:
> >>> >> >>>> >> > > >
> >>> >> >>>> >> > > >> The first place I would start is grabbing thread
> dumps
> >>> >> every so
> >>> >> >>>> >> often
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
> >>> >> minutes.
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> Justin
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> >>> >> >>>> >> danlangford@gmail.com>
> >>> >> >>>> >> > > >> wrote:
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > >> > so my server startup times and failover times are
> >>> growing
> >>> >> >>>> pretty
> >>> >> >>>> >> > big.
> >>> >> >>>> >> > > >> but i
> >>> >> >>>> >> > > >> > dont really know where to start looking.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
> >>> >> stamps:
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> >>> >> >>>> >> > integration.bootstrap]
> >>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:31,818 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221000:
> >>> >> >>>> >> > > >> > live Message Broker is starting with configuration
> >>> Broker
> >>> >> >>>> >> > > Configuration
> >>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
> >>> >> >>>> >> > journal,bindingsDirectory=./
> >>> >> >>>> >> > > >> >
> >>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
> >>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,462 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221012:
> >>> >> >>>> >> > > >> > Using AIO Journal
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,493 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221057:
> >>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the
> JVM max
> >>> >> size
> >>> >> >>>> >> (-Xmx).
> >>> >> >>>> >> > > >> being
> >>> >> >>>> >> > > >> > defined as 1,073,741,824
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
> >>> protocol
> >>> >> >>>> support
> >>> >> >>>> >> > for:
> >>> >> >>>> >> > > >> CORE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,555 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: AMQP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: HORNETQ
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol].
> Adding
> >>> >> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: MQTT
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
> >>> Adding
> >>> >> >>>> >> protocol
> >>> >> >>>> >> > > >> support
> >>> >> >>>> >> > > >> > for: OPENWIRE
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:11:34,556 INFO
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ221043:
> >>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol].
> Adding
> >>> >> >>>> protocol
> >>> >> >>>> >> > > support
> >>> >> >>>> >> > > >> > for: STOMP
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,963 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
> >>> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue
> ExpiryQueue in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,983 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,984 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue example in
> >>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,985 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222165:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Dead Letter Address configured for queue
> exampleQueue
> >>> in
> >>> >> >>>> >> > > AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > 08:41:38,986 WARN
> >>> >> [org.apache.activemq.artemis.core.server]
> >>> >> >>>> >> > > AMQ222166:
> >>> >> >>>> >> > > >> No
> >>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
> >>> >> >>>> >> AddressSettings
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i
> need
> >>> >> to be
> >>> >> >>>> >> > looking
> >>> >> >>>> >> > > at
> >>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need
> to be
> >>> >> looking
> >>> >> >>>> >> at
> >>> >> >>>> >> > the
> >>> >> >>>> >> > > >> > number of messages persisted on these queues? where
> >>> would
> >>> >> you
> >>> >> >>>> >> look
> >>> >> >>>> >> > to
> >>> >> >>>> >> > > >> > determine why the startup times are so long?
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
> >>> off of
> >>> >> >>>> 2.1.0
> >>> >> >>>> >> due
> >>> >> >>>> >> > > to
> >>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
> >>> think
> >>> >> i have
> >>> >> >>>> >> > > another
> >>> >> >>>> >> > > >> > thread on here (that i need to update) regarding
> that
> >>> >> upgrade
> >>> >> >>>> >> issue.
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
> >>> >> >>>> >> > > >> >
> >>> >> >>>> >> > > >>
> >>> >> >>>> >> > > > --
> >>> >> >>>> >> > > Clebert Suconic
> >>> >> >>>> >> > >
> >>> >> >>>> >> >
> >>> >> >>>> >>
> >>> >> >>>> > --
> >>> >> >>>> > Clebert Suconic
> >>> >> >>>> >
> >>> >> >>>> --
> >>> >> >>>> Clebert Suconic
> >>> >> >>>>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> --
> >>> >> >> Clebert Suconic
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > --
> >>> >> > Clebert Suconic
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Clebert Suconic
> >>> >>
> >>>
> >>>
> >>>
> >>> --
> >>> Clebert Suconic
> >>>
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

I'm considering doing a 2.6.3 very soon.. it will include the fix..
you should be able to move from 2.1.0 . let me know if you have any
other issues.

On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic
<cl...@gmail.com> wrote:
> https://github.com/apache/activemq-artemis/pull/2214
>
>
> I will make it into 2.6.x as soon as the PR tests are finished.
>
>
> If you could try a snapshot of master or 2.6.x (after I merge it of
> course). Let me know if you need help buildling it.
>
> On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com> wrote:
>> Michael for the win!
>> @Clebert im not opposed to coding up a test case it is just that i am
>> swamped at the moment so im glad to hear that Michael already has a bead on
>> this
>>
>> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
>> imp* on a clean 2.6.2. the new system didnt have Address Settings and
>> Security Settings that had previously been added via the jolokia api. i
>> should look into the export options more to see if i just failed to provide
>> a command line argument. fortunately in my scenario it was pretty trivial
>> to recreate those settings quickly. so i did that. the import created all
>> the addresses, routes, queues, and messages that i needed. it went pretty
>> well. so my cluster is now on 2.6.2
>>
>> i plan to rework our usage of AddressSetting to not create a new entry for
>> every Address we create but instead only create them for the few exceptions
>> to our default configuration
>>
>> then i can start working on running the broker in CF with the cluster in
>> more of a Master-Master mode.
>> thanks again for all your help. feel free to respond if there is any more
>> data you need from me.
>>
>> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
>> while handshaking has occurred. " but thats another issue i will figure
>> out. *
>>
>> my 30 minute startup times are now down to 30 seconds max. more like 10-15
>> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
>>
>> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
>> <mi...@me.com.invalid> wrote:
>>
>>> Clebert im able to recreate this and also worked out in code why this
>>> happens. Ill try catch you on irc tomorrow or friday.
>>>
>>> Sent from my Samsung Galaxy smartphone.
>>> -------- Original message --------From: Clebert Suconic <
>>> clebert.suconic@gmail.com> Date: 25/07/2018  20:20  (GMT+00:00) To:
>>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes
>>> to boot & failover
>>> Yes.. it should include it.
>>>
>>>
>>> Do you have an example of the API you used with 2.1?
>>>
>>>
>>> I can't write a compatibility test this week.. but if you provide me
>>> the example I will do it early next week.
>>>
>>>
>>> In a perfect world, if you did it .. it would be great :)
>>>
>>>
>>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
>>> wrote:
>>> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
>>> > tried a few months ago. unfortunately there was not. it may have failed
>>> > much faster than 2.5.0 i dont recall the timing specifics but the error
>>> is
>>> > the same
>>> >
>>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
>>> > Failure in initialisation: java.lang.NegativeArraySizeException
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
>>> > [artemis-commons-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
>>> > [artemis-server-2.6.2.jar:2.6.2]
>>> > at
>>> >
>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at
>>> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
>>> > [artemis-cli-2.6.2.jar:2.6.2]
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > [rt.jar:1.8.0_171]
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> > [rt.jar:1.8.0_171]
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > [rt.jar:1.8.0_171]
>>> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
>>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>> > [artemis-boot.jar:2.6.2]
>>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>> > [artemis-boot.jar:2.6.2]
>>> >
>>> >
>>> > right there in the AddressSettings.decode
>>> >
>>> >
>>> > I am going to start looking into a journal export and then a journal
>>> import
>>> > and see what happens there.
>>> > are programmatically created addresses / queues stored in the data export
>>> > along with the address and security settings?
>>> >
>>> >
>>> >
>>> >
>>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> > wrote:
>>> >
>>> >> @dan take a look at the compatibility tests in master.. We're help to
>>> >> help if you don't understand anything.. it's using ClassLoaders and
>>> >> Groovy so you can produce stuff with a combination of clients and
>>> >> servers on 2.1.0 and consume on the current version.
>>> >>
>>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
>>> >> <cl...@gmail.com> wrote:
>>> >> > @Dan: what about this:
>>> >> >
>>> >> > you provide us a code path that could generate the incompatibility in
>>> >> 2.1.0...
>>> >> >
>>> >> > an ultimate deal would be if you produced a compatibility test with
>>> >> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
>>> >> > and then consume the messages on master (or 2.6.x)...
>>> >> >
>>> >> >
>>> >> > if you generate the compatibility test showing the issue, we can try
>>> >> fixing it.
>>> >> >
>>> >> >
>>> >> > What about this idea? that would help you migrate into 2.6.x or
>>> master.
>>> >> >
>>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
>>> >> > <cl...@gmail.com> wrote:
>>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangford@gmail.com
>>> >
>>> >> wrote:
>>> >> >>> Thank you that was very helpful. we actually do have an address
>>> >> settings
>>> >> >>> entry for each queue. there could be a better pattern for us. but
>>> >> currently
>>> >> >>> our automated system for creating queues creates an address setting
>>> at
>>> >> the
>>> >> >>> same time. i will look into improved patterns.
>>> >> >>>
>>> >> >>> as far as upgrading goes. i agree we really want to upgrade. until i
>>> >> can
>>> >> >>> find a work around for the config-delete-queues deserialization bug
>>> >> >>> introduced in 2.2.0 i brought up back in April we will not be able
>>> to
>>> >> >>> easily move.
>>> >> >>
>>> >> >> You should be able to move to 2.6.2. if you're not able to I would
>>> >> >> like to know where it failed.
>>> >> >>
>>> >> >>
>>> >> >>>
>>> >> >>> thanks again for all the help
>>> >> >>>
>>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
>>> >> clebert.suconic@gmail.com>
>>> >> >>> wrote:
>>> >> >>>
>>> >> >>>> If you do not want to upgrade for any reason export the journal.
>>> >> Cleani
>>> >> >>>> uo.  Edit the text and remove the garbage (you will see) manually.
>>> >>  Delete
>>> >> >>>> all data and te import.
>>> >> >>>>
>>> >> >>>> (Make a backup to be safe of course)
>>> >> >>>>
>>> >> >>>>
>>> >> >>>> But I still recommend the upgrade.
>>> >> >>>>
>>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>>> >> >>>> clebert.suconic@gmail.com>
>>> >> >>>> wrote:
>>> >> >>>>
>>> >> >>>> > The address setting is the garbage I was talking about.  Upgrade
>>> to
>>> >> the
>>> >> >>>> > latest broker and there will be a cleanup done at the load
>>> before it
>>> >> >>>> > starts.
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > I highly recommend upgrade.
>>> >> >>>> >
>>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
>>> >> jbertram@apache.org>
>>> >> >>>> > wrote:
>>> >> >>>> >
>>> >> >>>> >> Analyzing thread dumps like this is pretty simple.  I generally
>>> >> just
>>> >> >>>> >> scroll
>>> >> >>>> >> through and look for long stack-traces with lots of calls from
>>> >> >>>> >> org.apache.activemq.artemis.  In your case every single thread
>>> >> dump has
>>> >> >>>> a
>>> >> >>>> >> thread doing something like this:
>>> >> >>>> >>
>>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
>>> >> runnable
>>> >> >>>> >> [0x00007f9031675000]
>>> >> >>>> >>    java.lang.Thread.State: RUNNABLE
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
>>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
>>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>> >> >>>> >>     - locked <0x000000008098be80> (a
>>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
>>> >> >>>> >>     at
>>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>>> >> >>>> >>     at
>>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>>> >> >>>> >>     at
>>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>>> >> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> >> >>>> >>     at
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>>
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>>> >> >>>> >>     at
>>> >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>> >> >>>> >>     at
>>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>> >> >>>> >>
>>> >> >>>> >> In every case it's the "main" thread which isn't surprising as
>>> >> that is
>>> >> >>>> the
>>> >> >>>> >> thread responsible for starting the broker.  Also, you can
>>> pretty
>>> >> >>>> clearly
>>> >> >>>> >> see in the trace that this is the thread starting the broker,
>>> it's
>>> >> >>>> loading
>>> >> >>>> >> the journals, & restoring stored configuration (either address
>>> >> settings
>>> >> >>>> or
>>> >> >>>> >> security settings).  I've seen high broker start times when
>>> there
>>> >> are
>>> >> >>>> lots
>>> >> >>>> >> and lots of addresses and lots of and lots of settings.  Do
>>> either
>>> >> (or
>>> >> >>>> >> both) of these situations apply to you?
>>> >> >>>> >>
>>> >> >>>> >>
>>> >> >>>> >> Justin
>>> >> >>>> >>
>>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
>>> >> danlangford@gmail.com>
>>> >> >>>> >> wrote:
>>> >> >>>> >>
>>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>>> >> >>>> >> config-delete-queues
>>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
>>> >> squashed
>>> >> >>>> in
>>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
>>> >> load) but
>>> >> >>>> we
>>> >> >>>> >> > discussed it back in April. I should go confirm that bug on
>>> 2.6
>>> >> and
>>> >> >>>> >> make a
>>> >> >>>> >> > jira for that
>>> >> >>>> >> >
>>> >> >>>> >> > Thanks
>>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>>> >> >>>> >> clebert.suconic@gmail.com
>>> >> >>>> >> > >
>>> >> >>>> >> > wrote:
>>> >> >>>> >> >
>>> >> >>>> >> > > There is an issue I remember where the journal would have
>>> some
>>> >> dirt
>>> >> >>>> >> that
>>> >> >>>> >> > > was fixed on 2.3/0.
>>> >> >>>> >> > >
>>> >> >>>> >> > > I would ipgrade to 2.6.2.
>>> >> >>>> >> > >
>>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
>>> >> danlangford@gmail.com
>>> >> >>>> >
>>> >> >>>> >> > > wrote:
>>> >> >>>> >> > >
>>> >> >>>> >> > > > would you be willing to help me translate these thread
>>> dumps?
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i
>>> will
>>> >> also
>>> >> >>>> >> share
>>> >> >>>> >> > > > the fasthread.io links for each file. (i was struggling
>>> >> getting
>>> >> >>>> >> > > > fastthread to do a combo report with the threads in the
>>> >> correct
>>> >> >>>> >> order)
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
>>> >> than at
>>> >> >>>> the
>>> >> >>>> >> > > other
>>> >> >>>> >> > > > sample times. but i dont know what to look for in here to
>>> >> >>>> determine
>>> >> >>>> >> > what
>>> >> >>>> >> > > > the broker is actually during during this time.
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > thanks
>>> >> >>>> >> > > >
>>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>>> >> >>>> jbertram@apache.org
>>> >> >>>> >> >
>>> >> >>>> >> > > > wrote:
>>> >> >>>> >> > > >
>>> >> >>>> >> > > >> The first place I would start is grabbing thread dumps
>>> >> every so
>>> >> >>>> >> often
>>> >> >>>> >> > to
>>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
>>> >> minutes.
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> Justin
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>>> >> >>>> >> danlangford@gmail.com>
>>> >> >>>> >> > > >> wrote:
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > >> > so my server startup times and failover times are
>>> growing
>>> >> >>>> pretty
>>> >> >>>> >> > big.
>>> >> >>>> >> > > >> but i
>>> >> >>>> >> > > >> > dont really know where to start looking.
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
>>> >> stamps:
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>>> >> >>>> >> > integration.bootstrap]
>>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:31,818 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221000:
>>> >> >>>> >> > > >> > live Message Broker is starting with configuration
>>> Broker
>>> >> >>>> >> > > Configuration
>>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
>>> >> >>>> >> > journal,bindingsDirectory=./
>>> >> >>>> >> > > >> >
>>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
>>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,462 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221012:
>>> >> >>>> >> > > >> > Using AIO Journal
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,493 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221057:
>>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
>>> >> size
>>> >> >>>> >> (-Xmx).
>>> >> >>>> >> > > >> being
>>> >> >>>> >> > > >> > defined as 1,073,741,824
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,555 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
>>> protocol
>>> >> >>>> support
>>> >> >>>> >> > for:
>>> >> >>>> >> > > >> CORE
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,555 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
>>> >> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: AMQP
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
>>> Adding
>>> >> >>>> >> protocol
>>> >> >>>> >> > > >> support
>>> >> >>>> >> > > >> > for: HORNETQ
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
>>> >> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: MQTT
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
>>> Adding
>>> >> >>>> >> protocol
>>> >> >>>> >> > > >> support
>>> >> >>>> >> > > >> > for: OPENWIRE
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:11:34,556 INFO
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ221043:
>>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>>> >> >>>> protocol
>>> >> >>>> >> > > support
>>> >> >>>> >> > > >> > for: STOMP
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,963 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,963 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
>>> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,983 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>>> >> >>>> >> > > AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,983 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,984 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,985 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue example in
>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,985 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222165:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue
>>> in
>>> >> >>>> >> > > AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > 08:41:38,986 WARN
>>> >> [org.apache.activemq.artemis.core.server]
>>> >> >>>> >> > > AMQ222166:
>>> >> >>>> >> > > >> No
>>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>>> >> >>>> >> AddressSettings
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
>>> >> to be
>>> >> >>>> >> > looking
>>> >> >>>> >> > > at
>>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
>>> >> looking
>>> >> >>>> >> at
>>> >> >>>> >> > the
>>> >> >>>> >> > > >> > number of messages persisted on these queues? where
>>> would
>>> >> you
>>> >> >>>> >> look
>>> >> >>>> >> > to
>>> >> >>>> >> > > >> > determine why the startup times are so long?
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
>>> off of
>>> >> >>>> 2.1.0
>>> >> >>>> >> due
>>> >> >>>> >> > > to
>>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
>>> think
>>> >> i have
>>> >> >>>> >> > > another
>>> >> >>>> >> > > >> > thread on here (that i need to update) regarding that
>>> >> upgrade
>>> >> >>>> >> issue.
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
>>> >> >>>> >> > > >> >
>>> >> >>>> >> > > >>
>>> >> >>>> >> > > > --
>>> >> >>>> >> > > Clebert Suconic
>>> >> >>>> >> > >
>>> >> >>>> >> >
>>> >> >>>> >>
>>> >> >>>> > --
>>> >> >>>> > Clebert Suconic
>>> >> >>>> >
>>> >> >>>> --
>>> >> >>>> Clebert Suconic
>>> >> >>>>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >> Clebert Suconic
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Clebert Suconic
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Clebert Suconic
>>> >>
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>>
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

https://github.com/apache/activemq-artemis/pull/2214


I will make it into 2.6.x as soon as the PR tests are finished.


If you could try a snapshot of master or 2.6.x (after I merge it of
course). Let me know if you need help buildling it.

On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford <da...@gmail.com> wrote:
> Michael for the win!
> @Clebert im not opposed to coding up a test case it is just that i am
> swamped at the moment so im glad to hear that Michael already has a bead on
> this
>
> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
> imp* on a clean 2.6.2. the new system didnt have Address Settings and
> Security Settings that had previously been added via the jolokia api. i
> should look into the export options more to see if i just failed to provide
> a command line argument. fortunately in my scenario it was pretty trivial
> to recreate those settings quickly. so i did that. the import created all
> the addresses, routes, queues, and messages that i needed. it went pretty
> well. so my cluster is now on 2.6.2
>
> i plan to rework our usage of AddressSetting to not create a new entry for
> every Address we create but instead only create them for the few exceptions
> to our default configuration
>
> then i can start working on running the broker in CF with the cluster in
> more of a Master-Master mode.
> thanks again for all your help. feel free to respond if there is any more
> data you need from me.
>
> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
> while handshaking has occurred. " but thats another issue i will figure
> out. *
>
> my 30 minute startup times are now down to 30 seconds max. more like 10-15
> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.
>
> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
> <mi...@me.com.invalid> wrote:
>
>> Clebert im able to recreate this and also worked out in code why this
>> happens. Ill try catch you on irc tomorrow or friday.
>>
>> Sent from my Samsung Galaxy smartphone.
>> -------- Original message --------From: Clebert Suconic <
>> clebert.suconic@gmail.com> Date: 25/07/2018  20:20  (GMT+00:00) To:
>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes
>> to boot & failover
>> Yes.. it should include it.
>>
>>
>> Do you have an example of the API you used with 2.1?
>>
>>
>> I can't write a compatibility test this week.. but if you provide me
>> the example I will do it early next week.
>>
>>
>> In a perfect world, if you did it .. it would be great :)
>>
>>
>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
>> wrote:
>> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
>> > tried a few months ago. unfortunately there was not. it may have failed
>> > much faster than 2.5.0 i dont recall the timing specifics but the error
>> is
>> > the same
>> >
>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
>> > Failure in initialisation: java.lang.NegativeArraySizeException
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
>> > [artemis-commons-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
>> > [artemis-server-2.6.2.jar:2.6.2]
>> > at
>> >
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at
>> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
>> > [artemis-cli-2.6.2.jar:2.6.2]
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > [rt.jar:1.8.0_171]
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > [rt.jar:1.8.0_171]
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > [rt.jar:1.8.0_171]
>> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
>> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> > [artemis-boot.jar:2.6.2]
>> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> > [artemis-boot.jar:2.6.2]
>> >
>> >
>> > right there in the AddressSettings.decode
>> >
>> >
>> > I am going to start looking into a journal export and then a journal
>> import
>> > and see what happens there.
>> > are programmatically created addresses / queues stored in the data export
>> > along with the address and security settings?
>> >
>> >
>> >
>> >
>> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> > wrote:
>> >
>> >> @dan take a look at the compatibility tests in master.. We're help to
>> >> help if you don't understand anything.. it's using ClassLoaders and
>> >> Groovy so you can produce stuff with a combination of clients and
>> >> servers on 2.1.0 and consume on the current version.
>> >>
>> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
>> >> <cl...@gmail.com> wrote:
>> >> > @Dan: what about this:
>> >> >
>> >> > you provide us a code path that could generate the incompatibility in
>> >> 2.1.0...
>> >> >
>> >> > an ultimate deal would be if you produced a compatibility test with
>> >> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
>> >> > and then consume the messages on master (or 2.6.x)...
>> >> >
>> >> >
>> >> > if you generate the compatibility test showing the issue, we can try
>> >> fixing it.
>> >> >
>> >> >
>> >> > What about this idea? that would help you migrate into 2.6.x or
>> master.
>> >> >
>> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
>> >> > <cl...@gmail.com> wrote:
>> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangford@gmail.com
>> >
>> >> wrote:
>> >> >>> Thank you that was very helpful. we actually do have an address
>> >> settings
>> >> >>> entry for each queue. there could be a better pattern for us. but
>> >> currently
>> >> >>> our automated system for creating queues creates an address setting
>> at
>> >> the
>> >> >>> same time. i will look into improved patterns.
>> >> >>>
>> >> >>> as far as upgrading goes. i agree we really want to upgrade. until i
>> >> can
>> >> >>> find a work around for the config-delete-queues deserialization bug
>> >> >>> introduced in 2.2.0 i brought up back in April we will not be able
>> to
>> >> >>> easily move.
>> >> >>
>> >> >> You should be able to move to 2.6.2. if you're not able to I would
>> >> >> like to know where it failed.
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> thanks again for all the help
>> >> >>>
>> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
>> >> clebert.suconic@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> If you do not want to upgrade for any reason export the journal.
>> >> Cleani
>> >> >>>> uo.  Edit the text and remove the garbage (you will see) manually.
>> >>  Delete
>> >> >>>> all data and te import.
>> >> >>>>
>> >> >>>> (Make a backup to be safe of course)
>> >> >>>>
>> >> >>>>
>> >> >>>> But I still recommend the upgrade.
>> >> >>>>
>> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>> >> >>>> clebert.suconic@gmail.com>
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>> > The address setting is the garbage I was talking about.  Upgrade
>> to
>> >> the
>> >> >>>> > latest broker and there will be a cleanup done at the load
>> before it
>> >> >>>> > starts.
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > I highly recommend upgrade.
>> >> >>>> >
>> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
>> >> jbertram@apache.org>
>> >> >>>> > wrote:
>> >> >>>> >
>> >> >>>> >> Analyzing thread dumps like this is pretty simple.  I generally
>> >> just
>> >> >>>> >> scroll
>> >> >>>> >> through and look for long stack-traces with lots of calls from
>> >> >>>> >> org.apache.activemq.artemis.  In your case every single thread
>> >> dump has
>> >> >>>> a
>> >> >>>> >> thread doing something like this:
>> >> >>>> >>
>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
>> >> runnable
>> >> >>>> >> [0x00007f9031675000]
>> >> >>>> >>    java.lang.Thread.State: RUNNABLE
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>> >> >>>> >>     - locked <0x0000000080a8fce8> (a
>> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> >> >>>> >>     - locked <0x000000008098be80> (a
>> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
>> >> >>>> >>     at
>> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>>
>> >>
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>> >> >>>> >>     at
>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>> >> >>>> >>     at
>> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>> >> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >> >>>> >>     at
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>> >> >>>> >>     at
>> >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> >> >>>> >>     at
>> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> >> >>>> >>
>> >> >>>> >> In every case it's the "main" thread which isn't surprising as
>> >> that is
>> >> >>>> the
>> >> >>>> >> thread responsible for starting the broker.  Also, you can
>> pretty
>> >> >>>> clearly
>> >> >>>> >> see in the trace that this is the thread starting the broker,
>> it's
>> >> >>>> loading
>> >> >>>> >> the journals, & restoring stored configuration (either address
>> >> settings
>> >> >>>> or
>> >> >>>> >> security settings).  I've seen high broker start times when
>> there
>> >> are
>> >> >>>> lots
>> >> >>>> >> and lots of addresses and lots of and lots of settings.  Do
>> either
>> >> (or
>> >> >>>> >> both) of these situations apply to you?
>> >> >>>> >>
>> >> >>>> >>
>> >> >>>> >> Justin
>> >> >>>> >>
>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
>> >> danlangford@gmail.com>
>> >> >>>> >> wrote:
>> >> >>>> >>
>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>> >> >>>> >> config-delete-queues
>> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
>> >> squashed
>> >> >>>> in
>> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
>> >> load) but
>> >> >>>> we
>> >> >>>> >> > discussed it back in April. I should go confirm that bug on
>> 2.6
>> >> and
>> >> >>>> >> make a
>> >> >>>> >> > jira for that
>> >> >>>> >> >
>> >> >>>> >> > Thanks
>> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>> >> >>>> >> clebert.suconic@gmail.com
>> >> >>>> >> > >
>> >> >>>> >> > wrote:
>> >> >>>> >> >
>> >> >>>> >> > > There is an issue I remember where the journal would have
>> some
>> >> dirt
>> >> >>>> >> that
>> >> >>>> >> > > was fixed on 2.3/0.
>> >> >>>> >> > >
>> >> >>>> >> > > I would ipgrade to 2.6.2.
>> >> >>>> >> > >
>> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
>> >> danlangford@gmail.com
>> >> >>>> >
>> >> >>>> >> > > wrote:
>> >> >>>> >> > >
>> >> >>>> >> > > > would you be willing to help me translate these thread
>> dumps?
>> >> >>>> >> > > >
>> >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i
>> will
>> >> also
>> >> >>>> >> share
>> >> >>>> >> > > > the fasthread.io links for each file. (i was struggling
>> >> getting
>> >> >>>> >> > > > fastthread to do a combo report with the threads in the
>> >> correct
>> >> >>>> >> order)
>> >> >>>> >> > > >
>> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>> >> >>>> >> > > >
>> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
>> >> than at
>> >> >>>> the
>> >> >>>> >> > > other
>> >> >>>> >> > > > sample times. but i dont know what to look for in here to
>> >> >>>> determine
>> >> >>>> >> > what
>> >> >>>> >> > > > the broker is actually during during this time.
>> >> >>>> >> > > >
>> >> >>>> >> > > > thanks
>> >> >>>> >> > > >
>> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>> >> >>>> jbertram@apache.org
>> >> >>>> >> >
>> >> >>>> >> > > > wrote:
>> >> >>>> >> > > >
>> >> >>>> >> > > >> The first place I would start is grabbing thread dumps
>> >> every so
>> >> >>>> >> often
>> >> >>>> >> > to
>> >> >>>> >> > > >> see what the broker is actually doing during the 30+
>> >> minutes.
>> >> >>>> >> > > >>
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> Justin
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>> >> >>>> >> danlangford@gmail.com>
>> >> >>>> >> > > >> wrote:
>> >> >>>> >> > > >>
>> >> >>>> >> > > >> > so my server startup times and failover times are
>> growing
>> >> >>>> pretty
>> >> >>>> >> > big.
>> >> >>>> >> > > >> but i
>> >> >>>> >> > > >> > dont really know where to start looking.
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
>> >> stamps:
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>> >> >>>> >> > integration.bootstrap]
>> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:31,818 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221000:
>> >> >>>> >> > > >> > live Message Broker is starting with configuration
>> Broker
>> >> >>>> >> > > Configuration
>> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
>> >> >>>> >> > journal,bindingsDirectory=./
>> >> >>>> >> > > >> >
>> >> data/bindings,largeMessagesDirectory=./data/large-messages,
>> >> >>>> >> > > >> > pagingDirectory=./data/paging)
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,462 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221012:
>> >> >>>> >> > > >> > Using AIO Journal
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,493 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221057:
>> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
>> >> size
>> >> >>>> >> (-Xmx).
>> >> >>>> >> > > >> being
>> >> >>>> >> > > >> > defined as 1,073,741,824
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,555 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
>> protocol
>> >> >>>> support
>> >> >>>> >> > for:
>> >> >>>> >> > > >> CORE
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,555 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
>> >> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: AMQP
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
>> Adding
>> >> >>>> >> protocol
>> >> >>>> >> > > >> support
>> >> >>>> >> > > >> > for: HORNETQ
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
>> >> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: MQTT
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
>> Adding
>> >> >>>> >> protocol
>> >> >>>> >> > > >> support
>> >> >>>> >> > > >> > for: OPENWIRE
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:11:34,556 INFO
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ221043:
>> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>> >> >>>> protocol
>> >> >>>> >> > > support
>> >> >>>> >> > > >> > for: STOMP
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,963 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,963 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
>> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,983 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>> >> >>>> >> > > AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,983 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,984 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,985 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue example in
>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,985 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222165:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue
>> in
>> >> >>>> >> > > AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > 08:41:38,986 WARN
>> >> [org.apache.activemq.artemis.core.server]
>> >> >>>> >> > > AMQ222166:
>> >> >>>> >> > > >> No
>> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>> >> >>>> >> AddressSettings
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
>> >> to be
>> >> >>>> >> > looking
>> >> >>>> >> > > at
>> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
>> >> looking
>> >> >>>> >> at
>> >> >>>> >> > the
>> >> >>>> >> > > >> > number of messages persisted on these queues? where
>> would
>> >> you
>> >> >>>> >> look
>> >> >>>> >> > to
>> >> >>>> >> > > >> > determine why the startup times are so long?
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
>> off of
>> >> >>>> 2.1.0
>> >> >>>> >> due
>> >> >>>> >> > > to
>> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
>> think
>> >> i have
>> >> >>>> >> > > another
>> >> >>>> >> > > >> > thread on here (that i need to update) regarding that
>> >> upgrade
>> >> >>>> >> issue.
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
>> >> >>>> >> > > >> >
>> >> >>>> >> > > >>
>> >> >>>> >> > > > --
>> >> >>>> >> > > Clebert Suconic
>> >> >>>> >> > >
>> >> >>>> >> >
>> >> >>>> >>
>> >> >>>> > --
>> >> >>>> > Clebert Suconic
>> >> >>>> >
>> >> >>>> --
>> >> >>>> Clebert Suconic
>> >> >>>>
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Clebert Suconic
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Clebert Suconic
>> >>
>> >>
>> >>
>> >> --
>> >> Clebert Suconic
>> >>
>>
>>
>>
>> --
>> Clebert Suconic
>>



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

Michael for the win!
@Clebert im not opposed to coding up a test case it is just that i am
swamped at the moment so im glad to hear that Michael already has a bead on
this

i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data
imp* on a clean 2.6.2. the new system didnt have Address Settings and
Security Settings that had previously been added via the jolokia api. i
should look into the export options more to see if i just failed to provide
a command line argument. fortunately in my scenario it was pretty trivial
to recreate those settings quickly. so i did that. the import created all
the addresses, routes, queues, and messages that i needed. it went pretty
well. so my cluster is now on 2.6.2

i plan to rework our usage of AddressSetting to not create a new entry for
every Address we create but instead only create them for the few exceptions
to our default configuration

then i can start working on running the broker in CF with the cluster in
more of a Master-Master mode.
thanks again for all your help. feel free to respond if there is any more
data you need from me.

*side note: my broker is now spamming " AMQ224088: Timeout (10 seconds)
while handshaking has occurred. " but thats another issue i will figure
out. *

my 30 minute startup times are now down to 30 seconds max. more like 10-15
seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP.

On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce
<mi...@me.com.invalid> wrote:

> Clebert im able to recreate this and also worked out in code why this
> happens. Ill try catch you on irc tomorrow or friday.
>
> Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <
> clebert.suconic@gmail.com> Date: 25/07/2018  20:20  (GMT+00:00) To:
> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes
> to boot & failover
> Yes.. it should include it.
>
>
> Do you have an example of the API you used with 2.1?
>
>
> I can't write a compatibility test this week.. but if you provide me
> the example I will do it early next week.
>
>
> In a perfect world, if you did it .. it would be great :)
>
>
> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com>
> wrote:
> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
> > tried a few months ago. unfortunately there was not. it may have failed
> > much faster than 2.5.0 i dont recall the timing specifics but the error
> is
> > the same
> >
> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
> > Failure in initialisation: java.lang.NegativeArraySizeException
> > at
> >
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
> > [artemis-commons-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
> > [artemis-commons-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
> > [artemis-commons-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
> > [artemis-commons-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
> > [artemis-server-2.6.2.jar:2.6.2]
> > at
> >
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> > [artemis-cli-2.6.2.jar:2.6.2]
> > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
> > [artemis-cli-2.6.2.jar:2.6.2]
> > at
> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
> > [artemis-cli-2.6.2.jar:2.6.2]
> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
> > [artemis-cli-2.6.2.jar:2.6.2]
> > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
> > [artemis-cli-2.6.2.jar:2.6.2]
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > [rt.jar:1.8.0_171]
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > [rt.jar:1.8.0_171]
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > [rt.jar:1.8.0_171]
> > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
> > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> > [artemis-boot.jar:2.6.2]
> > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> > [artemis-boot.jar:2.6.2]
> >
> >
> > right there in the AddressSettings.decode
> >
> >
> > I am going to start looking into a journal export and then a journal
> import
> > and see what happens there.
> > are programmatically created addresses / queues stored in the data export
> > along with the address and security settings?
> >
> >
> >
> >
> > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <
> clebert.suconic@gmail.com>
> > wrote:
> >
> >> @dan take a look at the compatibility tests in master.. We're help to
> >> help if you don't understand anything.. it's using ClassLoaders and
> >> Groovy so you can produce stuff with a combination of clients and
> >> servers on 2.1.0 and consume on the current version.
> >>
> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
> >> <cl...@gmail.com> wrote:
> >> > @Dan: what about this:
> >> >
> >> > you provide us a code path that could generate the incompatibility in
> >> 2.1.0...
> >> >
> >> > an ultimate deal would be if you produced a compatibility test with
> >> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
> >> > and then consume the messages on master (or 2.6.x)...
> >> >
> >> >
> >> > if you generate the compatibility test showing the issue, we can try
> >> fixing it.
> >> >
> >> >
> >> > What about this idea? that would help you migrate into 2.6.x or
> master.
> >> >
> >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> >> > <cl...@gmail.com> wrote:
> >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangford@gmail.com
> >
> >> wrote:
> >> >>> Thank you that was very helpful. we actually do have an address
> >> settings
> >> >>> entry for each queue. there could be a better pattern for us. but
> >> currently
> >> >>> our automated system for creating queues creates an address setting
> at
> >> the
> >> >>> same time. i will look into improved patterns.
> >> >>>
> >> >>> as far as upgrading goes. i agree we really want to upgrade. until i
> >> can
> >> >>> find a work around for the config-delete-queues deserialization bug
> >> >>> introduced in 2.2.0 i brought up back in April we will not be able
> to
> >> >>> easily move.
> >> >>
> >> >> You should be able to move to 2.6.2. if you're not able to I would
> >> >> like to know where it failed.
> >> >>
> >> >>
> >> >>>
> >> >>> thanks again for all the help
> >> >>>
> >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
> >> clebert.suconic@gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> If you do not want to upgrade for any reason export the journal.
> >> Cleani
> >> >>>> uo.  Edit the text and remove the garbage (you will see) manually.
> >>  Delete
> >> >>>> all data and te import.
> >> >>>>
> >> >>>> (Make a backup to be safe of course)
> >> >>>>
> >> >>>>
> >> >>>> But I still recommend the upgrade.
> >> >>>>
> >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
> >> >>>> clebert.suconic@gmail.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>> > The address setting is the garbage I was talking about.  Upgrade
> to
> >> the
> >> >>>> > latest broker and there will be a cleanup done at the load
> before it
> >> >>>> > starts.
> >> >>>> >
> >> >>>> >
> >> >>>> > I highly recommend upgrade.
> >> >>>> >
> >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
> >> jbertram@apache.org>
> >> >>>> > wrote:
> >> >>>> >
> >> >>>> >> Analyzing thread dumps like this is pretty simple.  I generally
> >> just
> >> >>>> >> scroll
> >> >>>> >> through and look for long stack-traces with lots of calls from
> >> >>>> >> org.apache.activemq.artemis.  In your case every single thread
> >> dump has
> >> >>>> a
> >> >>>> >> thread doing something like this:
> >> >>>> >>
> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
> >> runnable
> >> >>>> >> [0x00007f9031675000]
> >> >>>> >>    java.lang.Thread.State: RUNNABLE
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >> >>>> >>     - locked <0x0000000080a8fce8> (a
> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >> >>>> >>     - locked <0x0000000080a8fce8> (a
> >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >> >>>> >>     - locked <0x000000008098be80> (a
> >> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
> >> >>>> >>     at
> >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>>
> >>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >> >>>> >>     at
> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >> >>>> >>     at
> >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> >>>> >>     at
> >> >>>> >>
> >> >>>> >>
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
> >> >>>> >>     at
> >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >> >>>> >>     at
> >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >> >>>> >>
> >> >>>> >> In every case it's the "main" thread which isn't surprising as
> >> that is
> >> >>>> the
> >> >>>> >> thread responsible for starting the broker.  Also, you can
> pretty
> >> >>>> clearly
> >> >>>> >> see in the trace that this is the thread starting the broker,
> it's
> >> >>>> loading
> >> >>>> >> the journals, & restoring stored configuration (either address
> >> settings
> >> >>>> or
> >> >>>> >> security settings).  I've seen high broker start times when
> there
> >> are
> >> >>>> lots
> >> >>>> >> and lots of addresses and lots of and lots of settings.  Do
> either
> >> (or
> >> >>>> >> both) of these situations apply to you?
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> Justin
> >> >>>> >>
> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
> >> danlangford@gmail.com>
> >> >>>> >> wrote:
> >> >>>> >>
> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
> >> >>>> >> config-delete-queues
> >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
> >> squashed
> >> >>>> in
> >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
> >> load) but
> >> >>>> we
> >> >>>> >> > discussed it back in April. I should go confirm that bug on
> 2.6
> >> and
> >> >>>> >> make a
> >> >>>> >> > jira for that
> >> >>>> >> >
> >> >>>> >> > Thanks
> >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> >> >>>> >> clebert.suconic@gmail.com
> >> >>>> >> > >
> >> >>>> >> > wrote:
> >> >>>> >> >
> >> >>>> >> > > There is an issue I remember where the journal would have
> some
> >> dirt
> >> >>>> >> that
> >> >>>> >> > > was fixed on 2.3/0.
> >> >>>> >> > >
> >> >>>> >> > > I would ipgrade to 2.6.2.
> >> >>>> >> > >
> >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
> >> danlangford@gmail.com
> >> >>>> >
> >> >>>> >> > > wrote:
> >> >>>> >> > >
> >> >>>> >> > > > would you be willing to help me translate these thread
> dumps?
> >> >>>> >> > > >
> >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i
> will
> >> also
> >> >>>> >> share
> >> >>>> >> > > > the fasthread.io links for each file. (i was struggling
> >> getting
> >> >>>> >> > > > fastthread to do a combo report with the threads in the
> >> correct
> >> >>>> >> order)
> >> >>>> >> > > >
> >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >> >>>> >> > > >
> >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
> >> than at
> >> >>>> the
> >> >>>> >> > > other
> >> >>>> >> > > > sample times. but i dont know what to look for in here to
> >> >>>> determine
> >> >>>> >> > what
> >> >>>> >> > > > the broker is actually during during this time.
> >> >>>> >> > > >
> >> >>>> >> > > > thanks
> >> >>>> >> > > >
> >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
> >> >>>> jbertram@apache.org
> >> >>>> >> >
> >> >>>> >> > > > wrote:
> >> >>>> >> > > >
> >> >>>> >> > > >> The first place I would start is grabbing thread dumps
> >> every so
> >> >>>> >> often
> >> >>>> >> > to
> >> >>>> >> > > >> see what the broker is actually doing during the 30+
> >> minutes.
> >> >>>> >> > > >>
> >> >>>> >> > > >>
> >> >>>> >> > > >> Justin
> >> >>>> >> > > >>
> >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> >> >>>> >> danlangford@gmail.com>
> >> >>>> >> > > >> wrote:
> >> >>>> >> > > >>
> >> >>>> >> > > >> > so my server startup times and failover times are
> growing
> >> >>>> pretty
> >> >>>> >> > big.
> >> >>>> >> > > >> but i
> >> >>>> >> > > >> > dont really know where to start looking.
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > here is a snippet of some logs to show you the time
> >> stamps:
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> >> >>>> >> > integration.bootstrap]
> >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:31,818 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221000:
> >> >>>> >> > > >> > live Message Broker is starting with configuration
> Broker
> >> >>>> >> > > Configuration
> >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
> >> >>>> >> > journal,bindingsDirectory=./
> >> >>>> >> > > >> >
> >> data/bindings,largeMessagesDirectory=./data/large-messages,
> >> >>>> >> > > >> > pagingDirectory=./data/paging)
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,462 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221012:
> >> >>>> >> > > >> > Using AIO Journal
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,493 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221057:
> >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
> >> size
> >> >>>> >> (-Xmx).
> >> >>>> >> > > >> being
> >> >>>> >> > > >> > defined as 1,073,741,824
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,555 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding
> protocol
> >> >>>> support
> >> >>>> >> > for:
> >> >>>> >> > > >> CORE
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,555 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
> >> protocol
> >> >>>> >> > > support
> >> >>>> >> > > >> > for: AMQP
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,556 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol].
> Adding
> >> >>>> >> protocol
> >> >>>> >> > > >> support
> >> >>>> >> > > >> > for: HORNETQ
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,556 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
> >> protocol
> >> >>>> >> > > support
> >> >>>> >> > > >> > for: MQTT
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,556 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol].
> Adding
> >> >>>> >> protocol
> >> >>>> >> > > >> support
> >> >>>> >> > > >> > for: OPENWIRE
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:11:34,556 INFO
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ221043:
> >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
> >> >>>> protocol
> >> >>>> >> > > support
> >> >>>> >> > > >> > for: STOMP
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,963 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222165:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
> >> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,963 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222166:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Expiry Address configured for queue DLQ in
> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,983 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222165:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
> >> >>>> >> > > AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,983 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222166:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
> >> >>>> >> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,984 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222165:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Dead Letter Address configured for queue example in
> >> >>>> >> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,985 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222166:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Expiry Address configured for queue example in
> >> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,985 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222165:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue
> in
> >> >>>> >> > > AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > 08:41:38,986 WARN
> >> [org.apache.activemq.artemis.core.server]
> >> >>>> >> > > AMQ222166:
> >> >>>> >> > > >> No
> >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
> >> >>>> >> AddressSettings
> >> >>>> >> > > >> >
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
> >> to be
> >> >>>> >> > looking
> >> >>>> >> > > at
> >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
> >> looking
> >> >>>> >> at
> >> >>>> >> > the
> >> >>>> >> > > >> > number of messages persisted on these queues? where
> would
> >> you
> >> >>>> >> look
> >> >>>> >> > to
> >> >>>> >> > > >> > determine why the startup times are so long?
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade
> off of
> >> >>>> 2.1.0
> >> >>>> >> due
> >> >>>> >> > > to
> >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i
> think
> >> i have
> >> >>>> >> > > another
> >> >>>> >> > > >> > thread on here (that i need to update) regarding that
> >> upgrade
> >> >>>> >> issue.
> >> >>>> >> > > >> >
> >> >>>> >> > > >> > thanks for any ideas or insight you have for me
> >> >>>> >> > > >> >
> >> >>>> >> > > >>
> >> >>>> >> > > > --
> >> >>>> >> > > Clebert Suconic
> >> >>>> >> > >
> >> >>>> >> >
> >> >>>> >>
> >> >>>> > --
> >> >>>> > Clebert Suconic
> >> >>>> >
> >> >>>> --
> >> >>>> Clebert Suconic
> >> >>>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Clebert Suconic
> >> >
> >> >
> >> >
> >> > --
> >> > Clebert Suconic
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >>
>
>
>
> --
> Clebert Suconic
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

Yes.. it should include it.


Do you have an example of the API you used with 2.1?


I can't write a compatibility test this week.. but if you provide me
the example I will do it early next week.


In a perfect world, if you did it .. it would be great :)


On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <da...@gmail.com> wrote:
> i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
> tried a few months ago. unfortunately there was not. it may have failed
> much faster than 2.5.0 i dont recall the timing specifics but the error is
> the same
>
> 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
> Failure in initialisation: java.lang.NegativeArraySizeException
> at
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
> [artemis-commons-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
> [artemis-commons-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
> [artemis-commons-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
> [artemis-commons-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
> [artemis-server-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> [artemis-cli-2.6.2.jar:2.6.2]
> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
> [artemis-cli-2.6.2.jar:2.6.2]
> at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
> [artemis-cli-2.6.2.jar:2.6.2]
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
> [artemis-cli-2.6.2.jar:2.6.2]
> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
> [artemis-cli-2.6.2.jar:2.6.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [rt.jar:1.8.0_171]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [rt.jar:1.8.0_171]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [rt.jar:1.8.0_171]
> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
> at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> [artemis-boot.jar:2.6.2]
> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> [artemis-boot.jar:2.6.2]
>
>
> right there in the AddressSettings.decode
>
>
> I am going to start looking into a journal export and then a journal import
> and see what happens there.
> are programmatically created addresses / queues stored in the data export
> along with the address and security settings?
>
>
>
>
> On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> @dan take a look at the compatibility tests in master.. We're help to
>> help if you don't understand anything.. it's using ClassLoaders and
>> Groovy so you can produce stuff with a combination of clients and
>> servers on 2.1.0 and consume on the current version.
>>
>> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
>> <cl...@gmail.com> wrote:
>> > @Dan: what about this:
>> >
>> > you provide us a code path that could generate the incompatibility in
>> 2.1.0...
>> >
>> > an ultimate deal would be if you produced a compatibility test with
>> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
>> > and then consume the messages on master (or 2.6.x)...
>> >
>> >
>> > if you generate the compatibility test showing the issue, we can try
>> fixing it.
>> >
>> >
>> > What about this idea? that would help you migrate into 2.6.x or master.
>> >
>> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
>> > <cl...@gmail.com> wrote:
>> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <da...@gmail.com>
>> wrote:
>> >>> Thank you that was very helpful. we actually do have an address
>> settings
>> >>> entry for each queue. there could be a better pattern for us. but
>> currently
>> >>> our automated system for creating queues creates an address setting at
>> the
>> >>> same time. i will look into improved patterns.
>> >>>
>> >>> as far as upgrading goes. i agree we really want to upgrade. until i
>> can
>> >>> find a work around for the config-delete-queues deserialization bug
>> >>> introduced in 2.2.0 i brought up back in April we will not be able to
>> >>> easily move.
>> >>
>> >> You should be able to move to 2.6.2. if you're not able to I would
>> >> like to know where it failed.
>> >>
>> >>
>> >>>
>> >>> thanks again for all the help
>> >>>
>> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> >>> wrote:
>> >>>
>> >>>> If you do not want to upgrade for any reason export the journal.
>> Cleani
>> >>>> uo.  Edit the text and remove the garbage (you will see) manually.
>>  Delete
>> >>>> all data and te import.
>> >>>>
>> >>>> (Make a backup to be safe of course)
>> >>>>
>> >>>>
>> >>>> But I still recommend the upgrade.
>> >>>>
>> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>> >>>> clebert.suconic@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > The address setting is the garbage I was talking about.  Upgrade to
>> the
>> >>>> > latest broker and there will be a cleanup done at the load before it
>> >>>> > starts.
>> >>>> >
>> >>>> >
>> >>>> > I highly recommend upgrade.
>> >>>> >
>> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
>> jbertram@apache.org>
>> >>>> > wrote:
>> >>>> >
>> >>>> >> Analyzing thread dumps like this is pretty simple.  I generally
>> just
>> >>>> >> scroll
>> >>>> >> through and look for long stack-traces with lots of calls from
>> >>>> >> org.apache.activemq.artemis.  In your case every single thread
>> dump has
>> >>>> a
>> >>>> >> thread doing something like this:
>> >>>> >>
>> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
>> runnable
>> >>>> >> [0x00007f9031675000]
>> >>>> >>    java.lang.Thread.State: RUNNABLE
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>> >>>> >>     - locked <0x0000000080a8fce8> (a
>> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>> >>>> >>     - locked <0x0000000080a8fce8> (a
>> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> >>>> >>     - locked <0x000000008098be80> (a
>> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
>> >>>> >>     at
>> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>> >>>> >>     at
>> >>>> >>
>> >>>>
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>> >>>> >>     at
>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>> >>>> >>     at
>> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>>> >>     at
>> >>>> >>
>> >>>> >>
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>> >>>> >>     at
>> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> >>>> >>     at
>> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> >>>> >>
>> >>>> >> In every case it's the "main" thread which isn't surprising as
>> that is
>> >>>> the
>> >>>> >> thread responsible for starting the broker.  Also, you can pretty
>> >>>> clearly
>> >>>> >> see in the trace that this is the thread starting the broker, it's
>> >>>> loading
>> >>>> >> the journals, & restoring stored configuration (either address
>> settings
>> >>>> or
>> >>>> >> security settings).  I've seen high broker start times when there
>> are
>> >>>> lots
>> >>>> >> and lots of addresses and lots of and lots of settings.  Do either
>> (or
>> >>>> >> both) of these situations apply to you?
>> >>>> >>
>> >>>> >>
>> >>>> >> Justin
>> >>>> >>
>> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
>> danlangford@gmail.com>
>> >>>> >> wrote:
>> >>>> >>
>> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>> >>>> >> config-delete-queues
>> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
>> squashed
>> >>>> in
>> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
>> load) but
>> >>>> we
>> >>>> >> > discussed it back in April. I should go confirm that bug on 2.6
>> and
>> >>>> >> make a
>> >>>> >> > jira for that
>> >>>> >> >
>> >>>> >> > Thanks
>> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>> >>>> >> clebert.suconic@gmail.com
>> >>>> >> > >
>> >>>> >> > wrote:
>> >>>> >> >
>> >>>> >> > > There is an issue I remember where the journal would have some
>> dirt
>> >>>> >> that
>> >>>> >> > > was fixed on 2.3/0.
>> >>>> >> > >
>> >>>> >> > > I would ipgrade to 2.6.2.
>> >>>> >> > >
>> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
>> danlangford@gmail.com
>> >>>> >
>> >>>> >> > > wrote:
>> >>>> >> > >
>> >>>> >> > > > would you be willing to help me translate these thread dumps?
>> >>>> >> > > >
>> >>>> >> > > > i attached a Zip file with some thread dumps in them. i will
>> also
>> >>>> >> share
>> >>>> >> > > > the fasthread.io links for each file. (i was struggling
>> getting
>> >>>> >> > > > fastthread to do a combo report with the threads in the
>> correct
>> >>>> >> order)
>> >>>> >> > > >
>> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>> >>>> >> > > >
>> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
>> than at
>> >>>> the
>> >>>> >> > > other
>> >>>> >> > > > sample times. but i dont know what to look for in here to
>> >>>> determine
>> >>>> >> > what
>> >>>> >> > > > the broker is actually during during this time.
>> >>>> >> > > >
>> >>>> >> > > > thanks
>> >>>> >> > > >
>> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>> >>>> jbertram@apache.org
>> >>>> >> >
>> >>>> >> > > > wrote:
>> >>>> >> > > >
>> >>>> >> > > >> The first place I would start is grabbing thread dumps
>> every so
>> >>>> >> often
>> >>>> >> > to
>> >>>> >> > > >> see what the broker is actually doing during the 30+
>> minutes.
>> >>>> >> > > >>
>> >>>> >> > > >>
>> >>>> >> > > >> Justin
>> >>>> >> > > >>
>> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>> >>>> >> danlangford@gmail.com>
>> >>>> >> > > >> wrote:
>> >>>> >> > > >>
>> >>>> >> > > >> > so my server startup times and failover times are growing
>> >>>> pretty
>> >>>> >> > big.
>> >>>> >> > > >> but i
>> >>>> >> > > >> > dont really know where to start looking.
>> >>>> >> > > >> >
>> >>>> >> > > >> > here is a snippet of some logs to show you the time
>> stamps:
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>> >>>> >> > integration.bootstrap]
>> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:31,818 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221000:
>> >>>> >> > > >> > live Message Broker is starting with configuration Broker
>> >>>> >> > > Configuration
>> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
>> >>>> >> > journal,bindingsDirectory=./
>> >>>> >> > > >> >
>> data/bindings,largeMessagesDirectory=./data/large-messages,
>> >>>> >> > > >> > pagingDirectory=./data/paging)
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,462 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221012:
>> >>>> >> > > >> > Using AIO Journal
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,493 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221057:
>> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
>> size
>> >>>> >> (-Xmx).
>> >>>> >> > > >> being
>> >>>> >> > > >> > defined as 1,073,741,824
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,555 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
>> >>>> support
>> >>>> >> > for:
>> >>>> >> > > >> CORE
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,555 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
>> protocol
>> >>>> >> > > support
>> >>>> >> > > >> > for: AMQP
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,556 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
>> >>>> >> protocol
>> >>>> >> > > >> support
>> >>>> >> > > >> > for: HORNETQ
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,556 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
>> protocol
>> >>>> >> > > support
>> >>>> >> > > >> > for: MQTT
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,556 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
>> >>>> >> protocol
>> >>>> >> > > >> support
>> >>>> >> > > >> > for: OPENWIRE
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:11:34,556 INFO
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ221043:
>> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>> >>>> protocol
>> >>>> >> > > support
>> >>>> >> > > >> > for: STOMP
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,963 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222165:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
>> AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,963 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222166:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,983 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222165:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>> >>>> >> > > AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,983 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222166:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>> >>>> >> AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,984 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222165:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Dead Letter Address configured for queue example in
>> >>>> >> AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,985 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222166:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Expiry Address configured for queue example in
>> AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,985 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222165:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue in
>> >>>> >> > > AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> > 08:41:38,986 WARN
>> [org.apache.activemq.artemis.core.server]
>> >>>> >> > > AMQ222166:
>> >>>> >> > > >> No
>> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>> >>>> >> AddressSettings
>> >>>> >> > > >> >
>> >>>> >> > > >> >
>> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
>> to be
>> >>>> >> > looking
>> >>>> >> > > at
>> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
>> looking
>> >>>> >> at
>> >>>> >> > the
>> >>>> >> > > >> > number of messages persisted on these queues? where would
>> you
>> >>>> >> look
>> >>>> >> > to
>> >>>> >> > > >> > determine why the startup times are so long?
>> >>>> >> > > >> >
>> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
>> >>>> 2.1.0
>> >>>> >> due
>> >>>> >> > > to
>> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i think
>> i have
>> >>>> >> > > another
>> >>>> >> > > >> > thread on here (that i need to update) regarding that
>> upgrade
>> >>>> >> issue.
>> >>>> >> > > >> >
>> >>>> >> > > >> > thanks for any ideas or insight you have for me
>> >>>> >> > > >> >
>> >>>> >> > > >>
>> >>>> >> > > > --
>> >>>> >> > > Clebert Suconic
>> >>>> >> > >
>> >>>> >> >
>> >>>> >>
>> >>>> > --
>> >>>> > Clebert Suconic
>> >>>> >
>> >>>> --
>> >>>> Clebert Suconic
>> >>>>
>> >>
>> >>
>> >>
>> >> --
>> >> Clebert Suconic
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>>



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i
tried a few months ago. unfortunately there was not. it may have failed
much faster than 2.5.0 i dont recall the timing specifics but the error is
the same

11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000:
Failure in initialisation: java.lang.NegativeArraySizeException
at
org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182)
[artemis-commons-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171)
[artemis-commons-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158)
[artemis-commons-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69)
[artemis-commons-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474)
[artemis-server-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
[artemis-cli-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82)
[artemis-cli-2.6.2.jar:2.6.2]
at
org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149)
[artemis-cli-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97)
[artemis-cli-2.6.2.jar:2.6.2]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124)
[artemis-cli-2.6.2.jar:2.6.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[rt.jar:1.8.0_171]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[rt.jar:1.8.0_171]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[rt.jar:1.8.0_171]
at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
[artemis-boot.jar:2.6.2]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
[artemis-boot.jar:2.6.2]


right there in the AddressSettings.decode


I am going to start looking into a journal export and then a journal import
and see what happens there.
are programmatically created addresses / queues stored in the data export
along with the address and security settings?




On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic <cl...@gmail.com>
wrote:

> @dan take a look at the compatibility tests in master.. We're help to
> help if you don't understand anything.. it's using ClassLoaders and
> Groovy so you can produce stuff with a combination of clients and
> servers on 2.1.0 and consume on the current version.
>
> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
> <cl...@gmail.com> wrote:
> > @Dan: what about this:
> >
> > you provide us a code path that could generate the incompatibility in
> 2.1.0...
> >
> > an ultimate deal would be if you produced a compatibility test with
> > 2.1.0.... you add some code that will generate the journal with 2.1.0,
> > and then consume the messages on master (or 2.6.x)...
> >
> >
> > if you generate the compatibility test showing the issue, we can try
> fixing it.
> >
> >
> > What about this idea? that would help you migrate into 2.6.x or master.
> >
> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> > <cl...@gmail.com> wrote:
> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <da...@gmail.com>
> wrote:
> >>> Thank you that was very helpful. we actually do have an address
> settings
> >>> entry for each queue. there could be a better pattern for us. but
> currently
> >>> our automated system for creating queues creates an address setting at
> the
> >>> same time. i will look into improved patterns.
> >>>
> >>> as far as upgrading goes. i agree we really want to upgrade. until i
> can
> >>> find a work around for the config-delete-queues deserialization bug
> >>> introduced in 2.2.0 i brought up back in April we will not be able to
> >>> easily move.
> >>
> >> You should be able to move to 2.6.2. if you're not able to I would
> >> like to know where it failed.
> >>
> >>
> >>>
> >>> thanks again for all the help
> >>>
> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <
> clebert.suconic@gmail.com>
> >>> wrote:
> >>>
> >>>> If you do not want to upgrade for any reason export the journal.
> Cleani
> >>>> uo.  Edit the text and remove the garbage (you will see) manually.
>  Delete
> >>>> all data and te import.
> >>>>
> >>>> (Make a backup to be safe of course)
> >>>>
> >>>>
> >>>> But I still recommend the upgrade.
> >>>>
> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
> >>>> clebert.suconic@gmail.com>
> >>>> wrote:
> >>>>
> >>>> > The address setting is the garbage I was talking about.  Upgrade to
> the
> >>>> > latest broker and there will be a cleanup done at the load before it
> >>>> > starts.
> >>>> >
> >>>> >
> >>>> > I highly recommend upgrade.
> >>>> >
> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <
> jbertram@apache.org>
> >>>> > wrote:
> >>>> >
> >>>> >> Analyzing thread dumps like this is pretty simple.  I generally
> just
> >>>> >> scroll
> >>>> >> through and look for long stack-traces with lots of calls from
> >>>> >> org.apache.activemq.artemis.  In your case every single thread
> dump has
> >>>> a
> >>>> >> thread doing something like this:
> >>>> >>
> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe
> runnable
> >>>> >> [0x00007f9031675000]
> >>>> >>    java.lang.Thread.State: RUNNABLE
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >>>> >>     - locked <0x0000000080a8fce8> (a
> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >>>> >>     - locked <0x0000000080a8fce8> (a
> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>>> >>     - locked <0x000000008098be80> (a
> >>>> >> org.apache.activemq.artemis.integration.FileBroker)
> >>>> >>     at
> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >>>> >>     at
> >>>> >>
> >>>>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >>>> >>     at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >>>> >>     at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>>> >>     at
> >>>> >>
> >>>> >>
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
> >>>> >>     at
> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>>> >>     at
> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>>> >>
> >>>> >> In every case it's the "main" thread which isn't surprising as
> that is
> >>>> the
> >>>> >> thread responsible for starting the broker.  Also, you can pretty
> >>>> clearly
> >>>> >> see in the trace that this is the thread starting the broker, it's
> >>>> loading
> >>>> >> the journals, & restoring stored configuration (either address
> settings
> >>>> or
> >>>> >> security settings).  I've seen high broker start times when there
> are
> >>>> lots
> >>>> >> and lots of addresses and lots of and lots of settings.  Do either
> (or
> >>>> >> both) of these situations apply to you?
> >>>> >>
> >>>> >>
> >>>> >> Justin
> >>>> >>
> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <
> danlangford@gmail.com>
> >>>> >> wrote:
> >>>> >>
> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
> >>>> >> config-delete-queues
> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was
> squashed
> >>>> in
> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work
> load) but
> >>>> we
> >>>> >> > discussed it back in April. I should go confirm that bug on 2.6
> and
> >>>> >> make a
> >>>> >> > jira for that
> >>>> >> >
> >>>> >> > Thanks
> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> >>>> >> clebert.suconic@gmail.com
> >>>> >> > >
> >>>> >> > wrote:
> >>>> >> >
> >>>> >> > > There is an issue I remember where the journal would have some
> dirt
> >>>> >> that
> >>>> >> > > was fixed on 2.3/0.
> >>>> >> > >
> >>>> >> > > I would ipgrade to 2.6.2.
> >>>> >> > >
> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <
> danlangford@gmail.com
> >>>> >
> >>>> >> > > wrote:
> >>>> >> > >
> >>>> >> > > > would you be willing to help me translate these thread dumps?
> >>>> >> > > >
> >>>> >> > > > i attached a Zip file with some thread dumps in them. i will
> also
> >>>> >> share
> >>>> >> > > > the fasthread.io links for each file. (i was struggling
> getting
> >>>> >> > > > fastthread to do a combo report with the threads in the
> correct
> >>>> >> order)
> >>>> >> > > >
> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >>>> >> > > >
> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger
> than at
> >>>> the
> >>>> >> > > other
> >>>> >> > > > sample times. but i dont know what to look for in here to
> >>>> determine
> >>>> >> > what
> >>>> >> > > > the broker is actually during during this time.
> >>>> >> > > >
> >>>> >> > > > thanks
> >>>> >> > > >
> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
> >>>> jbertram@apache.org
> >>>> >> >
> >>>> >> > > > wrote:
> >>>> >> > > >
> >>>> >> > > >> The first place I would start is grabbing thread dumps
> every so
> >>>> >> often
> >>>> >> > to
> >>>> >> > > >> see what the broker is actually doing during the 30+
> minutes.
> >>>> >> > > >>
> >>>> >> > > >>
> >>>> >> > > >> Justin
> >>>> >> > > >>
> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> >>>> >> danlangford@gmail.com>
> >>>> >> > > >> wrote:
> >>>> >> > > >>
> >>>> >> > > >> > so my server startup times and failover times are growing
> >>>> pretty
> >>>> >> > big.
> >>>> >> > > >> but i
> >>>> >> > > >> > dont really know where to start looking.
> >>>> >> > > >> >
> >>>> >> > > >> > here is a snippet of some logs to show you the time
> stamps:
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> >>>> >> > integration.bootstrap]
> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:31,818 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221000:
> >>>> >> > > >> > live Message Broker is starting with configuration Broker
> >>>> >> > > Configuration
> >>>> >> > > >> > (clustered=true,journalDirectory=./data/
> >>>> >> > journal,bindingsDirectory=./
> >>>> >> > > >> >
> data/bindings,largeMessagesDirectory=./data/large-messages,
> >>>> >> > > >> > pagingDirectory=./data/paging)
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,462 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221012:
> >>>> >> > > >> > Using AIO Journal
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,493 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221057:
> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max
> size
> >>>> >> (-Xmx).
> >>>> >> > > >> being
> >>>> >> > > >> > defined as 1,073,741,824
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,555 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
> >>>> support
> >>>> >> > for:
> >>>> >> > > >> CORE
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,555 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding
> protocol
> >>>> >> > > support
> >>>> >> > > >> > for: AMQP
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,556 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
> >>>> >> protocol
> >>>> >> > > >> support
> >>>> >> > > >> > for: HORNETQ
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,556 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding
> protocol
> >>>> >> > > support
> >>>> >> > > >> > for: MQTT
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,556 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
> >>>> >> protocol
> >>>> >> > > >> support
> >>>> >> > > >> > for: OPENWIRE
> >>>> >> > > >> >
> >>>> >> > > >> > 08:11:34,556 INFO
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ221043:
> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
> >>>> protocol
> >>>> >> > > support
> >>>> >> > > >> > for: STOMP
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,963 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222165:
> >>>> >> > > >> No
> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in
> AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,963 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222166:
> >>>> >> > > >> No
> >>>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,983 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222165:
> >>>> >> > > >> No
> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
> >>>> >> > > AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,983 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222166:
> >>>> >> > > >> No
> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
> >>>> >> AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,984 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222165:
> >>>> >> > > >> No
> >>>> >> > > >> > Dead Letter Address configured for queue example in
> >>>> >> AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,985 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222166:
> >>>> >> > > >> No
> >>>> >> > > >> > Expiry Address configured for queue example in
> AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,985 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222165:
> >>>> >> > > >> No
> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue in
> >>>> >> > > AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> > 08:41:38,986 WARN
> [org.apache.activemq.artemis.core.server]
> >>>> >> > > AMQ222166:
> >>>> >> > > >> No
> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in
> >>>> >> AddressSettings
> >>>> >> > > >> >
> >>>> >> > > >> >
> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need
> to be
> >>>> >> > looking
> >>>> >> > > at
> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be
> looking
> >>>> >> at
> >>>> >> > the
> >>>> >> > > >> > number of messages persisted on these queues? where would
> you
> >>>> >> look
> >>>> >> > to
> >>>> >> > > >> > determine why the startup times are so long?
> >>>> >> > > >> >
> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
> >>>> 2.1.0
> >>>> >> due
> >>>> >> > > to
> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i think
> i have
> >>>> >> > > another
> >>>> >> > > >> > thread on here (that i need to update) regarding that
> upgrade
> >>>> >> issue.
> >>>> >> > > >> >
> >>>> >> > > >> > thanks for any ideas or insight you have for me
> >>>> >> > > >> >
> >>>> >> > > >>
> >>>> >> > > > --
> >>>> >> > > Clebert Suconic
> >>>> >> > >
> >>>> >> >
> >>>> >>
> >>>> > --
> >>>> > Clebert Suconic
> >>>> >
> >>>> --
> >>>> Clebert Suconic
> >>>>
> >>
> >>
> >>
> >> --
> >> Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

@dan take a look at the compatibility tests in master.. We're help to
help if you don't understand anything.. it's using ClassLoaders and
Groovy so you can produce stuff with a combination of clients and
servers on 2.1.0 and consume on the current version.

On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic
<cl...@gmail.com> wrote:
> @Dan: what about this:
>
> you provide us a code path that could generate the incompatibility in 2.1.0...
>
> an ultimate deal would be if you produced a compatibility test with
> 2.1.0.... you add some code that will generate the journal with 2.1.0,
> and then consume the messages on master (or 2.6.x)...
>
>
> if you generate the compatibility test showing the issue, we can try fixing it.
>
>
> What about this idea? that would help you migrate into 2.6.x or master.
>
> On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
> <cl...@gmail.com> wrote:
>> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <da...@gmail.com> wrote:
>>> Thank you that was very helpful. we actually do have an address settings
>>> entry for each queue. there could be a better pattern for us. but currently
>>> our automated system for creating queues creates an address setting at the
>>> same time. i will look into improved patterns.
>>>
>>> as far as upgrading goes. i agree we really want to upgrade. until i can
>>> find a work around for the config-delete-queues deserialization bug
>>> introduced in 2.2.0 i brought up back in April we will not be able to
>>> easily move.
>>
>> You should be able to move to 2.6.2. if you're not able to I would
>> like to know where it failed.
>>
>>
>>>
>>> thanks again for all the help
>>>
>>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <cl...@gmail.com>
>>> wrote:
>>>
>>>> If you do not want to upgrade for any reason export the journal. Cleani
>>>> uo.  Edit the text and remove the garbage (you will see) manually.   Delete
>>>> all data and te import.
>>>>
>>>> (Make a backup to be safe of course)
>>>>
>>>>
>>>> But I still recommend the upgrade.
>>>>
>>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>>>> clebert.suconic@gmail.com>
>>>> wrote:
>>>>
>>>> > The address setting is the garbage I was talking about.  Upgrade to the
>>>> > latest broker and there will be a cleanup done at the load before it
>>>> > starts.
>>>> >
>>>> >
>>>> > I highly recommend upgrade.
>>>> >
>>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org>
>>>> > wrote:
>>>> >
>>>> >> Analyzing thread dumps like this is pretty simple.  I generally just
>>>> >> scroll
>>>> >> through and look for long stack-traces with lots of calls from
>>>> >> org.apache.activemq.artemis.  In your case every single thread dump has
>>>> a
>>>> >> thread doing something like this:
>>>> >>
>>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
>>>> >> [0x00007f9031675000]
>>>> >>    java.lang.Thread.State: RUNNABLE
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>>>> >>     - locked <0x0000000080a8fce8> (a
>>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>>>> >>     - locked <0x0000000080a8fce8> (a
>>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>>> >>     at
>>>> >>
>>>> >>
>>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>>> >>     - locked <0x000000008098be80> (a
>>>> >> org.apache.activemq.artemis.integration.FileBroker)
>>>> >>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>>>> >>     at
>>>> >>
>>>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>>>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>>>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> >>     at
>>>> >>
>>>> >>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> >>     at
>>>> >>
>>>> >>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>>>> >>     at
>>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>>> >>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>>> >>
>>>> >> In every case it's the "main" thread which isn't surprising as that is
>>>> the
>>>> >> thread responsible for starting the broker.  Also, you can pretty
>>>> clearly
>>>> >> see in the trace that this is the thread starting the broker, it's
>>>> loading
>>>> >> the journals, & restoring stored configuration (either address settings
>>>> or
>>>> >> security settings).  I've seen high broker start times when there are
>>>> lots
>>>> >> and lots of addresses and lots of and lots of settings.  Do either (or
>>>> >> both) of these situations apply to you?
>>>> >>
>>>> >>
>>>> >> Justin
>>>> >>
>>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>>>> >> config-delete-queues
>>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed
>>>> in
>>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but
>>>> we
>>>> >> > discussed it back in April. I should go confirm that bug on 2.6 and
>>>> >> make a
>>>> >> > jira for that
>>>> >> >
>>>> >> > Thanks
>>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>>>> >> clebert.suconic@gmail.com
>>>> >> > >
>>>> >> > wrote:
>>>> >> >
>>>> >> > > There is an issue I remember where the journal would have some dirt
>>>> >> that
>>>> >> > > was fixed on 2.3/0.
>>>> >> > >
>>>> >> > > I would ipgrade to 2.6.2.
>>>> >> > >
>>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangford@gmail.com
>>>> >
>>>> >> > > wrote:
>>>> >> > >
>>>> >> > > > would you be willing to help me translate these thread dumps?
>>>> >> > > >
>>>> >> > > > i attached a Zip file with some thread dumps in them. i will also
>>>> >> share
>>>> >> > > > the fasthread.io links for each file. (i was struggling getting
>>>> >> > > > fastthread to do a combo report with the threads in the correct
>>>> >> order)
>>>> >> > > >
>>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>>>> >> > > >
>>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger than at
>>>> the
>>>> >> > > other
>>>> >> > > > sample times. but i dont know what to look for in here to
>>>> determine
>>>> >> > what
>>>> >> > > > the broker is actually during during this time.
>>>> >> > > >
>>>> >> > > > thanks
>>>> >> > > >
>>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>>>> jbertram@apache.org
>>>> >> >
>>>> >> > > > wrote:
>>>> >> > > >
>>>> >> > > >> The first place I would start is grabbing thread dumps every so
>>>> >> often
>>>> >> > to
>>>> >> > > >> see what the broker is actually doing during the 30+ minutes.
>>>> >> > > >>
>>>> >> > > >>
>>>> >> > > >> Justin
>>>> >> > > >>
>>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>>>> >> danlangford@gmail.com>
>>>> >> > > >> wrote:
>>>> >> > > >>
>>>> >> > > >> > so my server startup times and failover times are growing
>>>> pretty
>>>> >> > big.
>>>> >> > > >> but i
>>>> >> > > >> > dont really know where to start looking.
>>>> >> > > >> >
>>>> >> > > >> > here is a snippet of some logs to show you the time stamps:
>>>> >> > > >> >
>>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>>>> >> > integration.bootstrap]
>>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>>>> >> > > >> >
>>>> >> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221000:
>>>> >> > > >> > live Message Broker is starting with configuration Broker
>>>> >> > > Configuration
>>>> >> > > >> > (clustered=true,journalDirectory=./data/
>>>> >> > journal,bindingsDirectory=./
>>>> >> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
>>>> >> > > >> > pagingDirectory=./data/paging)
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221012:
>>>> >> > > >> > Using AIO Journal
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221057:
>>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
>>>> >> (-Xmx).
>>>> >> > > >> being
>>>> >> > > >> > defined as 1,073,741,824
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
>>>> support
>>>> >> > for:
>>>> >> > > >> CORE
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
>>>> >> > > support
>>>> >> > > >> > for: AMQP
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
>>>> >> protocol
>>>> >> > > >> support
>>>> >> > > >> > for: HORNETQ
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
>>>> >> > > support
>>>> >> > > >> > for: MQTT
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
>>>> >> protocol
>>>> >> > > >> support
>>>> >> > > >> > for: OPENWIRE
>>>> >> > > >> >
>>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ221043:
>>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>>>> protocol
>>>> >> > > support
>>>> >> > > >> > for: STOMP
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222165:
>>>> >> > > >> No
>>>> >> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222166:
>>>> >> > > >> No
>>>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222165:
>>>> >> > > >> No
>>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>>>> >> > > AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222166:
>>>> >> > > >> No
>>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>>>> >> AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222165:
>>>> >> > > >> No
>>>> >> > > >> > Dead Letter Address configured for queue example in
>>>> >> AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222166:
>>>> >> > > >> No
>>>> >> > > >> > Expiry Address configured for queue example in AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222165:
>>>> >> > > >> No
>>>> >> > > >> > Dead Letter Address configured for queue exampleQueue in
>>>> >> > > AddressSettings
>>>> >> > > >> >
>>>> >> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
>>>> >> > > AMQ222166:
>>>> >> > > >> No
>>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>>>> >> AddressSettings
>>>> >> > > >> >
>>>> >> > > >> >
>>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need to be
>>>> >> > looking
>>>> >> > > at
>>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be looking
>>>> >> at
>>>> >> > the
>>>> >> > > >> > number of messages persisted on these queues? where would you
>>>> >> look
>>>> >> > to
>>>> >> > > >> > determine why the startup times are so long?
>>>> >> > > >> >
>>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
>>>> 2.1.0
>>>> >> due
>>>> >> > > to
>>>> >> > > >> > some deseralization changes introduced in 2.2.0. i think i have
>>>> >> > > another
>>>> >> > > >> > thread on here (that i need to update) regarding that upgrade
>>>> >> issue.
>>>> >> > > >> >
>>>> >> > > >> > thanks for any ideas or insight you have for me
>>>> >> > > >> >
>>>> >> > > >>
>>>> >> > > > --
>>>> >> > > Clebert Suconic
>>>> >> > >
>>>> >> >
>>>> >>
>>>> > --
>>>> > Clebert Suconic
>>>> >
>>>> --
>>>> Clebert Suconic
>>>>
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

@Dan: what about this:

you provide us a code path that could generate the incompatibility in 2.1.0...

an ultimate deal would be if you produced a compatibility test with
2.1.0.... you add some code that will generate the journal with 2.1.0,
and then consume the messages on master (or 2.6.x)...


if you generate the compatibility test showing the issue, we can try fixing it.


What about this idea? that would help you migrate into 2.6.x or master.

On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic
<cl...@gmail.com> wrote:
> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <da...@gmail.com> wrote:
>> Thank you that was very helpful. we actually do have an address settings
>> entry for each queue. there could be a better pattern for us. but currently
>> our automated system for creating queues creates an address setting at the
>> same time. i will look into improved patterns.
>>
>> as far as upgrading goes. i agree we really want to upgrade. until i can
>> find a work around for the config-delete-queues deserialization bug
>> introduced in 2.2.0 i brought up back in April we will not be able to
>> easily move.
>
> You should be able to move to 2.6.2. if you're not able to I would
> like to know where it failed.
>
>
>>
>> thanks again for all the help
>>
>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <cl...@gmail.com>
>> wrote:
>>
>>> If you do not want to upgrade for any reason export the journal. Cleani
>>> uo.  Edit the text and remove the garbage (you will see) manually.   Delete
>>> all data and te import.
>>>
>>> (Make a backup to be safe of course)
>>>
>>>
>>> But I still recommend the upgrade.
>>>
>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>>> clebert.suconic@gmail.com>
>>> wrote:
>>>
>>> > The address setting is the garbage I was talking about.  Upgrade to the
>>> > latest broker and there will be a cleanup done at the load before it
>>> > starts.
>>> >
>>> >
>>> > I highly recommend upgrade.
>>> >
>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org>
>>> > wrote:
>>> >
>>> >> Analyzing thread dumps like this is pretty simple.  I generally just
>>> >> scroll
>>> >> through and look for long stack-traces with lots of calls from
>>> >> org.apache.activemq.artemis.  In your case every single thread dump has
>>> a
>>> >> thread doing something like this:
>>> >>
>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
>>> >> [0x00007f9031675000]
>>> >>    java.lang.Thread.State: RUNNABLE
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>>> >>     - locked <0x0000000080a8fce8> (a
>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>>> >>     - locked <0x0000000080a8fce8> (a
>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>> >>     at
>>> >>
>>> >>
>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>> >>     - locked <0x000000008098be80> (a
>>> >> org.apache.activemq.artemis.integration.FileBroker)
>>> >>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>>> >>     at
>>> >>
>>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> >>     at
>>> >>
>>> >>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> >>     at
>>> >>
>>> >>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>>> >>     at
>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>> >>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>> >>
>>> >> In every case it's the "main" thread which isn't surprising as that is
>>> the
>>> >> thread responsible for starting the broker.  Also, you can pretty
>>> clearly
>>> >> see in the trace that this is the thread starting the broker, it's
>>> loading
>>> >> the journals, & restoring stored configuration (either address settings
>>> or
>>> >> security settings).  I've seen high broker start times when there are
>>> lots
>>> >> and lots of addresses and lots of and lots of settings.  Do either (or
>>> >> both) of these situations apply to you?
>>> >>
>>> >>
>>> >> Justin
>>> >>
>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
>>> >> wrote:
>>> >>
>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>>> >> config-delete-queues
>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed
>>> in
>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but
>>> we
>>> >> > discussed it back in April. I should go confirm that bug on 2.6 and
>>> >> make a
>>> >> > jira for that
>>> >> >
>>> >> > Thanks
>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>>> >> clebert.suconic@gmail.com
>>> >> > >
>>> >> > wrote:
>>> >> >
>>> >> > > There is an issue I remember where the journal would have some dirt
>>> >> that
>>> >> > > was fixed on 2.3/0.
>>> >> > >
>>> >> > > I would ipgrade to 2.6.2.
>>> >> > >
>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangford@gmail.com
>>> >
>>> >> > > wrote:
>>> >> > >
>>> >> > > > would you be willing to help me translate these thread dumps?
>>> >> > > >
>>> >> > > > i attached a Zip file with some thread dumps in them. i will also
>>> >> share
>>> >> > > > the fasthread.io links for each file. (i was struggling getting
>>> >> > > > fastthread to do a combo report with the threads in the correct
>>> >> order)
>>> >> > > >
>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>>> >> > > >
>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger than at
>>> the
>>> >> > > other
>>> >> > > > sample times. but i dont know what to look for in here to
>>> determine
>>> >> > what
>>> >> > > > the broker is actually during during this time.
>>> >> > > >
>>> >> > > > thanks
>>> >> > > >
>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>>> jbertram@apache.org
>>> >> >
>>> >> > > > wrote:
>>> >> > > >
>>> >> > > >> The first place I would start is grabbing thread dumps every so
>>> >> often
>>> >> > to
>>> >> > > >> see what the broker is actually doing during the 30+ minutes.
>>> >> > > >>
>>> >> > > >>
>>> >> > > >> Justin
>>> >> > > >>
>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>>> >> danlangford@gmail.com>
>>> >> > > >> wrote:
>>> >> > > >>
>>> >> > > >> > so my server startup times and failover times are growing
>>> pretty
>>> >> > big.
>>> >> > > >> but i
>>> >> > > >> > dont really know where to start looking.
>>> >> > > >> >
>>> >> > > >> > here is a snippet of some logs to show you the time stamps:
>>> >> > > >> >
>>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>>> >> > integration.bootstrap]
>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>>> >> > > >> >
>>> >> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221000:
>>> >> > > >> > live Message Broker is starting with configuration Broker
>>> >> > > Configuration
>>> >> > > >> > (clustered=true,journalDirectory=./data/
>>> >> > journal,bindingsDirectory=./
>>> >> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
>>> >> > > >> > pagingDirectory=./data/paging)
>>> >> > > >> >
>>> >> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221012:
>>> >> > > >> > Using AIO Journal
>>> >> > > >> >
>>> >> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221057:
>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
>>> >> (-Xmx).
>>> >> > > >> being
>>> >> > > >> > defined as 1,073,741,824
>>> >> > > >> >
>>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
>>> support
>>> >> > for:
>>> >> > > >> CORE
>>> >> > > >> >
>>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
>>> >> > > support
>>> >> > > >> > for: AMQP
>>> >> > > >> >
>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
>>> >> protocol
>>> >> > > >> support
>>> >> > > >> > for: HORNETQ
>>> >> > > >> >
>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
>>> >> > > support
>>> >> > > >> > for: MQTT
>>> >> > > >> >
>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
>>> >> protocol
>>> >> > > >> support
>>> >> > > >> > for: OPENWIRE
>>> >> > > >> >
>>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ221043:
>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>>> protocol
>>> >> > > support
>>> >> > > >> > for: STOMP
>>> >> > > >> >
>>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222165:
>>> >> > > >> No
>>> >> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222166:
>>> >> > > >> No
>>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222165:
>>> >> > > >> No
>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>>> >> > > AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222166:
>>> >> > > >> No
>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>>> >> AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222165:
>>> >> > > >> No
>>> >> > > >> > Dead Letter Address configured for queue example in
>>> >> AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222166:
>>> >> > > >> No
>>> >> > > >> > Expiry Address configured for queue example in AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222165:
>>> >> > > >> No
>>> >> > > >> > Dead Letter Address configured for queue exampleQueue in
>>> >> > > AddressSettings
>>> >> > > >> >
>>> >> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
>>> >> > > AMQ222166:
>>> >> > > >> No
>>> >> > > >> > Expiry Address configured for queue exampleQueue in
>>> >> AddressSettings
>>> >> > > >> >
>>> >> > > >> >
>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need to be
>>> >> > looking
>>> >> > > at
>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be looking
>>> >> at
>>> >> > the
>>> >> > > >> > number of messages persisted on these queues? where would you
>>> >> look
>>> >> > to
>>> >> > > >> > determine why the startup times are so long?
>>> >> > > >> >
>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
>>> 2.1.0
>>> >> due
>>> >> > > to
>>> >> > > >> > some deseralization changes introduced in 2.2.0. i think i have
>>> >> > > another
>>> >> > > >> > thread on here (that i need to update) regarding that upgrade
>>> >> issue.
>>> >> > > >> >
>>> >> > > >> > thanks for any ideas or insight you have for me
>>> >> > > >> >
>>> >> > > >>
>>> >> > > > --
>>> >> > > Clebert Suconic
>>> >> > >
>>> >> >
>>> >>
>>> > --
>>> > Clebert Suconic
>>> >
>>> --
>>> Clebert Suconic
>>>
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <da...@gmail.com> wrote:
> Thank you that was very helpful. we actually do have an address settings
> entry for each queue. there could be a better pattern for us. but currently
> our automated system for creating queues creates an address setting at the
> same time. i will look into improved patterns.
>
> as far as upgrading goes. i agree we really want to upgrade. until i can
> find a work around for the config-delete-queues deserialization bug
> introduced in 2.2.0 i brought up back in April we will not be able to
> easily move.

You should be able to move to 2.6.2. if you're not able to I would
like to know where it failed.


>
> thanks again for all the help
>
> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <cl...@gmail.com>
> wrote:
>
>> If you do not want to upgrade for any reason export the journal. Cleani
>> uo.  Edit the text and remove the garbage (you will see) manually.   Delete
>> all data and te import.
>>
>> (Make a backup to be safe of course)
>>
>>
>> But I still recommend the upgrade.
>>
>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
>> clebert.suconic@gmail.com>
>> wrote:
>>
>> > The address setting is the garbage I was talking about.  Upgrade to the
>> > latest broker and there will be a cleanup done at the load before it
>> > starts.
>> >
>> >
>> > I highly recommend upgrade.
>> >
>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org>
>> > wrote:
>> >
>> >> Analyzing thread dumps like this is pretty simple.  I generally just
>> >> scroll
>> >> through and look for long stack-traces with lots of calls from
>> >> org.apache.activemq.artemis.  In your case every single thread dump has
>> a
>> >> thread doing something like this:
>> >>
>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
>> >> [0x00007f9031675000]
>> >>    java.lang.Thread.State: RUNNABLE
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>> >>     - locked <0x0000000080a8fce8> (a
>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>> >>     - locked <0x0000000080a8fce8> (a
>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>> >>     at
>> >>
>> >>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>> >>     - locked <0x000000008098be80> (a
>> >> org.apache.activemq.artemis.integration.FileBroker)
>> >>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>> >>     at
>> >>
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>     at
>> >>
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> >>     at
>> >>
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >>     at java.lang.reflect.Method.invoke(Method.java:498)
>> >>     at
>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>> >>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>> >>
>> >> In every case it's the "main" thread which isn't surprising as that is
>> the
>> >> thread responsible for starting the broker.  Also, you can pretty
>> clearly
>> >> see in the trace that this is the thread starting the broker, it's
>> loading
>> >> the journals, & restoring stored configuration (either address settings
>> or
>> >> security settings).  I've seen high broker start times when there are
>> lots
>> >> and lots of addresses and lots of and lots of settings.  Do either (or
>> >> both) of these situations apply to you?
>> >>
>> >>
>> >> Justin
>> >>
>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
>> >> wrote:
>> >>
>> >> > Dang I can’t easily upgrade past 2.1.0 because of the
>> >> config-delete-queues
>> >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed
>> in
>> >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but
>> we
>> >> > discussed it back in April. I should go confirm that bug on 2.6 and
>> >> make a
>> >> > jira for that
>> >> >
>> >> > Thanks
>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>> >> clebert.suconic@gmail.com
>> >> > >
>> >> > wrote:
>> >> >
>> >> > > There is an issue I remember where the journal would have some dirt
>> >> that
>> >> > > was fixed on 2.3/0.
>> >> > >
>> >> > > I would ipgrade to 2.6.2.
>> >> > >
>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangford@gmail.com
>> >
>> >> > > wrote:
>> >> > >
>> >> > > > would you be willing to help me translate these thread dumps?
>> >> > > >
>> >> > > > i attached a Zip file with some thread dumps in them. i will also
>> >> share
>> >> > > > the fasthread.io links for each file. (i was struggling getting
>> >> > > > fastthread to do a combo report with the threads in the correct
>> >> order)
>> >> > > >
>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>> >> > > >
>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger than at
>> the
>> >> > > other
>> >> > > > sample times. but i dont know what to look for in here to
>> determine
>> >> > what
>> >> > > > the broker is actually during during this time.
>> >> > > >
>> >> > > > thanks
>> >> > > >
>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
>> jbertram@apache.org
>> >> >
>> >> > > > wrote:
>> >> > > >
>> >> > > >> The first place I would start is grabbing thread dumps every so
>> >> often
>> >> > to
>> >> > > >> see what the broker is actually doing during the 30+ minutes.
>> >> > > >>
>> >> > > >>
>> >> > > >> Justin
>> >> > > >>
>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>> >> danlangford@gmail.com>
>> >> > > >> wrote:
>> >> > > >>
>> >> > > >> > so my server startup times and failover times are growing
>> pretty
>> >> > big.
>> >> > > >> but i
>> >> > > >> > dont really know where to start looking.
>> >> > > >> >
>> >> > > >> > here is a snippet of some logs to show you the time stamps:
>> >> > > >> >
>> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>> >> > integration.bootstrap]
>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>> >> > > >> >
>> >> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221000:
>> >> > > >> > live Message Broker is starting with configuration Broker
>> >> > > Configuration
>> >> > > >> > (clustered=true,journalDirectory=./data/
>> >> > journal,bindingsDirectory=./
>> >> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
>> >> > > >> > pagingDirectory=./data/paging)
>> >> > > >> >
>> >> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221012:
>> >> > > >> > Using AIO Journal
>> >> > > >> >
>> >> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221057:
>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
>> >> (-Xmx).
>> >> > > >> being
>> >> > > >> > defined as 1,073,741,824
>> >> > > >> >
>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
>> support
>> >> > for:
>> >> > > >> CORE
>> >> > > >> >
>> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
>> >> > > support
>> >> > > >> > for: AMQP
>> >> > > >> >
>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
>> >> protocol
>> >> > > >> support
>> >> > > >> > for: HORNETQ
>> >> > > >> >
>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
>> >> > > support
>> >> > > >> > for: MQTT
>> >> > > >> >
>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
>> >> protocol
>> >> > > >> support
>> >> > > >> > for: OPENWIRE
>> >> > > >> >
>> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ221043:
>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
>> protocol
>> >> > > support
>> >> > > >> > for: STOMP
>> >> > > >> >
>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222165:
>> >> > > >> No
>> >> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222166:
>> >> > > >> No
>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222165:
>> >> > > >> No
>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>> >> > > AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222166:
>> >> > > >> No
>> >> > > >> > Expiry Address configured for queue ExpiryQueue in
>> >> AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222165:
>> >> > > >> No
>> >> > > >> > Dead Letter Address configured for queue example in
>> >> AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222166:
>> >> > > >> No
>> >> > > >> > Expiry Address configured for queue example in AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222165:
>> >> > > >> No
>> >> > > >> > Dead Letter Address configured for queue exampleQueue in
>> >> > > AddressSettings
>> >> > > >> >
>> >> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
>> >> > > AMQ222166:
>> >> > > >> No
>> >> > > >> > Expiry Address configured for queue exampleQueue in
>> >> AddressSettings
>> >> > > >> >
>> >> > > >> >
>> >> > > >> > and it continues. i have 138 queues. i wonder if i need to be
>> >> > looking
>> >> > > at
>> >> > > >> > PAGE configuration or some cache sizes. do i need to be looking
>> >> at
>> >> > the
>> >> > > >> > number of messages persisted on these queues? where would you
>> >> look
>> >> > to
>> >> > > >> > determine why the startup times are so long?
>> >> > > >> >
>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
>> 2.1.0
>> >> due
>> >> > > to
>> >> > > >> > some deseralization changes introduced in 2.2.0. i think i have
>> >> > > another
>> >> > > >> > thread on here (that i need to update) regarding that upgrade
>> >> issue.
>> >> > > >> >
>> >> > > >> > thanks for any ideas or insight you have for me
>> >> > > >> >
>> >> > > >>
>> >> > > > --
>> >> > > Clebert Suconic
>> >> > >
>> >> >
>> >>
>> > --
>> > Clebert Suconic
>> >
>> --
>> Clebert Suconic
>>



-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

Thank you that was very helpful. we actually do have an address settings
entry for each queue. there could be a better pattern for us. but currently
our automated system for creating queues creates an address setting at the
same time. i will look into improved patterns.

as far as upgrading goes. i agree we really want to upgrade. until i can
find a work around for the config-delete-queues deserialization bug
introduced in 2.2.0 i brought up back in April we will not be able to
easily move.

thanks again for all the help

On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <cl...@gmail.com>
wrote:

> If you do not want to upgrade for any reason export the journal. Cleani
> uo.  Edit the text and remove the garbage (you will see) manually.   Delete
> all data and te import.
>
> (Make a backup to be safe of course)
>
>
> But I still recommend the upgrade.
>
> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <
> clebert.suconic@gmail.com>
> wrote:
>
> > The address setting is the garbage I was talking about.  Upgrade to the
> > latest broker and there will be a cleanup done at the load before it
> > starts.
> >
> >
> > I highly recommend upgrade.
> >
> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org>
> > wrote:
> >
> >> Analyzing thread dumps like this is pretty simple.  I generally just
> >> scroll
> >> through and look for long stack-traces with lots of calls from
> >> org.apache.activemq.artemis.  In your case every single thread dump has
> a
> >> thread doing something like this:
> >>
> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
> >> [0x00007f9031675000]
> >>    java.lang.Thread.State: RUNNABLE
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
> >>     - locked <0x0000000080a8fce8> (a
> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
> >>     - locked <0x0000000080a8fce8> (a
> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
> >>     at
> >>
> >>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
> >>     - locked <0x000000008098be80> (a
> >> org.apache.activemq.artemis.integration.FileBroker)
> >>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
> >>     at
> >>
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
> >>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
> >>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>     at
> >>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >>     at
> >>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>     at java.lang.reflect.Method.invoke(Method.java:498)
> >>     at
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
> >>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
> >>
> >> In every case it's the "main" thread which isn't surprising as that is
> the
> >> thread responsible for starting the broker.  Also, you can pretty
> clearly
> >> see in the trace that this is the thread starting the broker, it's
> loading
> >> the journals, & restoring stored configuration (either address settings
> or
> >> security settings).  I've seen high broker start times when there are
> lots
> >> and lots of addresses and lots of and lots of settings.  Do either (or
> >> both) of these situations apply to you?
> >>
> >>
> >> Justin
> >>
> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
> >> wrote:
> >>
> >> > Dang I can’t easily upgrade past 2.1.0 because of the
> >> config-delete-queues
> >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed
> in
> >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but
> we
> >> > discussed it back in April. I should go confirm that bug on 2.6 and
> >> make a
> >> > jira for that
> >> >
> >> > Thanks
> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> >> clebert.suconic@gmail.com
> >> > >
> >> > wrote:
> >> >
> >> > > There is an issue I remember where the journal would have some dirt
> >> that
> >> > > was fixed on 2.3/0.
> >> > >
> >> > > I would ipgrade to 2.6.2.
> >> > >
> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangford@gmail.com
> >
> >> > > wrote:
> >> > >
> >> > > > would you be willing to help me translate these thread dumps?
> >> > > >
> >> > > > i attached a Zip file with some thread dumps in them. i will also
> >> share
> >> > > > the fasthread.io links for each file. (i was struggling getting
> >> > > > fastthread to do a combo report with the threads in the correct
> >> order)
> >> > > >
> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >> > > >
> >> > > > at :35, :45, and :55 the young+old gen space gets bigger than at
> the
> >> > > other
> >> > > > sample times. but i dont know what to look for in here to
> determine
> >> > what
> >> > > > the broker is actually during during this time.
> >> > > >
> >> > > > thanks
> >> > > >
> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <
> jbertram@apache.org
> >> >
> >> > > > wrote:
> >> > > >
> >> > > >> The first place I would start is grabbing thread dumps every so
> >> often
> >> > to
> >> > > >> see what the broker is actually doing during the 30+ minutes.
> >> > > >>
> >> > > >>
> >> > > >> Justin
> >> > > >>
> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> >> danlangford@gmail.com>
> >> > > >> wrote:
> >> > > >>
> >> > > >> > so my server startup times and failover times are growing
> pretty
> >> > big.
> >> > > >> but i
> >> > > >> > dont really know where to start looking.
> >> > > >> >
> >> > > >> > here is a snippet of some logs to show you the time stamps:
> >> > > >> >
> >> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> >> > integration.bootstrap]
> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> >> > > >> >
> >> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221000:
> >> > > >> > live Message Broker is starting with configuration Broker
> >> > > Configuration
> >> > > >> > (clustered=true,journalDirectory=./data/
> >> > journal,bindingsDirectory=./
> >> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
> >> > > >> > pagingDirectory=./data/paging)
> >> > > >> >
> >> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221012:
> >> > > >> > Using AIO Journal
> >> > > >> >
> >> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221057:
> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
> >> (-Xmx).
> >> > > >> being
> >> > > >> > defined as 1,073,741,824
> >> > > >> >
> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-server]. Adding protocol
> support
> >> > for:
> >> > > >> CORE
> >> > > >> >
> >> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
> >> > > support
> >> > > >> > for: AMQP
> >> > > >> >
> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
> >> protocol
> >> > > >> support
> >> > > >> > for: HORNETQ
> >> > > >> >
> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
> >> > > support
> >> > > >> > for: MQTT
> >> > > >> >
> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
> >> protocol
> >> > > >> support
> >> > > >> > for: OPENWIRE
> >> > > >> >
> >> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> >> > > AMQ221043:
> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding
> protocol
> >> > > support
> >> > > >> > for: STOMP
> >> > > >> >
> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222165:
> >> > > >> No
> >> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222166:
> >> > > >> No
> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222165:
> >> > > >> No
> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in
> >> > > AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222166:
> >> > > >> No
> >> > > >> > Expiry Address configured for queue ExpiryQueue in
> >> AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222165:
> >> > > >> No
> >> > > >> > Dead Letter Address configured for queue example in
> >> AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222166:
> >> > > >> No
> >> > > >> > Expiry Address configured for queue example in AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222165:
> >> > > >> No
> >> > > >> > Dead Letter Address configured for queue exampleQueue in
> >> > > AddressSettings
> >> > > >> >
> >> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
> >> > > AMQ222166:
> >> > > >> No
> >> > > >> > Expiry Address configured for queue exampleQueue in
> >> AddressSettings
> >> > > >> >
> >> > > >> >
> >> > > >> > and it continues. i have 138 queues. i wonder if i need to be
> >> > looking
> >> > > at
> >> > > >> > PAGE configuration or some cache sizes. do i need to be looking
> >> at
> >> > the
> >> > > >> > number of messages persisted on these queues? where would you
> >> look
> >> > to
> >> > > >> > determine why the startup times are so long?
> >> > > >> >
> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of
> 2.1.0
> >> due
> >> > > to
> >> > > >> > some deseralization changes introduced in 2.2.0. i think i have
> >> > > another
> >> > > >> > thread on here (that i need to update) regarding that upgrade
> >> issue.
> >> > > >> >
> >> > > >> > thanks for any ideas or insight you have for me
> >> > > >> >
> >> > > >>
> >> > > > --
> >> > > Clebert Suconic
> >> > >
> >> >
> >>
> > --
> > Clebert Suconic
> >
> --
> Clebert Suconic
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

If you do not want to upgrade for any reason export the journal. Cleani
uo.  Edit the text and remove the garbage (you will see) manually.   Delete
all data and te import.

(Make a backup to be safe of course)


But I still recommend the upgrade.

On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic <cl...@gmail.com>
wrote:

> The address setting is the garbage I was talking about.  Upgrade to the
> latest broker and there will be a cleanup done at the load before it
> starts.
>
>
> I highly recommend upgrade.
>
> On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org>
> wrote:
>
>> Analyzing thread dumps like this is pretty simple.  I generally just
>> scroll
>> through and look for long stack-traces with lots of calls from
>> org.apache.activemq.artemis.  In your case every single thread dump has a
>> thread doing something like this:
>>
>> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
>> [0x00007f9031675000]
>>    java.lang.Thread.State: RUNNABLE
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>>     at
>>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>>     at
>>
>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>>     at
>>
>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>>     - locked <0x0000000080a8fce8> (a
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>>     at
>>
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>>     - locked <0x0000000080a8fce8> (a
>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>>     at
>>
>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>>     - locked <0x000000008098be80> (a
>> org.apache.activemq.artemis.integration.FileBroker)
>>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>>     at
>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>     at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>     at java.lang.reflect.Method.invoke(Method.java:498)
>>     at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>>
>> In every case it's the "main" thread which isn't surprising as that is the
>> thread responsible for starting the broker.  Also, you can pretty clearly
>> see in the trace that this is the thread starting the broker, it's loading
>> the journals, & restoring stored configuration (either address settings or
>> security settings).  I've seen high broker start times when there are lots
>> and lots of addresses and lots of and lots of settings.  Do either (or
>> both) of these situations apply to you?
>>
>>
>> Justin
>>
>> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
>> wrote:
>>
>> > Dang I can’t easily upgrade past 2.1.0 because of the
>> config-delete-queues
>> > deserialization bug introduced in 2.2.0. Unless that bug was squashed in
>> > 2.6+. I don’t think I made a jira for it (vacation and work load) but we
>> > discussed it back in April. I should go confirm that bug on 2.6 and
>> make a
>> > jira for that
>> >
>> > Thanks
>> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
>> clebert.suconic@gmail.com
>> > >
>> > wrote:
>> >
>> > > There is an issue I remember where the journal would have some dirt
>> that
>> > > was fixed on 2.3/0.
>> > >
>> > > I would ipgrade to 2.6.2.
>> > >
>> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <da...@gmail.com>
>> > > wrote:
>> > >
>> > > > would you be willing to help me translate these thread dumps?
>> > > >
>> > > > i attached a Zip file with some thread dumps in them. i will also
>> share
>> > > > the fasthread.io links for each file. (i was struggling getting
>> > > > fastthread to do a combo report with the threads in the correct
>> order)
>> > > >
>> > > > artemis04-20180719-1525 https://goo.gl/d88azU
>> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
>> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
>> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
>> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
>> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
>> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
>> > > >
>> > > > at :35, :45, and :55 the young+old gen space gets bigger than at the
>> > > other
>> > > > sample times. but i dont know what to look for in here to determine
>> > what
>> > > > the broker is actually during during this time.
>> > > >
>> > > > thanks
>> > > >
>> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jbertram@apache.org
>> >
>> > > > wrote:
>> > > >
>> > > >> The first place I would start is grabbing thread dumps every so
>> often
>> > to
>> > > >> see what the broker is actually doing during the 30+ minutes.
>> > > >>
>> > > >>
>> > > >> Justin
>> > > >>
>> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
>> danlangford@gmail.com>
>> > > >> wrote:
>> > > >>
>> > > >> > so my server startup times and failover times are growing pretty
>> > big.
>> > > >> but i
>> > > >> > dont really know where to start looking.
>> > > >> >
>> > > >> > here is a snippet of some logs to show you the time stamps:
>> > > >> >
>> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
>> > integration.bootstrap]
>> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
>> > > >> >
>> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221000:
>> > > >> > live Message Broker is starting with configuration Broker
>> > > Configuration
>> > > >> > (clustered=true,journalDirectory=./data/
>> > journal,bindingsDirectory=./
>> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
>> > > >> > pagingDirectory=./data/paging)
>> > > >> >
>> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221012:
>> > > >> > Using AIO Journal
>> > > >> >
>> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221057:
>> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
>> (-Xmx).
>> > > >> being
>> > > >> > defined as 1,073,741,824
>> > > >> >
>> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-server]. Adding protocol support
>> > for:
>> > > >> CORE
>> > > >> >
>> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
>> > > support
>> > > >> > for: AMQP
>> > > >> >
>> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding
>> protocol
>> > > >> support
>> > > >> > for: HORNETQ
>> > > >> >
>> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
>> > > support
>> > > >> > for: MQTT
>> > > >> >
>> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
>> protocol
>> > > >> support
>> > > >> > for: OPENWIRE
>> > > >> >
>> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
>> > > AMQ221043:
>> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol
>> > > support
>> > > >> > for: STOMP
>> > > >> >
>> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222165:
>> > > >> No
>> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
>> > > >> >
>> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222166:
>> > > >> No
>> > > >> > Expiry Address configured for queue DLQ in AddressSettings
>> > > >> >
>> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222165:
>> > > >> No
>> > > >> > Dead Letter Address configured for queue ExpiryQueue in
>> > > AddressSettings
>> > > >> >
>> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222166:
>> > > >> No
>> > > >> > Expiry Address configured for queue ExpiryQueue in
>> AddressSettings
>> > > >> >
>> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222165:
>> > > >> No
>> > > >> > Dead Letter Address configured for queue example in
>> AddressSettings
>> > > >> >
>> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222166:
>> > > >> No
>> > > >> > Expiry Address configured for queue example in AddressSettings
>> > > >> >
>> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222165:
>> > > >> No
>> > > >> > Dead Letter Address configured for queue exampleQueue in
>> > > AddressSettings
>> > > >> >
>> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
>> > > AMQ222166:
>> > > >> No
>> > > >> > Expiry Address configured for queue exampleQueue in
>> AddressSettings
>> > > >> >
>> > > >> >
>> > > >> > and it continues. i have 138 queues. i wonder if i need to be
>> > looking
>> > > at
>> > > >> > PAGE configuration or some cache sizes. do i need to be looking
>> at
>> > the
>> > > >> > number of messages persisted on these queues? where would you
>> look
>> > to
>> > > >> > determine why the startup times are so long?
>> > > >> >
>> > > >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0
>> due
>> > > to
>> > > >> > some deseralization changes introduced in 2.2.0. i think i have
>> > > another
>> > > >> > thread on here (that i need to update) regarding that upgrade
>> issue.
>> > > >> >
>> > > >> > thanks for any ideas or insight you have for me
>> > > >> >
>> > > >>
>> > > > --
>> > > Clebert Suconic
>> > >
>> >
>>
> --
> Clebert Suconic
>
-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

The address setting is the garbage I was talking about.  Upgrade to the
latest broker and there will be a cleanup done at the load before it
starts.


I highly recommend upgrade.

On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jb...@apache.org> wrote:

> Analyzing thread dumps like this is pretty simple.  I generally just scroll
> through and look for long stack-traces with lots of calls from
> org.apache.activemq.artemis.  In your case every single thread dump has a
> thread doing something like this:
>
> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
> [0x00007f9031675000]
>    java.lang.Thread.State: RUNNABLE
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
>     at
>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
>     at
>
> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
>     at
>
> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
>     at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
>     at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
>     at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
>     - locked <0x0000000080a8fce8> (a
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>     at
>
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
>     at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
>     at
>
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
>     - locked <0x0000000080a8fce8> (a
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
>     at
>
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
>     - locked <0x000000008098be80> (a
> org.apache.activemq.artemis.integration.FileBroker)
>     at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
>     at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
>     at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
>     at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)
>
> In every case it's the "main" thread which isn't surprising as that is the
> thread responsible for starting the broker.  Also, you can pretty clearly
> see in the trace that this is the thread starting the broker, it's loading
> the journals, & restoring stored configuration (either address settings or
> security settings).  I've seen high broker start times when there are lots
> and lots of addresses and lots of and lots of settings.  Do either (or
> both) of these situations apply to you?
>
>
> Justin
>
> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com>
> wrote:
>
> > Dang I can’t easily upgrade past 2.1.0 because of the
> config-delete-queues
> > deserialization bug introduced in 2.2.0. Unless that bug was squashed in
> > 2.6+. I don’t think I made a jira for it (vacation and work load) but we
> > discussed it back in April. I should go confirm that bug on 2.6 and make
> a
> > jira for that
> >
> > Thanks
> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <
> clebert.suconic@gmail.com
> > >
> > wrote:
> >
> > > There is an issue I remember where the journal would have some dirt
> that
> > > was fixed on 2.3/0.
> > >
> > > I would ipgrade to 2.6.2.
> > >
> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <da...@gmail.com>
> > > wrote:
> > >
> > > > would you be willing to help me translate these thread dumps?
> > > >
> > > > i attached a Zip file with some thread dumps in them. i will also
> share
> > > > the fasthread.io links for each file. (i was struggling getting
> > > > fastthread to do a combo report with the threads in the correct
> order)
> > > >
> > > > artemis04-20180719-1525 https://goo.gl/d88azU
> > > > artemis04-20180719-1530 https://goo.gl/G78qn3
> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> > > > artemis04-20180719-1540 https://goo.gl/brKxxk
> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> > > > artemis04-20180719-1550 https://goo.gl/r5dndK
> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> > > >
> > > > at :35, :45, and :55 the young+old gen space gets bigger than at the
> > > other
> > > > sample times. but i dont know what to look for in here to determine
> > what
> > > > the broker is actually during during this time.
> > > >
> > > > thanks
> > > >
> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jb...@apache.org>
> > > > wrote:
> > > >
> > > >> The first place I would start is grabbing thread dumps every so
> often
> > to
> > > >> see what the broker is actually doing during the 30+ minutes.
> > > >>
> > > >>
> > > >> Justin
> > > >>
> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <
> danlangford@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > so my server startup times and failover times are growing pretty
> > big.
> > > >> but i
> > > >> > dont really know where to start looking.
> > > >> >
> > > >> > here is a snippet of some logs to show you the time stamps:
> > > >> >
> > > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> > integration.bootstrap]
> > > >> > AMQ101000: Starting ActiveMQ Artemis Server
> > > >> >
> > > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221000:
> > > >> > live Message Broker is starting with configuration Broker
> > > Configuration
> > > >> > (clustered=true,journalDirectory=./data/
> > journal,bindingsDirectory=./
> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
> > > >> > pagingDirectory=./data/paging)
> > > >> >
> > > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221012:
> > > >> > Using AIO Journal
> > > >> >
> > > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221057:
> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size
> (-Xmx).
> > > >> being
> > > >> > defined as 1,073,741,824
> > > >> >
> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-server]. Adding protocol support
> > for:
> > > >> CORE
> > > >> >
> > > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
> > > support
> > > >> > for: AMQP
> > > >> >
> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol
> > > >> support
> > > >> > for: HORNETQ
> > > >> >
> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
> > > support
> > > >> > for: MQTT
> > > >> >
> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding
> protocol
> > > >> support
> > > >> > for: OPENWIRE
> > > >> >
> > > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > > AMQ221043:
> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol
> > > support
> > > >> > for: STOMP
> > > >> >
> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222165:
> > > >> No
> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings
> > > >> >
> > > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222166:
> > > >> No
> > > >> > Expiry Address configured for queue DLQ in AddressSettings
> > > >> >
> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222165:
> > > >> No
> > > >> > Dead Letter Address configured for queue ExpiryQueue in
> > > AddressSettings
> > > >> >
> > > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222166:
> > > >> No
> > > >> > Expiry Address configured for queue ExpiryQueue in AddressSettings
> > > >> >
> > > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222165:
> > > >> No
> > > >> > Dead Letter Address configured for queue example in
> AddressSettings
> > > >> >
> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222166:
> > > >> No
> > > >> > Expiry Address configured for queue example in AddressSettings
> > > >> >
> > > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222165:
> > > >> No
> > > >> > Dead Letter Address configured for queue exampleQueue in
> > > AddressSettings
> > > >> >
> > > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
> > > AMQ222166:
> > > >> No
> > > >> > Expiry Address configured for queue exampleQueue in
> AddressSettings
> > > >> >
> > > >> >
> > > >> > and it continues. i have 138 queues. i wonder if i need to be
> > looking
> > > at
> > > >> > PAGE configuration or some cache sizes. do i need to be looking at
> > the
> > > >> > number of messages persisted on these queues? where would you look
> > to
> > > >> > determine why the startup times are so long?
> > > >> >
> > > >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0
> due
> > > to
> > > >> > some deseralization changes introduced in 2.2.0. i think i have
> > > another
> > > >> > thread on here (that i need to update) regarding that upgrade
> issue.
> > > >> >
> > > >> > thanks for any ideas or insight you have for me
> > > >> >
> > > >>
> > > > --
> > > Clebert Suconic
> > >
> >
>
-- 
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Justin Bertram <jb...@apache.org>.

Analyzing thread dumps like this is pretty simple.  I generally just scroll
through and look for long stack-traces with lots of calls from
org.apache.activemq.artemis.  In your case every single thread dump has a
thread doing something like this:

"main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable
[0x00007f9031675000]
   java.lang.Thread.State: RUNNABLE
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373)
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192)
    at
org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113)
    at
org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108)
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348)
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168)
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147)
    at
org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120)
    at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424)
    at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374)
    at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219)
    - locked <0x0000000080a8fce8> (a
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
    at
org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109)
    at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518)
    at
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466)
    - locked <0x0000000080a8fce8> (a
org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl)
    at
org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111)
    - locked <0x000000008098be80> (a
org.apache.activemq.artemis.integration.FileBroker)
    at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73)
    at
org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148)
    at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95)
    at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129)
    at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49)

In every case it's the "main" thread which isn't surprising as that is the
thread responsible for starting the broker.  Also, you can pretty clearly
see in the trace that this is the thread starting the broker, it's loading
the journals, & restoring stored configuration (either address settings or
security settings).  I've seen high broker start times when there are lots
and lots of addresses and lots of and lots of settings.  Do either (or
both) of these situations apply to you?


Justin

On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <da...@gmail.com> wrote:

> Dang I can’t easily upgrade past 2.1.0 because of the config-delete-queues
> deserialization bug introduced in 2.2.0. Unless that bug was squashed in
> 2.6+. I don’t think I made a jira for it (vacation and work load) but we
> discussed it back in April. I should go confirm that bug on 2.6 and make a
> jira for that
>
> Thanks
> On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <clebert.suconic@gmail.com
> >
> wrote:
>
> > There is an issue I remember where the journal would have some dirt that
> > was fixed on 2.3/0.
> >
> > I would ipgrade to 2.6.2.
> >
> > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <da...@gmail.com>
> > wrote:
> >
> > > would you be willing to help me translate these thread dumps?
> > >
> > > i attached a Zip file with some thread dumps in them. i will also share
> > > the fasthread.io links for each file. (i was struggling getting
> > > fastthread to do a combo report with the threads in the correct order)
> > >
> > > artemis04-20180719-1525 https://goo.gl/d88azU
> > > artemis04-20180719-1530 https://goo.gl/G78qn3
> > > artemis04-20180719-1535 https://goo.gl/aMBSBw
> > > artemis04-20180719-1540 https://goo.gl/brKxxk
> > > artemis04-20180719-1545 https://goo.gl/RaXXCs
> > > artemis04-20180719-1550 https://goo.gl/r5dndK
> > > artemis04-20180719-1555 https://goo.gl/YJRLxe
> > >
> > > at :35, :45, and :55 the young+old gen space gets bigger than at the
> > other
> > > sample times. but i dont know what to look for in here to determine
> what
> > > the broker is actually during during this time.
> > >
> > > thanks
> > >
> > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jb...@apache.org>
> > > wrote:
> > >
> > >> The first place I would start is grabbing thread dumps every so often
> to
> > >> see what the broker is actually doing during the 30+ minutes.
> > >>
> > >>
> > >> Justin
> > >>
> > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <da...@gmail.com>
> > >> wrote:
> > >>
> > >> > so my server startup times and failover times are growing pretty
> big.
> > >> but i
> > >> > dont really know where to start looking.
> > >> >
> > >> > here is a snippet of some logs to show you the time stamps:
> > >> >
> > >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.
> integration.bootstrap]
> > >> > AMQ101000: Starting ActiveMQ Artemis Server
> > >> >
> > >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221000:
> > >> > live Message Broker is starting with configuration Broker
> > Configuration
> > >> > (clustered=true,journalDirectory=./data/
> journal,bindingsDirectory=./
> > >> > data/bindings,largeMessagesDirectory=./data/large-messages,
> > >> > pagingDirectory=./data/paging)
> > >> >
> > >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221012:
> > >> > Using AIO Journal
> > >> >
> > >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221057:
> > >> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx).
> > >> being
> > >> > defined as 1,073,741,824
> > >> >
> > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-server]. Adding protocol support
> for:
> > >> CORE
> > >> >
> > >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
> > support
> > >> > for: AMQP
> > >> >
> > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol
> > >> support
> > >> > for: HORNETQ
> > >> >
> > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
> > support
> > >> > for: MQTT
> > >> >
> > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-openwire-protocol]. Adding protocol
> > >> support
> > >> > for: OPENWIRE
> > >> >
> > >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> > AMQ221043:
> > >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol
> > support
> > >> > for: STOMP
> > >> >
> > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222165:
> > >> No
> > >> > Dead Letter Address configured for queue DLQ in AddressSettings
> > >> >
> > >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222166:
> > >> No
> > >> > Expiry Address configured for queue DLQ in AddressSettings
> > >> >
> > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222165:
> > >> No
> > >> > Dead Letter Address configured for queue ExpiryQueue in
> > AddressSettings
> > >> >
> > >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222166:
> > >> No
> > >> > Expiry Address configured for queue ExpiryQueue in AddressSettings
> > >> >
> > >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222165:
> > >> No
> > >> > Dead Letter Address configured for queue example in AddressSettings
> > >> >
> > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222166:
> > >> No
> > >> > Expiry Address configured for queue example in AddressSettings
> > >> >
> > >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222165:
> > >> No
> > >> > Dead Letter Address configured for queue exampleQueue in
> > AddressSettings
> > >> >
> > >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
> > AMQ222166:
> > >> No
> > >> > Expiry Address configured for queue exampleQueue in AddressSettings
> > >> >
> > >> >
> > >> > and it continues. i have 138 queues. i wonder if i need to be
> looking
> > at
> > >> > PAGE configuration or some cache sizes. do i need to be looking at
> the
> > >> > number of messages persisted on these queues? where would you look
> to
> > >> > determine why the startup times are so long?
> > >> >
> > >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due
> > to
> > >> > some deseralization changes introduced in 2.2.0. i think i have
> > another
> > >> > thread on here (that i need to update) regarding that upgrade issue.
> > >> >
> > >> > thanks for any ideas or insight you have for me
> > >> >
> > >>
> > > --
> > Clebert Suconic
> >
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

Dang I can’t easily upgrade past 2.1.0 because of the config-delete-queues
deserialization bug introduced in 2.2.0. Unless that bug was squashed in
2.6+. I don’t think I made a jira for it (vacation and work load) but we
discussed it back in April. I should go confirm that bug on 2.6 and make a
jira for that

Thanks
On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic <cl...@gmail.com>
wrote:

> There is an issue I remember where the journal would have some dirt that
> was fixed on 2.3/0.
>
> I would ipgrade to 2.6.2.
>
> On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <da...@gmail.com>
> wrote:
>
> > would you be willing to help me translate these thread dumps?
> >
> > i attached a Zip file with some thread dumps in them. i will also share
> > the fasthread.io links for each file. (i was struggling getting
> > fastthread to do a combo report with the threads in the correct order)
> >
> > artemis04-20180719-1525 https://goo.gl/d88azU
> > artemis04-20180719-1530 https://goo.gl/G78qn3
> > artemis04-20180719-1535 https://goo.gl/aMBSBw
> > artemis04-20180719-1540 https://goo.gl/brKxxk
> > artemis04-20180719-1545 https://goo.gl/RaXXCs
> > artemis04-20180719-1550 https://goo.gl/r5dndK
> > artemis04-20180719-1555 https://goo.gl/YJRLxe
> >
> > at :35, :45, and :55 the young+old gen space gets bigger than at the
> other
> > sample times. but i dont know what to look for in here to determine what
> > the broker is actually during during this time.
> >
> > thanks
> >
> > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jb...@apache.org>
> > wrote:
> >
> >> The first place I would start is grabbing thread dumps every so often to
> >> see what the broker is actually doing during the 30+ minutes.
> >>
> >>
> >> Justin
> >>
> >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <da...@gmail.com>
> >> wrote:
> >>
> >> > so my server startup times and failover times are growing pretty big.
> >> but i
> >> > dont really know where to start looking.
> >> >
> >> > here is a snippet of some logs to show you the time stamps:
> >> >
> >> > 08:11:31,801 INFO  [org.apache.activemq.artemis.integration.bootstrap]
> >> > AMQ101000: Starting ActiveMQ Artemis Server
> >> >
> >> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221000:
> >> > live Message Broker is starting with configuration Broker
> Configuration
> >> > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./
> >> > data/bindings,largeMessagesDirectory=./data/large-messages,
> >> > pagingDirectory=./data/paging)
> >> >
> >> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221012:
> >> > Using AIO Journal
> >> >
> >> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221057:
> >> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx).
> >> being
> >> > defined as 1,073,741,824
> >> >
> >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-server]. Adding protocol support for:
> >> CORE
> >> >
> >> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol
> support
> >> > for: AMQP
> >> >
> >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol
> >> support
> >> > for: HORNETQ
> >> >
> >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol
> support
> >> > for: MQTT
> >> >
> >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-openwire-protocol]. Adding protocol
> >> support
> >> > for: OPENWIRE
> >> >
> >> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server]
> AMQ221043:
> >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol
> support
> >> > for: STOMP
> >> >
> >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222165:
> >> No
> >> > Dead Letter Address configured for queue DLQ in AddressSettings
> >> >
> >> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222166:
> >> No
> >> > Expiry Address configured for queue DLQ in AddressSettings
> >> >
> >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222165:
> >> No
> >> > Dead Letter Address configured for queue ExpiryQueue in
> AddressSettings
> >> >
> >> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222166:
> >> No
> >> > Expiry Address configured for queue ExpiryQueue in AddressSettings
> >> >
> >> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222165:
> >> No
> >> > Dead Letter Address configured for queue example in AddressSettings
> >> >
> >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222166:
> >> No
> >> > Expiry Address configured for queue example in AddressSettings
> >> >
> >> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222165:
> >> No
> >> > Dead Letter Address configured for queue exampleQueue in
> AddressSettings
> >> >
> >> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server]
> AMQ222166:
> >> No
> >> > Expiry Address configured for queue exampleQueue in AddressSettings
> >> >
> >> >
> >> > and it continues. i have 138 queues. i wonder if i need to be looking
> at
> >> > PAGE configuration or some cache sizes. do i need to be looking at the
> >> > number of messages persisted on these queues? where would you look to
> >> > determine why the startup times are so long?
> >> >
> >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due
> to
> >> > some deseralization changes introduced in 2.2.0. i think i have
> another
> >> > thread on here (that i need to update) regarding that upgrade issue.
> >> >
> >> > thanks for any ideas or insight you have for me
> >> >
> >>
> > --
> Clebert Suconic
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Clebert Suconic <cl...@gmail.com>.

There is an issue I remember where the journal would have some dirt that
was fixed on 2.3/0.

I would ipgrade to 2.6.2.

On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <da...@gmail.com> wrote:

> would you be willing to help me translate these thread dumps?
>
> i attached a Zip file with some thread dumps in them. i will also share
> the fasthread.io links for each file. (i was struggling getting
> fastthread to do a combo report with the threads in the correct order)
>
> artemis04-20180719-1525 https://goo.gl/d88azU
> artemis04-20180719-1530 https://goo.gl/G78qn3
> artemis04-20180719-1535 https://goo.gl/aMBSBw
> artemis04-20180719-1540 https://goo.gl/brKxxk
> artemis04-20180719-1545 https://goo.gl/RaXXCs
> artemis04-20180719-1550 https://goo.gl/r5dndK
> artemis04-20180719-1555 https://goo.gl/YJRLxe
>
> at :35, :45, and :55 the young+old gen space gets bigger than at the other
> sample times. but i dont know what to look for in here to determine what
> the broker is actually during during this time.
>
> thanks
>
> On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jb...@apache.org>
> wrote:
>
>> The first place I would start is grabbing thread dumps every so often to
>> see what the broker is actually doing during the 30+ minutes.
>>
>>
>> Justin
>>
>> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <da...@gmail.com>
>> wrote:
>>
>> > so my server startup times and failover times are growing pretty big.
>> but i
>> > dont really know where to start looking.
>> >
>> > here is a snippet of some logs to show you the time stamps:
>> >
>> > 08:11:31,801 INFO  [org.apache.activemq.artemis.integration.bootstrap]
>> > AMQ101000: Starting ActiveMQ Artemis Server
>> >
>> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server] AMQ221000:
>> > live Message Broker is starting with configuration Broker Configuration
>> > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./
>> > data/bindings,largeMessagesDirectory=./data/large-messages,
>> > pagingDirectory=./data/paging)
>> >
>> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server] AMQ221012:
>> > Using AIO Journal
>> >
>> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server] AMQ221057:
>> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx).
>> being
>> > defined as 1,073,741,824
>> >
>> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-server]. Adding protocol support for:
>> CORE
>> >
>> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-amqp-protocol]. Adding protocol support
>> > for: AMQP
>> >
>> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol
>> support
>> > for: HORNETQ
>> >
>> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol support
>> > for: MQTT
>> >
>> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-openwire-protocol]. Adding protocol
>> support
>> > for: OPENWIRE
>> >
>> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
>> > Protocol module found: [artemis-stomp-protocol]. Adding protocol support
>> > for: STOMP
>> >
>> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
>> No
>> > Dead Letter Address configured for queue DLQ in AddressSettings
>> >
>> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
>> No
>> > Expiry Address configured for queue DLQ in AddressSettings
>> >
>> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
>> No
>> > Dead Letter Address configured for queue ExpiryQueue in AddressSettings
>> >
>> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
>> No
>> > Expiry Address configured for queue ExpiryQueue in AddressSettings
>> >
>> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
>> No
>> > Dead Letter Address configured for queue example in AddressSettings
>> >
>> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
>> No
>> > Expiry Address configured for queue example in AddressSettings
>> >
>> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
>> No
>> > Dead Letter Address configured for queue exampleQueue in AddressSettings
>> >
>> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
>> No
>> > Expiry Address configured for queue exampleQueue in AddressSettings
>> >
>> >
>> > and it continues. i have 138 queues. i wonder if i need to be looking at
>> > PAGE configuration or some cache sizes. do i need to be looking at the
>> > number of messages persisted on these queues? where would you look to
>> > determine why the startup times are so long?
>> >
>> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to
>> > some deseralization changes introduced in 2.2.0. i think i have another
>> > thread on here (that i need to update) regarding that upgrade issue.
>> >
>> > thanks for any ideas or insight you have for me
>> >
>>
> --
Clebert Suconic

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Dan Langford <da...@gmail.com>.

would you be willing to help me translate these thread dumps?

i attached a Zip file with some thread dumps in them. i will also share the
fasthread.io links for each file. (i was struggling getting fastthread to
do a combo report with the threads in the correct order)

artemis04-20180719-1525 https://goo.gl/d88azU
artemis04-20180719-1530 https://goo.gl/G78qn3
artemis04-20180719-1535 https://goo.gl/aMBSBw
artemis04-20180719-1540 https://goo.gl/brKxxk
artemis04-20180719-1545 https://goo.gl/RaXXCs
artemis04-20180719-1550 https://goo.gl/r5dndK
artemis04-20180719-1555 https://goo.gl/YJRLxe

at :35, :45, and :55 the young+old gen space gets bigger than at the other
sample times. but i dont know what to look for in here to determine what
the broker is actually during during this time.

thanks

On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram <jb...@apache.org> wrote:

> The first place I would start is grabbing thread dumps every so often to
> see what the broker is actually doing during the 30+ minutes.
>
>
> Justin
>
> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <da...@gmail.com>
> wrote:
>
> > so my server startup times and failover times are growing pretty big.
> but i
> > dont really know where to start looking.
> >
> > here is a snippet of some logs to show you the time stamps:
> >
> > 08:11:31,801 INFO  [org.apache.activemq.artemis.integration.bootstrap]
> > AMQ101000: Starting ActiveMQ Artemis Server
> >
> > 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server] AMQ221000:
> > live Message Broker is starting with configuration Broker Configuration
> > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./
> > data/bindings,largeMessagesDirectory=./data/large-messages,
> > pagingDirectory=./data/paging)
> >
> > 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server] AMQ221012:
> > Using AIO Journal
> >
> > 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server] AMQ221057:
> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx).
> being
> > defined as 1,073,741,824
> >
> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-server]. Adding protocol support for:
> CORE
> >
> > 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-amqp-protocol]. Adding protocol support
> > for: AMQP
> >
> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol
> support
> > for: HORNETQ
> >
> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol support
> > for: MQTT
> >
> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-openwire-protocol]. Adding protocol
> support
> > for: OPENWIRE
> >
> > 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> > Protocol module found: [artemis-stomp-protocol]. Adding protocol support
> > for: STOMP
> >
> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
> No
> > Dead Letter Address configured for queue DLQ in AddressSettings
> >
> > 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
> No
> > Expiry Address configured for queue DLQ in AddressSettings
> >
> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
> No
> > Dead Letter Address configured for queue ExpiryQueue in AddressSettings
> >
> > 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
> No
> > Expiry Address configured for queue ExpiryQueue in AddressSettings
> >
> > 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
> No
> > Dead Letter Address configured for queue example in AddressSettings
> >
> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
> No
> > Expiry Address configured for queue example in AddressSettings
> >
> > 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
> No
> > Dead Letter Address configured for queue exampleQueue in AddressSettings
> >
> > 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
> No
> > Expiry Address configured for queue exampleQueue in AddressSettings
> >
> >
> > and it continues. i have 138 queues. i wonder if i need to be looking at
> > PAGE configuration or some cache sizes. do i need to be looking at the
> > number of messages persisted on these queues? where would you look to
> > determine why the startup times are so long?
> >
> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to
> > some deseralization changes introduced in 2.2.0. i think i have another
> > thread on here (that i need to update) regarding that upgrade issue.
> >
> > thanks for any ideas or insight you have for me
> >
>

Re: [artemis 2.1.0] taking 30+ minutes to boot & failover

Posted by Justin Bertram <jb...@apache.org>.

The first place I would start is grabbing thread dumps every so often to
see what the broker is actually doing during the 30+ minutes.


Justin

On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford <da...@gmail.com> wrote:

> so my server startup times and failover times are growing pretty big. but i
> dont really know where to start looking.
>
> here is a snippet of some logs to show you the time stamps:
>
> 08:11:31,801 INFO  [org.apache.activemq.artemis.integration.bootstrap]
> AMQ101000: Starting ActiveMQ Artemis Server
>
> 08:11:31,818 INFO  [org.apache.activemq.artemis.core.server] AMQ221000:
> live Message Broker is starting with configuration Broker Configuration
> (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./
> data/bindings,largeMessagesDirectory=./data/large-messages,
> pagingDirectory=./data/paging)
>
> 08:11:34,462 INFO  [org.apache.activemq.artemis.core.server] AMQ221012:
> Using AIO Journal
>
> 08:11:34,493 INFO  [org.apache.activemq.artemis.core.server] AMQ221057:
> Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being
> defined as 1,073,741,824
>
> 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-server]. Adding protocol support for: CORE
>
> 08:11:34,555 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-amqp-protocol]. Adding protocol support
> for: AMQP
>
> 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-hornetq-protocol]. Adding protocol support
> for: HORNETQ
>
> 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-mqtt-protocol]. Adding protocol support
> for: MQTT
>
> 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-openwire-protocol]. Adding protocol support
> for: OPENWIRE
>
> 08:11:34,556 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-stomp-protocol]. Adding protocol support
> for: STOMP
>
> 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
> Dead Letter Address configured for queue DLQ in AddressSettings
>
> 08:41:38,963 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
> Expiry Address configured for queue DLQ in AddressSettings
>
> 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
> Dead Letter Address configured for queue ExpiryQueue in AddressSettings
>
> 08:41:38,983 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
> Expiry Address configured for queue ExpiryQueue in AddressSettings
>
> 08:41:38,984 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
> Dead Letter Address configured for queue example in AddressSettings
>
> 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
> Expiry Address configured for queue example in AddressSettings
>
> 08:41:38,985 WARN  [org.apache.activemq.artemis.core.server] AMQ222165: No
> Dead Letter Address configured for queue exampleQueue in AddressSettings
>
> 08:41:38,986 WARN  [org.apache.activemq.artemis.core.server] AMQ222166: No
> Expiry Address configured for queue exampleQueue in AddressSettings
>
>
> and it continues. i have 138 queues. i wonder if i need to be looking at
> PAGE configuration or some cache sizes. do i need to be looking at the
> number of messages persisted on these queues? where would you look to
> determine why the startup times are so long?
>
> also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to
> some deseralization changes introduced in 2.2.0. i think i have another
> thread on here (that i need to update) regarding that upgrade issue.
>
> thanks for any ideas or insight you have for me
>