You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by MILOVIDOV Aleksandr <Al...@raiffeisen.ru.INVALID> on 2024/04/18 12:15:15 UTC
Error after upgrading Artemis from 2.32.0 to 2.33.0
Hi All,
We have faced problem when upgrading one of ActiveMQ Artemis servers from 2.32.0 to 2.33.0.
Artemis cannot start and writes an error to the log:
2024-04-18 13:31:44,975 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation
java.lang.IndexOutOfBoundsException: readerIndex(82) + length(1) exceeds writerIndex(82): UnpooledHeapByteBuf(ridx: 82, widx: 82, cap: 82/82)
at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442) ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730) ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529) ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
at org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:150) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:79) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.persistence.config.PersistedSecuritySetting.decode(PersistedSecuritySetting.java:255) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newSecurityRecord(AbstractJournalStorageManager.java:2127) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.lambda$loadBindingJournal$5(AbstractJournalStorageManager.java:1640) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList$SparseArray.clear(SparseArrayLinkedList.java:114) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clearSparseArrayList(SparseArrayLinkedList.java:173) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clear(SparseArrayLinkedList.java:227) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1613) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:3856) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3489) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ReplicationPrimaryActivation.run(ReplicationPrimaryActivation.java:145) [artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:738) [artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:628) [artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221) [artemis-cli-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167) [artemis-cli-2.33.0.jar:2.33.0]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:157) [artemis-boot.jar:2.33.0]
at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64) [artemis-boot.jar:2.33.0]
and warning:
2024-04-18 13:31:45,977 WARN [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator is closed (maybe it was garbage collected)
java.lang.IllegalStateException: Server locator is closed (maybe it was garbage collected)
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1906) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:678) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:579) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:558) ~[artemis-core-client-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:501) ~[artemis-server-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.33.0.jar:2.33.0]
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.33.0.jar:2.33.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.33.0.jar:2.33.0]
It can start with new version only if we remove all existing data from Artemis instance data directory.
Also we have exported all data from this non-working instance with version 2.33.0 using artemis data exp utility, and imported to another instance with version 2.33.0, and it has been started successfully.
Artemis is configured using primary/backup replication with pluggable quorum.
We have also successfully upgraded several other instances of Artemis which has smaller number of producers/consumers and lower throughput.
--
Best regards,
Aleksandr
-----------------------------------
This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed.
-----------------------------------
RE: Error after upgrading Artemis from 2.32.0 to 2.33.0
Posted by MILOVIDOV Aleksandr <Al...@raiffeisen.ru.INVALID>.
Hi,
On Mon, Apr 22, 2024 at 6:46 PM Justin Bertram <jb...@apache.org> wrote:
> This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and
> sent a PR [3] to address the problem.
Thanks Justin.
On Tuesday, April 23, 2024 3:27 PM Stephen Baker <st...@motorolasolutions.com.INVALID> wrote:
> Can you clarify what condition triggers this (i.e. when would there be a peristable security setting with an empty write or view buffer)?
I have found one manually created security-setting:
getRolesAsJSON(TEST.QUEUE)
[{"name":"amq","send":true,"consume":false,"createDurableQueue":false,"deleteDurableQueue":false,"createNonDurableQueue":false,"deleteNonDurableQueue":false,"manage":false,"browse":false,"createAddress":false,"deleteAddress":false,"view":false,"edit":false}]
It was created for testing purposes using API with command:
curl -X POST -H "Content-Type: application/json" -H "Origin:https://${HOSTNAME}:8161" -u ${ARTEMIS_USER}:${ARTEMIS_PASSWORD} https://${HOSTNAME}:8161/console/jolokia -d '{"type" : "EXEC", "mbean" : "org.apache.activemq.artemis:broker=\"'${HOSTNAME}'\"", "operation" : "addSecuritySettings(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String)", "arguments" : ["TEST.QUEUE","amq","","","","","",""]}'
And I have reproduced the problem: created new instance of Artemis 2.32.0, added this security setting using curl (this command works in linux shell), and then stopped and upgraded this instance to 2.33.0.
After upgrading, it has failed to start with similar error.
BTW it is difficult to find security setting for arbitrary address match without knowing it's exact name, but one of our developers has written an application which starts embedded Artemis with binding files which were copied from working instance of Artemis and exports all found address matches with permissions. It could be great to have some API method to export all persisted security settings.
--
Best regards,
Aleksandr
-----------------------------------
This message and any attachment are confidential and may be privileged or otherwise protected from disclosure. If you are not the intended recipient any use, distribution, copying or disclosure is strictly prohibited. If you have received this message in error, please notify the sender immediately either by telephone or by e-mail and delete this message and any attachment from your system. Correspondence via e-mail is for information purposes only. AO Raiffeisenbank neither makes nor accepts legally binding statements by e-mail unless otherwise agreed.
-----------------------------------
Re: Error after upgrading Artemis from 2.32.0 to 2.33.0
Posted by Justin Bertram <jb...@apache.org>.
> It could be great to have some API method to export all persisted
security settings.
It's not part of the management API, per se, but the "data print" command
will print all the entries from the bindings journal - included
PersistedSecuritySetting records. The simplest way to do this is with this
command while the broker is stopped:
./artemis data print --skip-journal
Justin
On Tue, Apr 23, 2024 at 11:04 AM Justin Bertram <jb...@apache.org> wrote:
> > Can you clarify what condition triggers this...?
>
> Security settings are persisted to the bindings journal when they are
> added via the management API (e.g. via addSecuritySettings on the
> ActiveMQServerControl MBean). If this was done before upgrading to 2.33.0
> then the broker will fail to decode these settings after the upgrade.
>
>
> Justin
>
> On Tue, Apr 23, 2024 at 7:27 AM Stephen Baker
> <st...@motorolasolutions.com.invalid> wrote:
>
>> On Mon, Apr 22, 2024 at 6:46 PM Justin Bertram <jb...@apache.org>
>> wrote:
>>
>> > This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and
>> sent
>> > a PR [3] to address the problem.
>> >
>> > To be clear, this has nothing to do with replication or load/throughput.
>> > It's just a buffer formatting issue caused by a configuration change.
>> >
>>
>> Can you clarify what condition triggers this (i.e. when would there be a
>> peristable security setting with an empty write or view buffer)?
>>
>> --
>>
>>
>> *For more information on how and why we collect your personal
>> information, please visit our Privacy Policy
>> <
>> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
>> >.*
>>
>
Re: Error after upgrading Artemis from 2.32.0 to 2.33.0
Posted by Justin Bertram <jb...@apache.org>.
> Can you clarify what condition triggers this...?
Security settings are persisted to the bindings journal when they are added
via the management API (e.g. via addSecuritySettings on the
ActiveMQServerControl MBean). If this was done before upgrading to 2.33.0
then the broker will fail to decode these settings after the upgrade.
Justin
On Tue, Apr 23, 2024 at 7:27 AM Stephen Baker
<st...@motorolasolutions.com.invalid> wrote:
> On Mon, Apr 22, 2024 at 6:46 PM Justin Bertram <jb...@apache.org>
> wrote:
>
> > This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and
> sent
> > a PR [3] to address the problem.
> >
> > To be clear, this has nothing to do with replication or load/throughput.
> > It's just a buffer formatting issue caused by a configuration change.
> >
>
> Can you clarify what condition triggers this (i.e. when would there be a
> peristable security setting with an empty write or view buffer)?
>
> --
>
>
> *For more information on how and why we collect your personal
> information, please visit our Privacy Policy
> <
> https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement
> >.*
>
Re: Error after upgrading Artemis from 2.32.0 to 2.33.0
Posted by Stephen Baker <st...@motorolasolutions.com.INVALID>.
On Mon, Apr 22, 2024 at 6:46 PM Justin Bertram <jb...@apache.org> wrote:
> This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and sent
> a PR [3] to address the problem.
>
> To be clear, this has nothing to do with replication or load/throughput.
> It's just a buffer formatting issue caused by a configuration change.
>
Can you clarify what condition triggers this (i.e. when would there be a
peristable security setting with an empty write or view buffer)?
--
*For more information on how and why we collect your personal
information, please visit our Privacy Policy
<https://www.motorolasolutions.com/en_us/about/privacy-policy.html?elqTrackId=8980d888905940e39a2613a7a3dcb0a7&elqaid=2786&elqat=2#privacystatement>.*
Re: Error after upgrading Artemis from 2.32.0 to 2.33.0
Posted by Justin Bertram <jb...@apache.org>.
This was caused by ARTEMIS-4582 [1]. I've opened ARTEMIS-4742 [2] and sent
a PR [3] to address the problem.
To be clear, this has nothing to do with replication or load/throughput.
It's just a buffer formatting issue caused by a configuration change.
Justin
[1] https://issues.apache.org/jira/browse/ARTEMIS-4582
[2] https://issues.apache.org/jira/browse/ARTEMIS-4742
[3] https://github.com/apache/activemq-artemis/pull/4904
On Thu, Apr 18, 2024 at 7:15 AM MILOVIDOV Aleksandr
<Al...@raiffeisen.ru.invalid> wrote:
> Hi All,
>
> We have faced problem when upgrading one of ActiveMQ Artemis servers from
> 2.32.0 to 2.33.0.
> Artemis cannot start and writes an error to the log:
>
> 2024-04-18 13:31:44,975 ERROR [org.apache.activemq.artemis.core.server]
> AMQ224000: Failure in initialisation
> java.lang.IndexOutOfBoundsException: readerIndex(82) + length(1) exceeds
> writerIndex(82): UnpooledHeapByteBuf(ridx: 82, widx: 82, cap: 82/82)
> at
> io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
> ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
> at
> io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730)
> ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
> at
> io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:529)
> ~[netty-buffer-4.1.107.Final.jar:4.1.107.Final]
> at
> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:150)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:79)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.persistence.config.PersistedSecuritySetting.decode(PersistedSecuritySetting.java:255)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newSecurityRecord(AbstractJournalStorageManager.java:2127)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.lambda$loadBindingJournal$5(AbstractJournalStorageManager.java:1640)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList$SparseArray.clear(SparseArrayLinkedList.java:114)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clearSparseArrayList(SparseArrayLinkedList.java:173)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.collections.SparseArrayLinkedList.clear(SparseArrayLinkedList.java:227)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1613)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:3856)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:3489)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.impl.ReplicationPrimaryActivation.run(ReplicationPrimaryActivation.java:145)
> [artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:738)
> [artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:628)
> [artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:66)
> [artemis-cli-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:130)
> [artemis-cli-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:221)
> [artemis-cli-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:167)
> [artemis-cli-2.33.0.jar:2.33.0]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) ~[?:?]
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> ~[?:?]
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> ~[?:?]
> at
> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:157)
> [artemis-boot.jar:2.33.0]
> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:64)
> [artemis-boot.jar:2.33.0]
>
> and warning:
>
> 2024-04-18 13:31:45,977 WARN
> [org.apache.activemq.artemis.utils.actors.OrderedExecutor] Server locator
> is closed (maybe it was garbage collected)
> java.lang.IllegalStateException: Server locator is closed (maybe it was
> garbage collected)
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.assertOpen(ServerLocatorImpl.java:1906)
> ~[artemis-core-client-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:678)
> ~[artemis-core-client-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:579)
> ~[artemis-core-client-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:558)
> ~[artemis-core-client-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:501)
> ~[artemis-server-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
> ~[artemis-commons-2.33.0.jar:2.33.0]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> [?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> [?:?]
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.33.0.jar:2.33.0]
>
> It can start with new version only if we remove all existing data from
> Artemis instance data directory.
> Also we have exported all data from this non-working instance with version
> 2.33.0 using artemis data exp utility, and imported to another instance
> with version 2.33.0, and it has been started successfully.
> Artemis is configured using primary/backup replication with pluggable
> quorum.
> We have also successfully upgraded several other instances of Artemis
> which has smaller number of producers/consumers and lower throughput.
>
> --
> Best regards,
> Aleksandr
>
>
>
> -----------------------------------
>
> This message and any attachment are confidential and may be privileged or
> otherwise protected from disclosure. If you are not the intended recipient
> any use, distribution, copying or disclosure is strictly prohibited. If you
> have received this message in error, please notify the sender immediately
> either by telephone or by e-mail and delete this message and any attachment
> from your system. Correspondence via e-mail is for information purposes
> only. AO Raiffeisenbank neither makes nor accepts legally binding
> statements by e-mail unless otherwise agreed.
>
> -----------------------------------
>