You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Stephen Baker <st...@rmssoftwareinc.com> on 2022/10/11 14:06:43 UTC
Mirror compatibility across versions
We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
Stephen E Baker
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
I’m finishing that today.
On Tue, Oct 25, 2022 at 3:53 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Is there anything I can do to help get the PR moving?
>
> Is it still in draft due to the Travis failures?
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Friday, October 14, 2022 at 9:41 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> It's being done as part of this PR:
>
> https://github.com/apache/activemq-artemis/pull/4246
>
> On Fri, Oct 14, 2022 at 4:40 AM Robbie Gemmell <ro...@gmail.com>
> wrote:
> >
> > That helper command doesnt exist yet as Clebert said since the idea
> > only came from discussing something else the other day, but the
> > pre-existing logging related changes coming for 2.27.0 are covered in
> > the docs along with diffs of the script changes from 2.26.0, you can
> > see the current source version of those docs at
> >
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/versions.md#2270
> > . They will be updated to reference the helper once it exits.
> >
> > On Thu, 13 Oct 2022 at 18:52, Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Because bin/artemis includes references to the jboss logmanager
> causing artemis to fail on startup
> > >
> > > Diffing my two instances I see:
> > > # Set Defaults Properties
> > > ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> > > ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
> > >
> > >
> > > # finding the Log Manager
> > > LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
> > >
> > > if [ -z "$LOG_MANAGER" ] ; then
> > > # this is the one found when the server was created
> > > LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> > > fi
> > >
> > > WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> > > if [ -z "$WILDFLY_COMMON" ] ; then
> > > # this is the one found when the server was created
> > > WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> > > fi
> > >
> > > -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
> > >
> > > and
> > >
> > > -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> > > -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
> > >
> > > all have to go.
> > >
> > > I also see a change in the schema for bootstrap.xml (binding child of
> web), and the browse permission added to management.xml
> > >
> > > Along with the new log4j.properties you mentioned
> > >
> > > Some of those changes might be earlier if they’re backwards
> compatible, I’ve carried forward this configuration for awhile, but in
> particular the shell script changes appear to be manditory.
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Thursday, October 13, 2022 at 12:49 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > I think the record would pile up unacked at the source mirror.
> > >
> > >
> > > and @Stephen baker: sorry about my mistake on this fix...
> > >
> > >
> > > Why would the upgrade be difficult on 2.27? it's just adding a
> > > log4j2.properties.. everything else should be the same.
> > >
> > >
> > > You should probably bring a patched version yourself until we can make
> > > a release? I'm thinking we should make a release next week.
> > >
> > > On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > Your patch does resolve the error. Artemis 2.27 looks like it will
> be a difficult upgrade, I ended up making a new instance and merging config
> over.
> > > >
> > > > I have pasted a trace in
> https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > >
> > > > What is the impact of this issue? I’m trying to decide whether to
> advise our IT team to continue with the planned upgrade or hold off until
> 2.27. We will definitely encounter this condition in production. From a
> surface reading, possibly a resource leak?
> > > >
> > > >
> > > >
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Wednesday, October 12, 2022 at 9:54 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > Notice that main is now using SLF4j / log4j... (in case you manually
> > > > upgrade to a snapshot)
> > > >
> > > >
> > > > We are still working the details for an upgrade.
> > > >
> > > >
> > > > if you need to patch your 2.25.0 it's a straight change to make there
> > > >
> > > > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > > > <cl...@gmail.com> wrote:
> > > > >
> > > > > I don't know how I would test it yet. It's fairly late in the night
> > > > > for me.. I will think about it tomorrow.
> > > > >
> > > > >
> > > > > but here is a tentative fix:
> > > > >
> > > > > https://github.com/apache/activemq-artemis/pull/4256
> > > > >
> > > > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > That’s the full output with regular logging levels. I can
> reproduce at will so I have enabled trace level logging and pasted the
> result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > > > >
> > > > > > Let’s take further discussion there?
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > is this the actual trace? or you cut some to post here?
> > > > > >
> > > > > >
> > > > > > Just puzzled by skipDelivery calling performAck..
> > > > > >
> > > > > > artemis-test-artemis-1-m-1 | at
> > > > > >
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> > > > > >
> org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > > >
> > > > > >
> > > > > > can you post the full stack if this is not it?
> > > > > >
> > > > > >
> > > > > > it definitely needs fixing... I'm investigating it.
> > > > > >
> > > > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > Having updated both sides to 2.25 I’m seeing this error in the
> logs, is it a concern that warrants further investigation?
> > > > > > >
> > > > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR
> [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver:
> java.lang.IllegalStateException: this method requires to be called within
> the handler, use the executor
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
> [artemis-commons-2.25.0.jar:]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.25.0.jar:]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
> [artemis-commons-2.25.0.jar:]
> > > > > > > artemis-test-artemis-1-m-1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> > > > > > > artemis-test-artemis-1-m-1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> > > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.25.0.jar:]
> > > > > > >
> > > > > > >
> > > > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > I set up some docker images in this configuration as a
> preliminary step. One oddity:
> > > > > > >
> > > > > > > Configure 2.25 side not to run the reaper
> > > > > > > Send message to 2.25 side
> > > > > > > Observe that after expiry the message shows up in the expiry
> queue on the 2.20 side, but not on the 2.25 side, the message is removed
> from the original queue on both sides.
> > > > > > >
> > > > > > > If the message is originally sent to the 2.20 side it shows up
> in both queues as expected.
> > > > > > >
> > > > > > > There’s probably a reason for it, but I didn’t expect this
> change. I thought that we would continue to see the old bugs until both
> sides were updated.
> > > > > > >
> > > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > Yeah.. something like that... not necessarily in there though.
> but a
> > > > > > > similar test.
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > > >
> > > > > > > > Ok, I agree based on a cursory reading of that patch. The
> extra ackReason defaults to normal in one direction and isn’t read in the
> other direction. Killed, replaced, and expired being interpreted as normal
> just means that the 2.20 bugs will persist until both sides are updated.
> > > > > > > >
> > > > > > > > I’ll test it out with different version docker containers. I
> suppose as far as writing tests you mean something like the
> MultiVersionReplicaTest.
> > > > > > > >
> > > > > > > > Stephen E. Baker
> > > > > > > >
> > > > > > > >
> > > > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > > In theory it should work.
> > > > > > > >
> > > > > > > >
> > > > > > > > Only change that might break compatibility is
> > > > > > > >
> https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror
> to
> > > > > > > > determine target operations and fixing Delivering statistics
> on Mirror
> > > > > > > >
> > > > > > > >
> > > > > > > > I tried to not break compatibility, but I just realized we
> should add
> > > > > > > > a test to validate compatibility between mirrors.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > so, I will say it should be compatible, but I would test it
> before
> > > > > > > > doing it in the real system.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > if you are willing to contribute to a compatibility test :)
> > > > > > > >
> > > > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > > > >
> > > > > > > > > We are planning our production upgrade from 2.20 to 2.25.
> These upgrades involve a loss of service in the window between stopping the
> live and when the backup instance becomes ready to process messages.
> > > > > > > > >
> > > > > > > > > I was wondering if the mirror protocol is expected to be
> compatible between those versions. If so we could upgrade our cold site,
> and then wait for a planned failover to avoid any additional down time. I
> know that quite a bit of work was done by Clebert in 2.24 so I was hoping
> he could weigh in.
> > > > > > > > >
> > > > > > > > > Stephen E Baker
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Clebert Suconic
> > > > > > > > [EXTERNAL]: This email originated from outside of Rave
> Mobile Safety. Do not click links or open attachments unless you recognize
> the sender and know the content is safe.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Clebert Suconic
> > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Is there anything I can do to help get the PR moving?
Is it still in draft due to the Travis failures?
From: Clebert Suconic <cl...@gmail.com>
Date: Friday, October 14, 2022 at 9:41 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
It's being done as part of this PR:
https://github.com/apache/activemq-artemis/pull/4246
On Fri, Oct 14, 2022 at 4:40 AM Robbie Gemmell <ro...@gmail.com> wrote:
>
> That helper command doesnt exist yet as Clebert said since the idea
> only came from discussing something else the other day, but the
> pre-existing logging related changes coming for 2.27.0 are covered in
> the docs along with diffs of the script changes from 2.26.0, you can
> see the current source version of those docs at
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/versions.md#2270
> . They will be updated to reference the helper once it exits.
>
> On Thu, 13 Oct 2022 at 18:52, Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
> >
> > Diffing my two instances I see:
> > # Set Defaults Properties
> > ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> > ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
> >
> >
> > # finding the Log Manager
> > LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
> >
> > if [ -z "$LOG_MANAGER" ] ; then
> > # this is the one found when the server was created
> > LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> > fi
> >
> > WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> > if [ -z "$WILDFLY_COMMON" ] ; then
> > # this is the one found when the server was created
> > WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> > fi
> >
> > -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
> >
> > and
> >
> > -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> > -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
> >
> > all have to go.
> >
> > I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
> >
> > Along with the new log4j.properties you mentioned
> >
> > Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Thursday, October 13, 2022 at 12:49 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > I think the record would pile up unacked at the source mirror.
> >
> >
> > and @Stephen baker: sorry about my mistake on this fix...
> >
> >
> > Why would the upgrade be difficult on 2.27? it's just adding a
> > log4j2.properties.. everything else should be the same.
> >
> >
> > You should probably bring a patched version yourself until we can make
> > a release? I'm thinking we should make a release next week.
> >
> > On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
> > >
> > > I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
> > >
> > >
> > >
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:54 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > Notice that main is now using SLF4j / log4j... (in case you manually
> > > upgrade to a snapshot)
> > >
> > >
> > > We are still working the details for an upgrade.
> > >
> > >
> > > if you need to patch your 2.25.0 it's a straight change to make there
> > >
> > > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > >
> > > > I don't know how I would test it yet. It's fairly late in the night
> > > > for me.. I will think about it tomorrow.
> > > >
> > > >
> > > > but here is a tentative fix:
> > > >
> > > > https://github.com/apache/activemq-artemis/pull/4256
> > > >
> > > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > > >
> > > > > Let’s take further discussion there?
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > is this the actual trace? or you cut some to post here?
> > > > >
> > > > >
> > > > > Just puzzled by skipDelivery calling performAck..
> > > > >
> > > > > artemis-test-artemis-1-m-1 | at
> > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at
> > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > >
> > > > >
> > > > > can you post the full stack if this is not it?
> > > > >
> > > > >
> > > > > it definitely needs fixing... I'm investigating it.
> > > > >
> > > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > > > >
> > > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > > > >
> > > > > >
> > > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > > > >
> > > > > > Configure 2.25 side not to run the reaper
> > > > > > Send message to 2.25 side
> > > > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > > > >
> > > > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > > > >
> > > > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > Yeah.. something like that... not necessarily in there though. but a
> > > > > > similar test.
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > > > >
> > > > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > > > >
> > > > > > > Stephen E. Baker
> > > > > > >
> > > > > > >
> > > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > In theory it should work.
> > > > > > >
> > > > > > >
> > > > > > > Only change that might break compatibility is
> > > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > > > >
> > > > > > >
> > > > > > > I tried to not break compatibility, but I just realized we should add
> > > > > > > a test to validate compatibility between mirrors.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > so, I will say it should be compatible, but I would test it before
> > > > > > > doing it in the real system.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > if you are willing to contribute to a compatibility test :)
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > > >
> > > > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > > > >
> > > > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > > > >
> > > > > > > > Stephen E Baker
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Clebert Suconic
> > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
It's being done as part of this PR:
https://github.com/apache/activemq-artemis/pull/4246
On Fri, Oct 14, 2022 at 4:40 AM Robbie Gemmell <ro...@gmail.com> wrote:
>
> That helper command doesnt exist yet as Clebert said since the idea
> only came from discussing something else the other day, but the
> pre-existing logging related changes coming for 2.27.0 are covered in
> the docs along with diffs of the script changes from 2.26.0, you can
> see the current source version of those docs at
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/versions.md#2270
> . They will be updated to reference the helper once it exits.
>
> On Thu, 13 Oct 2022 at 18:52, Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
> >
> > Diffing my two instances I see:
> > # Set Defaults Properties
> > ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> > ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
> >
> >
> > # finding the Log Manager
> > LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
> >
> > if [ -z "$LOG_MANAGER" ] ; then
> > # this is the one found when the server was created
> > LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> > fi
> >
> > WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> > if [ -z "$WILDFLY_COMMON" ] ; then
> > # this is the one found when the server was created
> > WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> > fi
> >
> > -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
> >
> > and
> >
> > -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> > -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
> >
> > all have to go.
> >
> > I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
> >
> > Along with the new log4j.properties you mentioned
> >
> > Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Thursday, October 13, 2022 at 12:49 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > I think the record would pile up unacked at the source mirror.
> >
> >
> > and @Stephen baker: sorry about my mistake on this fix...
> >
> >
> > Why would the upgrade be difficult on 2.27? it's just adding a
> > log4j2.properties.. everything else should be the same.
> >
> >
> > You should probably bring a patched version yourself until we can make
> > a release? I'm thinking we should make a release next week.
> >
> > On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
> > >
> > > I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
> > >
> > >
> > >
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:54 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > Notice that main is now using SLF4j / log4j... (in case you manually
> > > upgrade to a snapshot)
> > >
> > >
> > > We are still working the details for an upgrade.
> > >
> > >
> > > if you need to patch your 2.25.0 it's a straight change to make there
> > >
> > > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > >
> > > > I don't know how I would test it yet. It's fairly late in the night
> > > > for me.. I will think about it tomorrow.
> > > >
> > > >
> > > > but here is a tentative fix:
> > > >
> > > > https://github.com/apache/activemq-artemis/pull/4256
> > > >
> > > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > > >
> > > > > Let’s take further discussion there?
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > is this the actual trace? or you cut some to post here?
> > > > >
> > > > >
> > > > > Just puzzled by skipDelivery calling performAck..
> > > > >
> > > > > artemis-test-artemis-1-m-1 | at
> > > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at
> > > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > >
> > > > >
> > > > > can you post the full stack if this is not it?
> > > > >
> > > > >
> > > > > it definitely needs fixing... I'm investigating it.
> > > > >
> > > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > > > >
> > > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > > > >
> > > > > >
> > > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > > > >
> > > > > > Configure 2.25 side not to run the reaper
> > > > > > Send message to 2.25 side
> > > > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > > > >
> > > > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > > > >
> > > > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > Yeah.. something like that... not necessarily in there though. but a
> > > > > > similar test.
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > > > >
> > > > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > > > >
> > > > > > > Stephen E. Baker
> > > > > > >
> > > > > > >
> > > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > In theory it should work.
> > > > > > >
> > > > > > >
> > > > > > > Only change that might break compatibility is
> > > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > > > >
> > > > > > >
> > > > > > > I tried to not break compatibility, but I just realized we should add
> > > > > > > a test to validate compatibility between mirrors.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > so, I will say it should be compatible, but I would test it before
> > > > > > > doing it in the real system.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > if you are willing to contribute to a compatibility test :)
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > > >
> > > > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > > > >
> > > > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > > > >
> > > > > > > > Stephen E Baker
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Clebert Suconic
> > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Robbie Gemmell <ro...@gmail.com>.
That helper command doesnt exist yet as Clebert said since the idea
only came from discussing something else the other day, but the
pre-existing logging related changes coming for 2.27.0 are covered in
the docs along with diffs of the script changes from 2.26.0, you can
see the current source version of those docs at
https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/versions.md#2270
. They will be updated to reference the helper once it exits.
On Thu, 13 Oct 2022 at 18:52, Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
>
> Diffing my two instances I see:
> # Set Defaults Properties
> ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
>
>
> # finding the Log Manager
> LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
>
> if [ -z "$LOG_MANAGER" ] ; then
> # this is the one found when the server was created
> LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> fi
>
> WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> if [ -z "$WILDFLY_COMMON" ] ; then
> # this is the one found when the server was created
> WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> fi
>
> -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
>
> and
>
> -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
>
> all have to go.
>
> I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
>
> Along with the new log4j.properties you mentioned
>
> Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Thursday, October 13, 2022 at 12:49 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> I think the record would pile up unacked at the source mirror.
>
>
> and @Stephen baker: sorry about my mistake on this fix...
>
>
> Why would the upgrade be difficult on 2.27? it's just adding a
> log4j2.properties.. everything else should be the same.
>
>
> You should probably bring a patched version yourself until we can make
> a release? I'm thinking we should make a release next week.
>
> On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
> >
> > I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
> >
> > What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
> >
> >
> >
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Wednesday, October 12, 2022 at 9:54 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > Notice that main is now using SLF4j / log4j... (in case you manually
> > upgrade to a snapshot)
> >
> >
> > We are still working the details for an upgrade.
> >
> >
> > if you need to patch your 2.25.0 it's a straight change to make there
> >
> > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > <cl...@gmail.com> wrote:
> > >
> > > I don't know how I would test it yet. It's fairly late in the night
> > > for me.. I will think about it tomorrow.
> > >
> > >
> > > but here is a tentative fix:
> > >
> > > https://github.com/apache/activemq-artemis/pull/4256
> > >
> > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > >
> > > > Let’s take further discussion there?
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > is this the actual trace? or you cut some to post here?
> > > >
> > > >
> > > > Just puzzled by skipDelivery calling performAck..
> > > >
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > >
> > > >
> > > > can you post the full stack if this is not it?
> > > >
> > > >
> > > > it definitely needs fixing... I'm investigating it.
> > > >
> > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > > >
> > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > > >
> > > > >
> > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > > >
> > > > > Configure 2.25 side not to run the reaper
> > > > > Send message to 2.25 side
> > > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > > >
> > > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > > >
> > > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > Yeah.. something like that... not necessarily in there though. but a
> > > > > similar test.
> > > > >
> > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > > >
> > > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > > >
> > > > > > Stephen E. Baker
> > > > > >
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > In theory it should work.
> > > > > >
> > > > > >
> > > > > > Only change that might break compatibility is
> > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > > >
> > > > > >
> > > > > > I tried to not break compatibility, but I just realized we should add
> > > > > > a test to validate compatibility between mirrors.
> > > > > >
> > > > > >
> > > > > >
> > > > > > so, I will say it should be compatible, but I would test it before
> > > > > > doing it in the real system.
> > > > > >
> > > > > >
> > > > > >
> > > > > > if you are willing to contribute to a compatibility test :)
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > > >
> > > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > > >
> > > > > > > Stephen E Baker
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
On Thu, Oct 13, 2022 at 2:41 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Fancy, that upgrade tool sounds great – much better than the pile of sed
> scripts I usually write for migrations.
>
> Is there anywhere I can read more about it yet?
Not yet.
I had exchanged some ideas with some of my co workers where I work at.
Still an idea but it’s firm on what I have to do.
I will update a PR I have about config.
>
> Thanks for the quick patch!
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Thursday, October 13, 2022 at 2:03 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> you have to replace the script...
>
>
> We will have by next week, a tool where you would
>
>
> /path/to/newVersion/bin/artemis upgrade /path/to/old/instance
>
>
> We would replace the artemis script, and the logging configuration.
>
>
> On Thu, Oct 13, 2022 at 1:52 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Because bin/artemis includes references to the jboss logmanager causing
> artemis to fail on startup
> >
> > Diffing my two instances I see:
> > # Set Defaults Properties
> > ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> > ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
> >
> >
> > # finding the Log Manager
> > LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
> >
> > if [ -z "$LOG_MANAGER" ] ; then
> > # this is the one found when the server was created
> > LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> > fi
> >
> > WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> > if [ -z "$WILDFLY_COMMON" ] ; then
> > # this is the one found when the server was created
> > WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> > fi
> >
> > -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
> >
> > and
> >
> > -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> > -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
> >
> > all have to go.
> >
> > I also see a change in the schema for bootstrap.xml (binding child of
> web), and the browse permission added to management.xml
> >
> > Along with the new log4j.properties you mentioned
> >
> > Some of those changes might be earlier if they’re backwards compatible,
> I’ve carried forward this configuration for awhile, but in particular the
> shell script changes appear to be manditory.
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Thursday, October 13, 2022 at 12:49 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > I think the record would pile up unacked at the source mirror.
> >
> >
> > and @Stephen baker: sorry about my mistake on this fix...
> >
> >
> > Why would the upgrade be difficult on 2.27? it's just adding a
> > log4j2.properties.. everything else should be the same.
> >
> >
> > You should probably bring a patched version yourself until we can make
> > a release? I'm thinking we should make a release next week.
> >
> > On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Your patch does resolve the error. Artemis 2.27 looks like it will be
> a difficult upgrade, I ended up making a new instance and merging config
> over.
> > >
> > > I have pasted a trace in
> https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > What is the impact of this issue? I’m trying to decide whether to
> advise our IT team to continue with the planned upgrade or hold off until
> 2.27. We will definitely encounter this condition in production. From a
> surface reading, possibly a resource leak?
> > >
> > >
> > >
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:54 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > Notice that main is now using SLF4j / log4j... (in case you manually
> > > upgrade to a snapshot)
> > >
> > >
> > > We are still working the details for an upgrade.
> > >
> > >
> > > if you need to patch your 2.25.0 it's a straight change to make there
> > >
> > > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > > <cl...@gmail.com> wrote:
> > > >
> > > > I don't know how I would test it yet. It's fairly late in the night
> > > > for me.. I will think about it tomorrow.
> > > >
> > > >
> > > > but here is a tentative fix:
> > > >
> > > > https://github.com/apache/activemq-artemis/pull/4256
> > > >
> > > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > That’s the full output with regular logging levels. I can
> reproduce at will so I have enabled trace level logging and pasted the
> result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > > >
> > > > > Let’s take further discussion there?
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > is this the actual trace? or you cut some to post here?
> > > > >
> > > > >
> > > > > Just puzzled by skipDelivery calling performAck..
> > > > >
> > > > > artemis-test-artemis-1-m-1 | at
> > > > >
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at
> > > > >
> org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > > [artemis-server-2.25.0.jar:2.25.0]
> > > > >
> > > > >
> > > > > can you post the full stack if this is not it?
> > > > >
> > > > >
> > > > > it definitely needs fixing... I'm investigating it.
> > > > >
> > > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Having updated both sides to 2.25 I’m seeing this error in the
> logs, is it a concern that warrants further investigation?
> > > > > >
> > > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR
> [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver:
> java.lang.IllegalStateException: this method requires to be called within
> the handler, use the executor
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250)
> [artemis-server-2.25.0.jar:2.25.0]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
> [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
> [artemis-commons-2.25.0.jar:]
> > > > > > artemis-test-artemis-1-m-1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> [java.base:]
> > > > > > artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [artemis-commons-2.25.0.jar:]
> > > > > >
> > > > > >
> > > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > I set up some docker images in this configuration as a
> preliminary step. One oddity:
> > > > > >
> > > > > > Configure 2.25 side not to run the reaper
> > > > > > Send message to 2.25 side
> > > > > > Observe that after expiry the message shows up in the expiry
> queue on the 2.20 side, but not on the 2.25 side, the message is removed
> from the original queue on both sides.
> > > > > >
> > > > > > If the message is originally sent to the 2.20 side it shows up
> in both queues as expected.
> > > > > >
> > > > > > There’s probably a reason for it, but I didn’t expect this
> change. I thought that we would continue to see the old bugs until both
> sides were updated.
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > Yeah.. something like that... not necessarily in there though.
> but a
> > > > > > similar test.
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > Ok, I agree based on a cursory reading of that patch. The
> extra ackReason defaults to normal in one direction and isn’t read in the
> other direction. Killed, replaced, and expired being interpreted as normal
> just means that the 2.20 bugs will persist until both sides are updated.
> > > > > > >
> > > > > > > I’ll test it out with different version docker containers. I
> suppose as far as writing tests you mean something like the
> MultiVersionReplicaTest.
> > > > > > >
> > > > > > > Stephen E. Baker
> > > > > > >
> > > > > > >
> > > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > > In theory it should work.
> > > > > > >
> > > > > > >
> > > > > > > Only change that might break compatibility is
> > > > > > >
> https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > > determine target operations and fixing Delivering statistics
> on Mirror
> > > > > > >
> > > > > > >
> > > > > > > I tried to not break compatibility, but I just realized we
> should add
> > > > > > > a test to validate compatibility between mirrors.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > so, I will say it should be compatible, but I would test it
> before
> > > > > > > doing it in the real system.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > if you are willing to contribute to a compatibility test :)
> > > > > > >
> > > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > > >
> > > > > > > > We are planning our production upgrade from 2.20 to 2.25.
> These upgrades involve a loss of service in the window between stopping the
> live and when the backup instance becomes ready to process messages.
> > > > > > > >
> > > > > > > > I was wondering if the mirror protocol is expected to be
> compatible between those versions. If so we could upgrade our cold site,
> and then wait for a planned failover to avoid any additional down time. I
> know that quite a bit of work was done by Clebert in 2.24 so I was hoping
> he could weigh in.
> > > > > > > >
> > > > > > > > Stephen E Baker
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Clebert Suconic
> > > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile
> Safety. Do not click links or open attachments unless you recognize the
> sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Fancy, that upgrade tool sounds great – much better than the pile of sed scripts I usually write for migrations.
Is there anywhere I can read more about it yet?
Thanks for the quick patch!
From: Clebert Suconic <cl...@gmail.com>
Date: Thursday, October 13, 2022 at 2:03 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
you have to replace the script...
We will have by next week, a tool where you would
/path/to/newVersion/bin/artemis upgrade /path/to/old/instance
We would replace the artemis script, and the logging configuration.
On Thu, Oct 13, 2022 at 1:52 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
>
> Diffing my two instances I see:
> # Set Defaults Properties
> ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
>
>
> # finding the Log Manager
> LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
>
> if [ -z "$LOG_MANAGER" ] ; then
> # this is the one found when the server was created
> LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> fi
>
> WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> if [ -z "$WILDFLY_COMMON" ] ; then
> # this is the one found when the server was created
> WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> fi
>
> -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
>
> and
>
> -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
>
> all have to go.
>
> I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
>
> Along with the new log4j.properties you mentioned
>
> Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Thursday, October 13, 2022 at 12:49 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> I think the record would pile up unacked at the source mirror.
>
>
> and @Stephen baker: sorry about my mistake on this fix...
>
>
> Why would the upgrade be difficult on 2.27? it's just adding a
> log4j2.properties.. everything else should be the same.
>
>
> You should probably bring a patched version yourself until we can make
> a release? I'm thinking we should make a release next week.
>
> On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
> >
> > I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
> >
> > What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
> >
> >
> >
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Wednesday, October 12, 2022 at 9:54 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > Notice that main is now using SLF4j / log4j... (in case you manually
> > upgrade to a snapshot)
> >
> >
> > We are still working the details for an upgrade.
> >
> >
> > if you need to patch your 2.25.0 it's a straight change to make there
> >
> > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > <cl...@gmail.com> wrote:
> > >
> > > I don't know how I would test it yet. It's fairly late in the night
> > > for me.. I will think about it tomorrow.
> > >
> > >
> > > but here is a tentative fix:
> > >
> > > https://github.com/apache/activemq-artemis/pull/4256
> > >
> > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > >
> > > > Let’s take further discussion there?
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > is this the actual trace? or you cut some to post here?
> > > >
> > > >
> > > > Just puzzled by skipDelivery calling performAck..
> > > >
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > >
> > > >
> > > > can you post the full stack if this is not it?
> > > >
> > > >
> > > > it definitely needs fixing... I'm investigating it.
> > > >
> > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > > >
> > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > > >
> > > > >
> > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > > >
> > > > > Configure 2.25 side not to run the reaper
> > > > > Send message to 2.25 side
> > > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > > >
> > > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > > >
> > > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > Yeah.. something like that... not necessarily in there though. but a
> > > > > similar test.
> > > > >
> > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > > >
> > > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > > >
> > > > > > Stephen E. Baker
> > > > > >
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > In theory it should work.
> > > > > >
> > > > > >
> > > > > > Only change that might break compatibility is
> > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > > >
> > > > > >
> > > > > > I tried to not break compatibility, but I just realized we should add
> > > > > > a test to validate compatibility between mirrors.
> > > > > >
> > > > > >
> > > > > >
> > > > > > so, I will say it should be compatible, but I would test it before
> > > > > > doing it in the real system.
> > > > > >
> > > > > >
> > > > > >
> > > > > > if you are willing to contribute to a compatibility test :)
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > > >
> > > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > > >
> > > > > > > Stephen E Baker
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
you have to replace the script...
We will have by next week, a tool where you would
/path/to/newVersion/bin/artemis upgrade /path/to/old/instance
We would replace the artemis script, and the logging configuration.
On Thu, Oct 13, 2022 at 1:52 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
>
> Diffing my two instances I see:
> # Set Defaults Properties
> ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
> ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
>
>
> # finding the Log Manager
> LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
>
> if [ -z "$LOG_MANAGER" ] ; then
> # this is the one found when the server was created
> LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
> fi
>
> WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
> if [ -z "$WILDFLY_COMMON" ] ; then
> # this is the one found when the server was created
> WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
> fi
>
> -Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
>
> and
>
> -Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
> -Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
>
> all have to go.
>
> I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
>
> Along with the new log4j.properties you mentioned
>
> Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Thursday, October 13, 2022 at 12:49 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> I think the record would pile up unacked at the source mirror.
>
>
> and @Stephen baker: sorry about my mistake on this fix...
>
>
> Why would the upgrade be difficult on 2.27? it's just adding a
> log4j2.properties.. everything else should be the same.
>
>
> You should probably bring a patched version yourself until we can make
> a release? I'm thinking we should make a release next week.
>
> On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
> >
> > I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
> >
> > What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
> >
> >
> >
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Wednesday, October 12, 2022 at 9:54 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > Notice that main is now using SLF4j / log4j... (in case you manually
> > upgrade to a snapshot)
> >
> >
> > We are still working the details for an upgrade.
> >
> >
> > if you need to patch your 2.25.0 it's a straight change to make there
> >
> > On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> > <cl...@gmail.com> wrote:
> > >
> > > I don't know how I would test it yet. It's fairly late in the night
> > > for me.. I will think about it tomorrow.
> > >
> > >
> > > but here is a tentative fix:
> > >
> > > https://github.com/apache/activemq-artemis/pull/4256
> > >
> > > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > > >
> > > > Let’s take further discussion there?
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > is this the actual trace? or you cut some to post here?
> > > >
> > > >
> > > > Just puzzled by skipDelivery calling performAck..
> > > >
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at
> > > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > > [artemis-server-2.25.0.jar:2.25.0]
> > > >
> > > >
> > > > can you post the full stack if this is not it?
> > > >
> > > >
> > > > it definitely needs fixing... I'm investigating it.
> > > >
> > > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > > >
> > > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > > >
> > > > >
> > > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > > >
> > > > > Configure 2.25 side not to run the reaper
> > > > > Send message to 2.25 side
> > > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > > >
> > > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > > >
> > > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > Yeah.. something like that... not necessarily in there though. but a
> > > > > similar test.
> > > > >
> > > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > > >
> > > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > > >
> > > > > > Stephen E. Baker
> > > > > >
> > > > > >
> > > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > > Subject: Re: Mirror compatibility across versions
> > > > > > In theory it should work.
> > > > > >
> > > > > >
> > > > > > Only change that might break compatibility is
> > > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > > >
> > > > > >
> > > > > > I tried to not break compatibility, but I just realized we should add
> > > > > > a test to validate compatibility between mirrors.
> > > > > >
> > > > > >
> > > > > >
> > > > > > so, I will say it should be compatible, but I would test it before
> > > > > > doing it in the real system.
> > > > > >
> > > > > >
> > > > > >
> > > > > > if you are willing to contribute to a compatibility test :)
> > > > > >
> > > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > > >
> > > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > > >
> > > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > > >
> > > > > > > Stephen E Baker
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Clebert Suconic
> > > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Because bin/artemis includes references to the jboss logmanager causing artemis to fail on startup
Diffing my two instances I see:
# Set Defaults Properties
ARTEMIS_LOGGING_CONF="$ARTEMIS_INSTANCE_ETC_URI/logging.properties"
ARTEMIS_LOG_MANAGER=org.jboss.logmanager.LogManager
# finding the Log Manager
LOG_MANAGER=`ls $ARTEMIS_HOME/lib/jboss-logmanager*jar 2>/dev/null`
if [ -z "$LOG_MANAGER" ] ; then
# this is the one found when the server was created
LOG_MANAGER="$ARTEMIS_HOME/lib/jboss-logmanager-2.1.10.Final.jar"
fi
WILDFLY_COMMON=`ls $ARTEMIS_HOME/lib/wildfly-common*jar 2>/dev/null`
if [ -z "$WILDFLY_COMMON" ] ; then
# this is the one found when the server was created
WILDFLY_COMMON="$ARTEMIS_HOME/lib/wildfly-common-1.5.2.Final.jar"
fi
-Xbootclasspath/a:"$LOG_MANAGER:$WILDFLY_COMMON" \
and
-Djava.util.logging.manager="$ARTEMIS_LOG_MANAGER" \
-Dlogging.configuration="$ARTEMIS_LOGGING_CONF" \
all have to go.
I also see a change in the schema for bootstrap.xml (binding child of web), and the browse permission added to management.xml
Along with the new log4j.properties you mentioned
Some of those changes might be earlier if they’re backwards compatible, I’ve carried forward this configuration for awhile, but in particular the shell script changes appear to be manditory.
From: Clebert Suconic <cl...@gmail.com>
Date: Thursday, October 13, 2022 at 12:49 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
I think the record would pile up unacked at the source mirror.
and @Stephen baker: sorry about my mistake on this fix...
Why would the upgrade be difficult on 2.27? it's just adding a
log4j2.properties.. everything else should be the same.
You should probably bring a patched version yourself until we can make
a release? I'm thinking we should make a release next week.
On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
>
> I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
>
> What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
>
>
>
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Wednesday, October 12, 2022 at 9:54 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> Notice that main is now using SLF4j / log4j... (in case you manually
> upgrade to a snapshot)
>
>
> We are still working the details for an upgrade.
>
>
> if you need to patch your 2.25.0 it's a straight change to make there
>
> On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > I don't know how I would test it yet. It's fairly late in the night
> > for me.. I will think about it tomorrow.
> >
> >
> > but here is a tentative fix:
> >
> > https://github.com/apache/activemq-artemis/pull/4256
> >
> > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > Let’s take further discussion there?
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > is this the actual trace? or you cut some to post here?
> > >
> > >
> > > Just puzzled by skipDelivery calling performAck..
> > >
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > [artemis-server-2.25.0.jar:2.25.0]
> > >
> > >
> > > can you post the full stack if this is not it?
> > >
> > >
> > > it definitely needs fixing... I'm investigating it.
> > >
> > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > >
> > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > >
> > > >
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > >
> > > > Configure 2.25 side not to run the reaper
> > > > Send message to 2.25 side
> > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > >
> > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > >
> > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > Yeah.. something like that... not necessarily in there though. but a
> > > > similar test.
> > > >
> > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > >
> > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > >
> > > > > Stephen E. Baker
> > > > >
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > In theory it should work.
> > > > >
> > > > >
> > > > > Only change that might break compatibility is
> > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > >
> > > > >
> > > > > I tried to not break compatibility, but I just realized we should add
> > > > > a test to validate compatibility between mirrors.
> > > > >
> > > > >
> > > > >
> > > > > so, I will say it should be compatible, but I would test it before
> > > > > doing it in the real system.
> > > > >
> > > > >
> > > > >
> > > > > if you are willing to contribute to a compatibility test :)
> > > > >
> > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > >
> > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > >
> > > > > > Stephen E Baker
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
I think the record would pile up unacked at the source mirror.
and @Stephen baker: sorry about my mistake on this fix...
Why would the upgrade be difficult on 2.27? it's just adding a
log4j2.properties.. everything else should be the same.
You should probably bring a patched version yourself until we can make
a release? I'm thinking we should make a release next week.
On Thu, Oct 13, 2022 at 10:27 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
>
> I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
>
> What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
>
>
>
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Wednesday, October 12, 2022 at 9:54 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> Notice that main is now using SLF4j / log4j... (in case you manually
> upgrade to a snapshot)
>
>
> We are still working the details for an upgrade.
>
>
> if you need to patch your 2.25.0 it's a straight change to make there
>
> On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > I don't know how I would test it yet. It's fairly late in the night
> > for me.. I will think about it tomorrow.
> >
> >
> > but here is a tentative fix:
> >
> > https://github.com/apache/activemq-artemis/pull/4256
> >
> > On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> > >
> > > Let’s take further discussion there?
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Wednesday, October 12, 2022 at 9:10 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > is this the actual trace? or you cut some to post here?
> > >
> > >
> > > Just puzzled by skipDelivery calling performAck..
> > >
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at
> > > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > > [artemis-server-2.25.0.jar:2.25.0]
> > >
> > >
> > > can you post the full stack if this is not it?
> > >
> > >
> > > it definitely needs fixing... I'm investigating it.
> > >
> > > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > > >
> > > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > > >
> > > >
> > > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > > >
> > > > Configure 2.25 side not to run the reaper
> > > > Send message to 2.25 side
> > > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > > >
> > > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > > >
> > > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > Yeah.. something like that... not necessarily in there though. but a
> > > > similar test.
> > > >
> > > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > > >
> > > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > > >
> > > > > Stephen E. Baker
> > > > >
> > > > >
> > > > > From: Clebert Suconic <cl...@gmail.com>
> > > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > > Subject: Re: Mirror compatibility across versions
> > > > > In theory it should work.
> > > > >
> > > > >
> > > > > Only change that might break compatibility is
> > > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > > determine target operations and fixing Delivering statistics on Mirror
> > > > >
> > > > >
> > > > > I tried to not break compatibility, but I just realized we should add
> > > > > a test to validate compatibility between mirrors.
> > > > >
> > > > >
> > > > >
> > > > > so, I will say it should be compatible, but I would test it before
> > > > > doing it in the real system.
> > > > >
> > > > >
> > > > >
> > > > > if you are willing to contribute to a compatibility test :)
> > > > >
> > > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > > <st...@rmssoftwareinc.com> wrote:
> > > > > >
> > > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > > >
> > > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > > >
> > > > > > Stephen E Baker
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Clebert Suconic
> > > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Your patch does resolve the error. Artemis 2.27 looks like it will be a difficult upgrade, I ended up making a new instance and merging config over.
I have pasted a trace in https://issues.apache.org/jira/browse/ARTEMIS-4045
What is the impact of this issue? I’m trying to decide whether to advise our IT team to continue with the planned upgrade or hold off until 2.27. We will definitely encounter this condition in production. From a surface reading, possibly a resource leak?
From: Clebert Suconic <cl...@gmail.com>
Date: Wednesday, October 12, 2022 at 9:54 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
Notice that main is now using SLF4j / log4j... (in case you manually
upgrade to a snapshot)
We are still working the details for an upgrade.
if you need to patch your 2.25.0 it's a straight change to make there
On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> I don't know how I would test it yet. It's fairly late in the night
> for me.. I will think about it tomorrow.
>
>
> but here is a tentative fix:
>
> https://github.com/apache/activemq-artemis/pull/4256
>
> On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> >
> > Let’s take further discussion there?
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Wednesday, October 12, 2022 at 9:10 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > is this the actual trace? or you cut some to post here?
> >
> >
> > Just puzzled by skipDelivery calling performAck..
> >
> > artemis-test-artemis-1-m-1 | at
> > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at
> > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > [artemis-server-2.25.0.jar:2.25.0]
> >
> >
> > can you post the full stack if this is not it?
> >
> >
> > it definitely needs fixing... I'm investigating it.
> >
> > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > >
> > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > >
> > >
> > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > >
> > > Configure 2.25 side not to run the reaper
> > > Send message to 2.25 side
> > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > >
> > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > >
> > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > Yeah.. something like that... not necessarily in there though. but a
> > > similar test.
> > >
> > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > >
> > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > >
> > > > Stephen E. Baker
> > > >
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > In theory it should work.
> > > >
> > > >
> > > > Only change that might break compatibility is
> > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > determine target operations and fixing Delivering statistics on Mirror
> > > >
> > > >
> > > > I tried to not break compatibility, but I just realized we should add
> > > > a test to validate compatibility between mirrors.
> > > >
> > > >
> > > >
> > > > so, I will say it should be compatible, but I would test it before
> > > > doing it in the real system.
> > > >
> > > >
> > > >
> > > > if you are willing to contribute to a compatibility test :)
> > > >
> > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > >
> > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > >
> > > > > Stephen E Baker
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
Notice that main is now using SLF4j / log4j... (in case you manually
upgrade to a snapshot)
We are still working the details for an upgrade.
if you need to patch your 2.25.0 it's a straight change to make there
On Wed, Oct 12, 2022 at 9:52 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> I don't know how I would test it yet. It's fairly late in the night
> for me.. I will think about it tomorrow.
>
>
> but here is a tentative fix:
>
> https://github.com/apache/activemq-artemis/pull/4256
>
> On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
> >
> > Let’s take further discussion there?
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Wednesday, October 12, 2022 at 9:10 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > is this the actual trace? or you cut some to post here?
> >
> >
> > Just puzzled by skipDelivery calling performAck..
> >
> > artemis-test-artemis-1-m-1 | at
> > org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> > [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at
> > org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> > [artemis-server-2.25.0.jar:2.25.0]
> >
> >
> > can you post the full stack if this is not it?
> >
> >
> > it definitely needs fixing... I'm investigating it.
> >
> > On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> > >
> > > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> > >
> > >
> > > From: Stephen Baker <st...@rmssoftwareinc.com>
> > > Date: Wednesday, October 12, 2022 at 4:43 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > I set up some docker images in this configuration as a preliminary step. One oddity:
> > >
> > > Configure 2.25 side not to run the reaper
> > > Send message to 2.25 side
> > > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> > >
> > > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> > >
> > > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Tuesday, October 11, 2022 at 3:24 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > Yeah.. something like that... not necessarily in there though. but a
> > > similar test.
> > >
> > > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > > >
> > > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > > >
> > > > Stephen E. Baker
> > > >
> > > >
> > > > From: Clebert Suconic <cl...@gmail.com>
> > > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > > Subject: Re: Mirror compatibility across versions
> > > > In theory it should work.
> > > >
> > > >
> > > > Only change that might break compatibility is
> > > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > > determine target operations and fixing Delivering statistics on Mirror
> > > >
> > > >
> > > > I tried to not break compatibility, but I just realized we should add
> > > > a test to validate compatibility between mirrors.
> > > >
> > > >
> > > >
> > > > so, I will say it should be compatible, but I would test it before
> > > > doing it in the real system.
> > > >
> > > >
> > > >
> > > > if you are willing to contribute to a compatibility test :)
> > > >
> > > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > > <st...@rmssoftwareinc.com> wrote:
> > > > >
> > > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > > >
> > > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > > >
> > > > > Stephen E Baker
> > > >
> > > >
> > > >
> > > > --
> > > > Clebert Suconic
> > > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
I don't know how I would test it yet. It's fairly late in the night
for me.. I will think about it tomorrow.
but here is a tentative fix:
https://github.com/apache/activemq-artemis/pull/4256
On Wed, Oct 12, 2022 at 9:30 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
>
> Let’s take further discussion there?
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Wednesday, October 12, 2022 at 9:10 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> is this the actual trace? or you cut some to post here?
>
>
> Just puzzled by skipDelivery calling performAck..
>
> artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
> [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
> [artemis-server-2.25.0.jar:2.25.0]
>
>
> can you post the full stack if this is not it?
>
>
> it definitely needs fixing... I'm investigating it.
>
> On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
> >
> > artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> > artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> > artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
> >
> >
> > From: Stephen Baker <st...@rmssoftwareinc.com>
> > Date: Wednesday, October 12, 2022 at 4:43 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > I set up some docker images in this configuration as a preliminary step. One oddity:
> >
> > Configure 2.25 side not to run the reaper
> > Send message to 2.25 side
> > Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
> >
> > If the message is originally sent to the 2.20 side it shows up in both queues as expected.
> >
> > There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Tuesday, October 11, 2022 at 3:24 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > Yeah.. something like that... not necessarily in there though. but a
> > similar test.
> >
> > On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> > >
> > > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> > >
> > > Stephen E. Baker
> > >
> > >
> > > From: Clebert Suconic <cl...@gmail.com>
> > > Date: Tuesday, October 11, 2022 at 12:59 PM
> > > To: users@activemq.apache.org <us...@activemq.apache.org>
> > > Subject: Re: Mirror compatibility across versions
> > > In theory it should work.
> > >
> > >
> > > Only change that might break compatibility is
> > > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > > determine target operations and fixing Delivering statistics on Mirror
> > >
> > >
> > > I tried to not break compatibility, but I just realized we should add
> > > a test to validate compatibility between mirrors.
> > >
> > >
> > >
> > > so, I will say it should be compatible, but I would test it before
> > > doing it in the real system.
> > >
> > >
> > >
> > > if you are willing to contribute to a compatibility test :)
> > >
> > > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > > <st...@rmssoftwareinc.com> wrote:
> > > >
> > > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > > >
> > > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > > >
> > > > Stephen E Baker
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
That’s the full output with regular logging levels. I can reproduce at will so I have enabled trace level logging and pasted the result in https://issues.apache.org/jira/browse/ARTEMIS-4045
Let’s take further discussion there?
From: Clebert Suconic <cl...@gmail.com>
Date: Wednesday, October 12, 2022 at 9:10 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
is this the actual trace? or you cut some to post here?
Just puzzled by skipDelivery calling performAck..
artemis-test-artemis-1-m-1 | at
org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
[artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at
org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
[artemis-server-2.25.0.jar:2.25.0]
can you post the full stack if this is not it?
it definitely needs fixing... I'm investigating it.
On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
>
> artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
>
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Wednesday, October 12, 2022 at 4:43 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> I set up some docker images in this configuration as a preliminary step. One oddity:
>
> Configure 2.25 side not to run the reaper
> Send message to 2.25 side
> Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
>
> If the message is originally sent to the 2.20 side it shows up in both queues as expected.
>
> There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Tuesday, October 11, 2022 at 3:24 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> Yeah.. something like that... not necessarily in there though. but a
> similar test.
>
> On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> >
> > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> >
> > Stephen E. Baker
> >
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Tuesday, October 11, 2022 at 12:59 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > In theory it should work.
> >
> >
> > Only change that might break compatibility is
> > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > determine target operations and fixing Delivering statistics on Mirror
> >
> >
> > I tried to not break compatibility, but I just realized we should add
> > a test to validate compatibility between mirrors.
> >
> >
> >
> > so, I will say it should be compatible, but I would test it before
> > doing it in the real system.
> >
> >
> >
> > if you are willing to contribute to a compatibility test :)
> >
> > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > >
> > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > >
> > > Stephen E Baker
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
is this the actual trace? or you cut some to post here?
Just puzzled by skipDelivery calling performAck..
artemis-test-artemis-1-m-1 | at
org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377)
[artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at
org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203)
[artemis-server-2.25.0.jar:2.25.0]
can you post the full stack if this is not it?
it definitely needs fixing... I'm investigating it.
On Wed, Oct 12, 2022 at 6:05 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
>
> artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
> artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
> artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
> artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
>
>
> From: Stephen Baker <st...@rmssoftwareinc.com>
> Date: Wednesday, October 12, 2022 at 4:43 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> I set up some docker images in this configuration as a preliminary step. One oddity:
>
> Configure 2.25 side not to run the reaper
> Send message to 2.25 side
> Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
>
> If the message is originally sent to the 2.20 side it shows up in both queues as expected.
>
> There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Tuesday, October 11, 2022 at 3:24 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> Yeah.. something like that... not necessarily in there though. but a
> similar test.
>
> On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
> >
> > I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
> >
> > Stephen E. Baker
> >
> >
> > From: Clebert Suconic <cl...@gmail.com>
> > Date: Tuesday, October 11, 2022 at 12:59 PM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Mirror compatibility across versions
> > In theory it should work.
> >
> >
> > Only change that might break compatibility is
> > https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> > which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> > determine target operations and fixing Delivering statistics on Mirror
> >
> >
> > I tried to not break compatibility, but I just realized we should add
> > a test to validate compatibility between mirrors.
> >
> >
> >
> > so, I will say it should be compatible, but I would test it before
> > doing it in the real system.
> >
> >
> >
> > if you are willing to contribute to a compatibility test :)
> >
> > On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> > >
> > > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> > >
> > > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> > >
> > > Stephen E Baker
> >
> >
> >
> > --
> > Clebert Suconic
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Having updated both sides to 2.25 I’m seeing this error in the logs, is it a concern that warrants further investigation?
artemis-test-artemis-1-m-1 | 2022-10-12 22:01:43,632 ERROR [org.apache.activemq.artemis.core.server] AMQ224041: Failed to deliver: java.lang.IllegalStateException: this method requires to be called within the handler, use the executor
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.requireHandler(ProtonHandler.java:210) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.requireInHandler(AMQPConnectionContext.java:197) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.proton.ProtonAbstractReceiver.settle(ProtonAbstractReceiver.java:185) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget$ACKMessageOperation.run(AMQPMirrorControllerTarget.java:125) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.performAck(AMQPMirrorControllerTarget.java:388) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget.lambda$performAck$2(AMQPMirrorControllerTarget.java:377) [artemis-amqp-protocol-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$2.skipDelivery(QueueImpl.java:1203) [artemis-server-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.doInternalPoll(QueueImpl.java:2932) [artemis-server-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:2991) [artemis-server-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:4250) [artemis-server-2.25.0.jar:2.25.0]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) [artemis-commons-2.25.0.jar:]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.25.0.jar:]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) [artemis-commons-2.25.0.jar:]
artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
artemis-test-artemis-1-m-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
artemis-test-artemis-1-m-1 | at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.25.0.jar:]
From: Stephen Baker <st...@rmssoftwareinc.com>
Date: Wednesday, October 12, 2022 at 4:43 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
I set up some docker images in this configuration as a preliminary step. One oddity:
Configure 2.25 side not to run the reaper
Send message to 2.25 side
Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
If the message is originally sent to the 2.20 side it shows up in both queues as expected.
There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
From: Clebert Suconic <cl...@gmail.com>
Date: Tuesday, October 11, 2022 at 3:24 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
Yeah.. something like that... not necessarily in there though. but a
similar test.
On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
>
> I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
>
> Stephen E. Baker
>
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Tuesday, October 11, 2022 at 12:59 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> In theory it should work.
>
>
> Only change that might break compatibility is
> https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> determine target operations and fixing Delivering statistics on Mirror
>
>
> I tried to not break compatibility, but I just realized we should add
> a test to validate compatibility between mirrors.
>
>
>
> so, I will say it should be compatible, but I would test it before
> doing it in the real system.
>
>
>
> if you are willing to contribute to a compatibility test :)
>
> On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> >
> > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> >
> > Stephen E Baker
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
I set up some docker images in this configuration as a preliminary step. One oddity:
Configure 2.25 side not to run the reaper
Send message to 2.25 side
Observe that after expiry the message shows up in the expiry queue on the 2.20 side, but not on the 2.25 side, the message is removed from the original queue on both sides.
If the message is originally sent to the 2.20 side it shows up in both queues as expected.
There’s probably a reason for it, but I didn’t expect this change. I thought that we would continue to see the old bugs until both sides were updated.
From: Clebert Suconic <cl...@gmail.com>
Date: Tuesday, October 11, 2022 at 3:24 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
Yeah.. something like that... not necessarily in there though. but a
similar test.
On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
>
> I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
>
> Stephen E. Baker
>
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Tuesday, October 11, 2022 at 12:59 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> In theory it should work.
>
>
> Only change that might break compatibility is
> https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> determine target operations and fixing Delivering statistics on Mirror
>
>
> I tried to not break compatibility, but I just realized we should add
> a test to validate compatibility between mirrors.
>
>
>
> so, I will say it should be compatible, but I would test it before
> doing it in the real system.
>
>
>
> if you are willing to contribute to a compatibility test :)
>
> On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> >
> > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> >
> > Stephen E Baker
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
Yeah.. something like that... not necessarily in there though. but a
similar test.
On Tue, Oct 11, 2022 at 1:44 PM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
>
> I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
>
> Stephen E. Baker
>
>
> From: Clebert Suconic <cl...@gmail.com>
> Date: Tuesday, October 11, 2022 at 12:59 PM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Mirror compatibility across versions
> In theory it should work.
>
>
> Only change that might break compatibility is
> https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
> which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
> determine target operations and fixing Delivering statistics on Mirror
>
>
> I tried to not break compatibility, but I just realized we should add
> a test to validate compatibility between mirrors.
>
>
>
> so, I will say it should be compatible, but I would test it before
> doing it in the real system.
>
>
>
> if you are willing to contribute to a compatibility test :)
>
> On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
> >
> > We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
> >
> > I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
> >
> > Stephen E Baker
>
>
>
> --
> Clebert Suconic
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Mirror compatibility across versions
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
Ok, I agree based on a cursory reading of that patch. The extra ackReason defaults to normal in one direction and isn’t read in the other direction. Killed, replaced, and expired being interpreted as normal just means that the 2.20 bugs will persist until both sides are updated.
I’ll test it out with different version docker containers. I suppose as far as writing tests you mean something like the MultiVersionReplicaTest.
Stephen E. Baker
From: Clebert Suconic <cl...@gmail.com>
Date: Tuesday, October 11, 2022 at 12:59 PM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Mirror compatibility across versions
In theory it should work.
Only change that might break compatibility is
https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
determine target operations and fixing Delivering statistics on Mirror
I tried to not break compatibility, but I just realized we should add
a test to validate compatibility between mirrors.
so, I will say it should be compatible, but I would test it before
doing it in the real system.
if you are willing to contribute to a compatibility test :)
On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
>
> I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
>
> Stephen E Baker
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Re: Mirror compatibility across versions
Posted by Clebert Suconic <cl...@gmail.com>.
In theory it should work.
Only change that might break compatibility is
https://github.com/apache/activemq-artemis/commit/68f6d8263d8c795722805f0e4d6939e7a8b9ed48
which is ARTEMIS-3743 / ARTEMIS-3766 Use ACKReason on Mirror to
determine target operations and fixing Delivering statistics on Mirror
I tried to not break compatibility, but I just realized we should add
a test to validate compatibility between mirrors.
so, I will say it should be compatible, but I would test it before
doing it in the real system.
if you are willing to contribute to a compatibility test :)
On Tue, Oct 11, 2022 at 10:06 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> We are planning our production upgrade from 2.20 to 2.25. These upgrades involve a loss of service in the window between stopping the live and when the backup instance becomes ready to process messages.
>
> I was wondering if the mirror protocol is expected to be compatible between those versions. If so we could upgrade our cold site, and then wait for a planned failover to avoid any additional down time. I know that quite a bit of work was done by Clebert in 2.24 so I was hoping he could weigh in.
>
> Stephen E Baker
--
Clebert Suconic