You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Justin Bertram <jb...@apache.org> on 2022/06/01 04:33:29 UTC
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
I sent a commit to update the docs in the repo, and I also updated the
website.
Out of curiosity, how is your MySQL replication configured? Are you using
the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
Justin
On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Understood thank you. We (the company I work for) are definitely getting
> more value out of the product than we are contributing, so that point is
> taken. The JDBC replication route was recommended by a consultant from
> Savoir as more established/reliable than mirroring when delivery guarantees
> are important, which is why we are pursuing it.
>
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Илья Грушевский <il...@gmail.com>.
Thanks Clebert, I appreciate it.
Hope more to come soon :)
> 1 июля 2022 г., в 01:19, Clebert Suconic <cl...@gmail.com> написал(а):
>
> @Iliya Grushevskiy thanks for the contributions... pretty impressive
> actually as you tweaked OperationContext and other IO operations..
> thanks a lot. I would love to see more contributions coming :)
>
> On Thu, Jun 30, 2022 at 3:47 PM Clebert Suconic
> <cl...@gmail.com> wrote:
>>
>> PR sent: https://github.com/apache/activemq-artemis/pull/4130
>>
>> On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
>> <cl...@gmail.com> wrote:
>>>
>>> Thanks for checking on that.
>>>
>>> I will take a look today.
>>>
>>> On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
>>>>
>>>> Hi, Clebert
>>>>
>>>> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
>>>> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
>>>>
>>>>
>>>> Regards
>>>> Iliya Grushevskiy
>>>>
>>>>
>>>>
>>>>
>>>>> 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>>>>>
>>>>> I have sent a PR for the mirror issues:
>>>>>
>>>>> https://github.com/apache/activemq-artemis/pull/4127
>>>>>
>>>>> On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
>>>>> <st...@rmssoftwareinc.com> wrote:
>>>>>>
>>>>>> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>>>>>> replication to a secondary server in the same datacenter and a similar
>>>>>> primary/secondary server to the standby datacenter.
>>>>>>
>>>>>> Theoretically the mirroring in Artemis should offer the same resiliency
>>>>>> (synced to the disk, possibly not replicated in the event of a hard failure);
>>>>>> But in practice mirroring is Artemis is relatively new and SQL replication
>>>>>> has existed for decades with plenty of DBAs who are well versed in mitigating
>>>>>> common problems.
>>>>>>
>>>>>> Some of the issues we have with mirroring that I can remember:
>>>>>> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>>>>>> (not just the expiry queue – looking at one of our production sets right now on
>>>>>> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>>>>>> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>>>>>> are really there but I hope to learn more when we complete our 2.22 update.)
>>>>>> * The stats are not tracked correctly so it’s hard to tell how out of sync
>>>>>> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>>>>>> * We use to end up with extremely large journals on the cold side that would
>>>>>> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>>>>>> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>>>>>> somewhat with aggressive TTL.
>>>>>> * More often than not when performing a graceful failover we need to restart Artemis
>>>>>> on the new live. Consumers connect but they don’t receive any messages.
>>>>>> * In some instances the mirror queues have not shown up in the Artemis console
>>>>>> but have functioned. No known steps to reproduce, in all cases eventually resolved
>>>>>> themselves.
>>>>>>
>>>>>> * Messages in the logs that indicate problems we don’t know the effect of like:
>>>>>>
>>>>>> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>>>>>>
>>>>>>
>>>>>> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>>>>>> It’s a critical and highly performant part of our infrastructure. I have personally been
>>>>>> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>>>>>>
>>>>>> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>>>>>> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>>>>>> responsible for.
>>>>>>
>>>>>> A bit more than I intended to write, but I hope this helps understand our where
>>>>>> we are and our motivations.
>>>>>>
>>>>>>
>>>>>> From: Justin Bertram <jb...@apache.org>
>>>>>> Date: Wednesday, June 1, 2022 at 12:34 AM
>>>>>> To: users@activemq.apache.org <us...@activemq.apache.org>
>>>>>> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>>>>>> I sent a commit to update the docs in the repo, and I also updated the
>>>>>> website.
>>>>>>
>>>>>> Out of curiosity, how is your MySQL replication configured? Are you using
>>>>>> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>>>>>>
>>>>>>
>>>>>> Justin
>>>>>>
>>>>>> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>>>>>> stephen.baker@rmssoftwareinc.com> wrote:
>>>>>>
>>>>>>> Understood thank you. We (the company I work for) are definitely getting
>>>>>>> more value out of the product than we are contributing, so that point is
>>>>>>> taken. The JDBC replication route was recommended by a consultant from
>>>>>>> Savoir as more established/reliable than mirroring when delivery guarantees
>>>>>>> are important, which is why we are pursuing it.
>>>>>>>
>>>>>>>
>>>>>> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Clebert Suconic
>>>>
>>> --
>>> Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
@Iliya Grushevskiy thanks for the contributions... pretty impressive
actually as you tweaked OperationContext and other IO operations..
thanks a lot. I would love to see more contributions coming :)
On Thu, Jun 30, 2022 at 3:47 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> PR sent: https://github.com/apache/activemq-artemis/pull/4130
>
> On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
> <cl...@gmail.com> wrote:
> >
> > Thanks for checking on that.
> >
> > I will take a look today.
> >
> > On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
> >>
> >> Hi, Clebert
> >>
> >> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
> >> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
> >>
> >>
> >> Regards
> >> Iliya Grushevskiy
> >>
> >>
> >>
> >>
> >> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
> >> >
> >> > I have sent a PR for the mirror issues:
> >> >
> >> > https://github.com/apache/activemq-artemis/pull/4127
> >> >
> >> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> >> > <st...@rmssoftwareinc.com> wrote:
> >> >>
> >> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> >> >> replication to a secondary server in the same datacenter and a similar
> >> >> primary/secondary server to the standby datacenter.
> >> >>
> >> >> Theoretically the mirroring in Artemis should offer the same resiliency
> >> >> (synced to the disk, possibly not replicated in the event of a hard failure);
> >> >> But in practice mirroring is Artemis is relatively new and SQL replication
> >> >> has existed for decades with plenty of DBAs who are well versed in mitigating
> >> >> common problems.
> >> >>
> >> >> Some of the issues we have with mirroring that I can remember:
> >> >> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> >> >> (not just the expiry queue – looking at one of our production sets right now on
> >> >> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
> >> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
> >> >> are really there but I hope to learn more when we complete our 2.22 update.)
> >> >> * The stats are not tracked correctly so it’s hard to tell how out of sync
> >> >> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> >> >> * We use to end up with extremely large journals on the cold side that would
> >> >> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
> >> >> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
> >> >> somewhat with aggressive TTL.
> >> >> * More often than not when performing a graceful failover we need to restart Artemis
> >> >> on the new live. Consumers connect but they don’t receive any messages.
> >> >> * In some instances the mirror queues have not shown up in the Artemis console
> >> >> but have functioned. No known steps to reproduce, in all cases eventually resolved
> >> >> themselves.
> >> >>
> >> >> * Messages in the logs that indicate problems we don’t know the effect of like:
> >> >>
> >> >> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >> >>
> >> >>
> >> >> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> >> >> It’s a critical and highly performant part of our infrastructure. I have personally been
> >> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
> >> >>
> >> >> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
> >> >> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
> >> >> responsible for.
> >> >>
> >> >> A bit more than I intended to write, but I hope this helps understand our where
> >> >> we are and our motivations.
> >> >>
> >> >>
> >> >> From: Justin Bertram <jb...@apache.org>
> >> >> Date: Wednesday, June 1, 2022 at 12:34 AM
> >> >> To: users@activemq.apache.org <us...@activemq.apache.org>
> >> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
> >> >> I sent a commit to update the docs in the repo, and I also updated the
> >> >> website.
> >> >>
> >> >> Out of curiosity, how is your MySQL replication configured? Are you using
> >> >> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
> >> >>
> >> >>
> >> >> Justin
> >> >>
> >> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> >> >> stephen.baker@rmssoftwareinc.com> wrote:
> >> >>
> >> >>> Understood thank you. We (the company I work for) are definitely getting
> >> >>> more value out of the product than we are contributing, so that point is
> >> >>> taken. The JDBC replication route was recommended by a consultant from
> >> >>> Savoir as more established/reliable than mirroring when delivery guarantees
> >> >>> are important, which is why we are pursuing it.
> >> >>>
> >> >>>
> >> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> >> >
> >> >
> >> >
> >> > --
> >> > Clebert Suconic
> >>
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
PR sent: https://github.com/apache/activemq-artemis/pull/4130
On Thu, Jun 30, 2022 at 9:33 AM Clebert Suconic
<cl...@gmail.com> wrote:
>
> Thanks for checking on that.
>
> I will take a look today.
>
> On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com> wrote:
>>
>> Hi, Clebert
>>
>> I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
>> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
>>
>>
>> Regards
>> Iliya Grushevskiy
>>
>>
>>
>>
>> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>> >
>> > I have sent a PR for the mirror issues:
>> >
>> > https://github.com/apache/activemq-artemis/pull/4127
>> >
>> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
>> > <st...@rmssoftwareinc.com> wrote:
>> >>
>> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>> >> replication to a secondary server in the same datacenter and a similar
>> >> primary/secondary server to the standby datacenter.
>> >>
>> >> Theoretically the mirroring in Artemis should offer the same resiliency
>> >> (synced to the disk, possibly not replicated in the event of a hard failure);
>> >> But in practice mirroring is Artemis is relatively new and SQL replication
>> >> has existed for decades with plenty of DBAs who are well versed in mitigating
>> >> common problems.
>> >>
>> >> Some of the issues we have with mirroring that I can remember:
>> >> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>> >> (not just the expiry queue – looking at one of our production sets right now on
>> >> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>> >> are really there but I hope to learn more when we complete our 2.22 update.)
>> >> * The stats are not tracked correctly so it’s hard to tell how out of sync
>> >> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>> >> * We use to end up with extremely large journals on the cold side that would
>> >> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>> >> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>> >> somewhat with aggressive TTL.
>> >> * More often than not when performing a graceful failover we need to restart Artemis
>> >> on the new live. Consumers connect but they don’t receive any messages.
>> >> * In some instances the mirror queues have not shown up in the Artemis console
>> >> but have functioned. No known steps to reproduce, in all cases eventually resolved
>> >> themselves.
>> >>
>> >> * Messages in the logs that indicate problems we don’t know the effect of like:
>> >>
>> >> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>> >>
>> >>
>> >> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>> >> It’s a critical and highly performant part of our infrastructure. I have personally been
>> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>> >>
>> >> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>> >> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>> >> responsible for.
>> >>
>> >> A bit more than I intended to write, but I hope this helps understand our where
>> >> we are and our motivations.
>> >>
>> >>
>> >> From: Justin Bertram <jb...@apache.org>
>> >> Date: Wednesday, June 1, 2022 at 12:34 AM
>> >> To: users@activemq.apache.org <us...@activemq.apache.org>
>> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>> >> I sent a commit to update the docs in the repo, and I also updated the
>> >> website.
>> >>
>> >> Out of curiosity, how is your MySQL replication configured? Are you using
>> >> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>> >>
>> >>
>> >> Justin
>> >>
>> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>> >> stephen.baker@rmssoftwareinc.com> wrote:
>> >>
>> >>> Understood thank you. We (the company I work for) are definitely getting
>> >>> more value out of the product than we are contributing, so that point is
>> >>> taken. The JDBC replication route was recommended by a consultant from
>> >>> Savoir as more established/reliable than mirroring when delivery guarantees
>> >>> are important, which is why we are pursuing it.
>> >>>
>> >>>
>> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>>
> --
> Clebert Suconic
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
Thanks for checking on that.
I will take a look today.
On Thu, Jun 30, 2022 at 4:39 AM Iliya Grushevskiy <il...@gmail.com>
wrote:
> Hi, Clebert
>
> I think there is another related issue:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
> I have sent a PR https://github.com/apache/activemq-artemis/pull/4066.
> It’s outdated now, but it contains test that I think is still relevant.
>
>
> Regards
> Iliya Grushevskiy
>
>
>
>
> > 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com>
> написал(а):
> >
> > I have sent a PR for the mirror issues:
> >
> > https://github.com/apache/activemq-artemis/pull/4127
> >
> > On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> > <st...@rmssoftwareinc.com> wrote:
> >>
> >> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> >> replication to a secondary server in the same datacenter and a similar
> >> primary/secondary server to the standby datacenter.
> >>
> >> Theoretically the mirroring in Artemis should offer the same resiliency
> >> (synced to the disk, possibly not replicated in the event of a hard
> failure);
> >> But in practice mirroring is Artemis is relatively new and SQL
> replication
> >> has existed for decades with plenty of DBAs who are well versed in
> mitigating
> >> common problems.
> >>
> >> Some of the issues we have with mirroring that I can remember:
> >> * The two sides get out of sync quickly as in:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> >> (not just the expiry queue – looking at one of our production sets
> right now on
> >> two of 6 queues I see message counts of 172 and 96 on the cold side,
> and 0
> >> on the hot side. Because this is Artemis 2.20 I cannot browse them or
> see if they
> >> are really there but I hope to learn more when we complete our 2.22
> update.)
> >> * The stats are not tracked correctly so it’s hard to tell how out of
> sync
> >> we are:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> >> * We use to end up with extremely large journals on the cold side that
> would
> >> prevent start up. Mitigated with aggressive TTL and purging on the
> cold side.
> >> * We’ve ended up with delivery of ancient messages when failing over.
> Mitigated
> >> somewhat with aggressive TTL.
> >> * More often than not when performing a graceful failover we need to
> restart Artemis
> >> on the new live. Consumers connect but they don’t receive any
> messages.
> >> * In some instances the mirror queues have not shown up in the Artemis
> console
> >> but have functioned. No known steps to reproduce, in all cases
> eventually resolved
> >> themselves.
> >>
> >> * Messages in the logs that indicate problems we don’t know the effect
> of like:
> >>
> >> 2022-05-30 00:30:12,964 WARN
> [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> mirror target, ignoring ack for
> queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >>
> >>
> >> I hope as we grow we’ll be able to devote a resource to working on
> Artemis.
> >> It’s a critical and highly performant part of our infrastructure. I
> have personally been
> >> advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> infrastructure.
> >>
> >> The JDBC replication is part of that plan, to replace a similarly JDBC
> replicated
> >> JbossMQ 4 cluster that is for the first time, outside of products that
> I’m directly
> >> responsible for.
> >>
> >> A bit more than I intended to write, but I hope this helps understand
> our where
> >> we are and our motivations.
> >>
> >>
> >> From: Justin Bertram <jb...@apache.org>
> >> Date: Wednesday, June 1, 2022 at 12:34 AM
> >> To: users@activemq.apache.org <us...@activemq.apache.org>
> >> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> update
> >> I sent a commit to update the docs in the repo, and I also updated the
> >> website.
> >>
> >> Out of curiosity, how is your MySQL replication configured? Are you
> using
> >> the default asynchronous, semisynchronous, or fully synchronous NDB
> cluster?
> >>
> >>
> >> Justin
> >>
> >> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> >> stephen.baker@rmssoftwareinc.com> wrote:
> >>
> >>> Understood thank you. We (the company I work for) are definitely
> getting
> >>> more value out of the product than we are contributing, so that point
> is
> >>> taken. The JDBC replication route was recommended by a consultant from
> >>> Savoir as more established/reliable than mirroring when delivery
> guarantees
> >>> are important, which is why we are pursuing it.
> >>>
> >>>
> >> [EXTERNAL]: This email originated from outside of Rave Mobile Safety.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe.
> >
> >
> >
> > --
> > Clebert Suconic
>
> --
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Iliya Grushevskiy <il...@gmail.com>.
Hi, Clebert
I think there is another related issue: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815 <https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3815>
I have sent a PR https://github.com/apache/activemq-artemis/pull/4066. It’s outdated now, but it contains test that I think is still relevant.
Regards
Iliya Grushevskiy
> 29 июня 2022 г., в 18:34, Clebert Suconic <cl...@gmail.com> написал(а):
>
> I have sent a PR for the mirror issues:
>
> https://github.com/apache/activemq-artemis/pull/4127
>
> On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
> <st...@rmssoftwareinc.com> wrote:
>>
>> I’m not on the DBA team so I don’t know specifics but it is asynchronous
>> replication to a secondary server in the same datacenter and a similar
>> primary/secondary server to the standby datacenter.
>>
>> Theoretically the mirroring in Artemis should offer the same resiliency
>> (synced to the disk, possibly not replicated in the event of a hard failure);
>> But in practice mirroring is Artemis is relatively new and SQL replication
>> has existed for decades with plenty of DBAs who are well versed in mitigating
>> common problems.
>>
>> Some of the issues we have with mirroring that I can remember:
>> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
>> (not just the expiry queue – looking at one of our production sets right now on
>> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
>> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
>> are really there but I hope to learn more when we complete our 2.22 update.)
>> * The stats are not tracked correctly so it’s hard to tell how out of sync
>> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
>> * We use to end up with extremely large journals on the cold side that would
>> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
>> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
>> somewhat with aggressive TTL.
>> * More often than not when performing a graceful failover we need to restart Artemis
>> on the new live. Consumers connect but they don’t receive any messages.
>> * In some instances the mirror queues have not shown up in the Artemis console
>> but have functioned. No known steps to reproduce, in all cases eventually resolved
>> themselves.
>>
>> * Messages in the logs that indicate problems we don’t know the effect of like:
>>
>> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>>
>>
>> I hope as we grow we’ll be able to devote a resource to working on Artemis.
>> It’s a critical and highly performant part of our infrastructure. I have personally been
>> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>>
>> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
>> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
>> responsible for.
>>
>> A bit more than I intended to write, but I hope this helps understand our where
>> we are and our motivations.
>>
>>
>> From: Justin Bertram <jb...@apache.org>
>> Date: Wednesday, June 1, 2022 at 12:34 AM
>> To: users@activemq.apache.org <us...@activemq.apache.org>
>> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
>> I sent a commit to update the docs in the repo, and I also updated the
>> website.
>>
>> Out of curiosity, how is your MySQL replication configured? Are you using
>> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>>
>>
>> Justin
>>
>> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
>> stephen.baker@rmssoftwareinc.com> wrote:
>>
>>> Understood thank you. We (the company I work for) are definitely getting
>>> more value out of the product than we are contributing, so that point is
>>> taken. The JDBC replication route was recommended by a consultant from
>>> Savoir as more established/reliable than mirroring when delivery guarantees
>>> are important, which is why we are pursuing it.
>>>
>>>
>> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
>
>
> --
> Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
I have sent a PR for the mirror issues:
https://github.com/apache/activemq-artemis/pull/4127
On Wed, Jun 1, 2022 at 10:06 AM Stephen Baker
<st...@rmssoftwareinc.com> wrote:
>
> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> replication to a secondary server in the same datacenter and a similar
> primary/secondary server to the standby datacenter.
>
> Theoretically the mirroring in Artemis should offer the same resiliency
> (synced to the disk, possibly not replicated in the event of a hard failure);
> But in practice mirroring is Artemis is relatively new and SQL replication
> has existed for decades with plenty of DBAs who are well versed in mitigating
> common problems.
>
> Some of the issues we have with mirroring that I can remember:
> * The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> (not just the expiry queue – looking at one of our production sets right now on
> two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
> on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
> are really there but I hope to learn more when we complete our 2.22 update.)
> * The stats are not tracked correctly so it’s hard to tell how out of sync
> we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> * We use to end up with extremely large journals on the cold side that would
> prevent start up. Mitigated with aggressive TTL and purging on the cold side.
> * We’ve ended up with delivery of ancient messages when failing over. Mitigated
> somewhat with aggressive TTL.
> * More often than not when performing a graceful failover we need to restart Artemis
> on the new live. Consumers connect but they don’t receive any messages.
> * In some instances the mirror queues have not shown up in the Artemis console
> but have functioned. No known steps to reproduce, in all cases eventually resolved
> themselves.
>
> * Messages in the logs that indicate problems we don’t know the effect of like:
>
> 2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>
>
> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> It’s a critical and highly performant part of our infrastructure. I have personally been
> advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
>
> The JDBC replication is part of that plan, to replace a similarly JDBC replicated
> JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
> responsible for.
>
> A bit more than I intended to write, but I hope this helps understand our where
> we are and our motivations.
>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, June 1, 2022 at 12:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
> I sent a commit to update the docs in the repo, and I also updated the
> website.
>
> Out of curiosity, how is your MySQL replication configured? Are you using
> the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
>
>
> Justin
>
> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > Understood thank you. We (the company I work for) are definitely getting
> > more value out of the product than we are contributing, so that point is
> > taken. The JDBC replication route was recommended by a consultant from
> > Savoir as more established/reliable than mirroring when delivery guarantees
> > are important, which is why we are pursuing it.
> >
> >
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Clebert Suconic <cl...@gmail.com>.
I will address the issues you faced on mirroring.
it's definitely new functionality but it's a hot topic.. we should
address the issues.
The issue with the Expiry is that message was moved once on the main
broker.. then moved again on the target causing the duplicate.
and the statistics.. that's something I'm working on at the moment.
On Wed, Jun 1, 2022 at 11:26 AM Justin Bertram <jb...@apache.org> wrote:
>
> Thanks for elaborating! Those are all fair points about broker mirroring
> vs. DB replication.
>
> Moving from JBossMQ to ActiveMQ Artemis should be a major step up in both
> performance and features even with the continued use of JDBC. Also, I don't
> think JBossMQ has seen any maintenance of any kind for over a decade.
> You're probably stuck running a JDK from 1.5 or 1.6. I'm sure you're eager
> to upgrade!
>
>
> Justin
>
> On Wed, Jun 1, 2022 at 9:06 AM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > I’m not on the DBA team so I don’t know specifics but it is asynchronous
> > replication to a secondary server in the same datacenter and a similar
> > primary/secondary server to the standby datacenter.
> >
> > Theoretically the mirroring in Artemis should offer the same resiliency
> > (synced to the disk, possibly not replicated in the event of a hard
> > failure);
> > But in practice mirroring is Artemis is relatively new and SQL replication
> > has existed for decades with plenty of DBAs who are well versed in
> > mitigating
> > common problems.
> >
> > Some of the issues we have with mirroring that I can remember:
> > * The two sides get out of sync quickly as in:
> > https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> > (not just the expiry queue – looking at one of our production sets
> > right now on
> > two of 6 queues I see message counts of 172 and 96 on the cold side,
> > and 0
> > on the hot side. Because this is Artemis 2.20 I cannot browse them or
> > see if they
> > are really there but I hope to learn more when we complete our 2.22
> > update.)
> > * The stats are not tracked correctly so it’s hard to tell how out of sync
> > we are:
> > https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> > * We use to end up with extremely large journals on the cold side that
> > would
> > prevent start up. Mitigated with aggressive TTL and purging on the cold
> > side.
> > * We’ve ended up with delivery of ancient messages when failing over.
> > Mitigated
> > somewhat with aggressive TTL.
> > * More often than not when performing a graceful failover we need to
> > restart Artemis
> > on the new live. Consumers connect but they don’t receive any messages.
> > * In some instances the mirror queues have not shown up in the Artemis
> > console
> > but have functioned. No known steps to reproduce, in all cases
> > eventually resolved
> > themselves.
> >
> > * Messages in the logs that indicate problems we don’t know the effect of
> > like:
> >
> > 2022-05-30 00:30:12,964 WARN
> > [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> > Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> > mirror target, ignoring ack for
> > queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> > messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
> >
> >
> > I hope as we grow we’ll be able to devote a resource to working on Artemis.
> > It’s a critical and highly performant part of our infrastructure. I have
> > personally been
> > advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> > infrastructure.
> >
> > The JDBC replication is part of that plan, to replace a similarly JDBC
> > replicated
> > JbossMQ 4 cluster that is for the first time, outside of products that I’m
> > directly
> > responsible for.
> >
> > A bit more than I intended to write, but I hope this helps understand our
> > where
> > we are and our motivations.
> >
> >
> > From: Justin Bertram <jb...@apache.org>
> > Date: Wednesday, June 1, 2022 at 12:34 AM
> > To: users@activemq.apache.org <us...@activemq.apache.org>
> > Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> > update
> > I sent a commit to update the docs in the repo, and I also updated the
> > website.
> >
> > Out of curiosity, how is your MySQL replication configured? Are you using
> > the default asynchronous, semisynchronous, or fully synchronous NDB
> > cluster?
> >
> >
> > Justin
> >
> > On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> > stephen.baker@rmssoftwareinc.com> wrote:
> >
> > > Understood thank you. We (the company I work for) are definitely getting
> > > more value out of the product than we are contributing, so that point is
> > > taken. The JDBC replication route was recommended by a consultant from
> > > Savoir as more established/reliable than mirroring when delivery
> > guarantees
> > > are important, which is why we are pursuing it.
> > >
> > >
> > [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> > not click links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
--
Clebert Suconic
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Justin Bertram <jb...@apache.org>.
Thanks for elaborating! Those are all fair points about broker mirroring
vs. DB replication.
Moving from JBossMQ to ActiveMQ Artemis should be a major step up in both
performance and features even with the continued use of JDBC. Also, I don't
think JBossMQ has seen any maintenance of any kind for over a decade.
You're probably stuck running a JDK from 1.5 or 1.6. I'm sure you're eager
to upgrade!
Justin
On Wed, Jun 1, 2022 at 9:06 AM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> I’m not on the DBA team so I don’t know specifics but it is asynchronous
> replication to a secondary server in the same datacenter and a similar
> primary/secondary server to the standby datacenter.
>
> Theoretically the mirroring in Artemis should offer the same resiliency
> (synced to the disk, possibly not replicated in the event of a hard
> failure);
> But in practice mirroring is Artemis is relatively new and SQL replication
> has existed for decades with plenty of DBAs who are well versed in
> mitigating
> common problems.
>
> Some of the issues we have with mirroring that I can remember:
> * The two sides get out of sync quickly as in:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
> (not just the expiry queue – looking at one of our production sets
> right now on
> two of 6 queues I see message counts of 172 and 96 on the cold side,
> and 0
> on the hot side. Because this is Artemis 2.20 I cannot browse them or
> see if they
> are really there but I hope to learn more when we complete our 2.22
> update.)
> * The stats are not tracked correctly so it’s hard to tell how out of sync
> we are:
> https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
> * We use to end up with extremely large journals on the cold side that
> would
> prevent start up. Mitigated with aggressive TTL and purging on the cold
> side.
> * We’ve ended up with delivery of ancient messages when failing over.
> Mitigated
> somewhat with aggressive TTL.
> * More often than not when performing a graceful failover we need to
> restart Artemis
> on the new live. Consumers connect but they don’t receive any messages.
> * In some instances the mirror queues have not shown up in the Artemis
> console
> but have functioned. No known steps to reproduce, in all cases
> eventually resolved
> themselves.
>
> * Messages in the logs that indicate problems we don’t know the effect of
> like:
>
> 2022-05-30 00:30:12,964 WARN
> [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget]
> Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on
> mirror target, ignoring ack for
> queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c,
> messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
>
>
> I hope as we grow we’ll be able to devote a resource to working on Artemis.
> It’s a critical and highly performant part of our infrastructure. I have
> personally been
> advocating for it to replace our aging JbossMQ 4 (pre-hornet)
> infrastructure.
>
> The JDBC replication is part of that plan, to replace a similarly JDBC
> replicated
> JbossMQ 4 cluster that is for the first time, outside of products that I’m
> directly
> responsible for.
>
> A bit more than I intended to write, but I hope this helps understand our
> where
> we are and our motivations.
>
>
> From: Justin Bertram <jb...@apache.org>
> Date: Wednesday, June 1, 2022 at 12:34 AM
> To: users@activemq.apache.org <us...@activemq.apache.org>
> Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22
> update
> I sent a commit to update the docs in the repo, and I also updated the
> website.
>
> Out of curiosity, how is your MySQL replication configured? Are you using
> the default asynchronous, semisynchronous, or fully synchronous NDB
> cluster?
>
>
> Justin
>
> On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
> stephen.baker@rmssoftwareinc.com> wrote:
>
> > Understood thank you. We (the company I work for) are definitely getting
> > more value out of the product than we are contributing, so that point is
> > taken. The JDBC replication route was recommended by a consultant from
> > Savoir as more established/reliable than mirroring when delivery
> guarantees
> > are important, which is why we are pursuing it.
> >
> >
> [EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do
> not click links or open attachments unless you recognize the sender and
> know the content is safe.
>
Re: Artemis MQ with JDBC persistence not starting since 2.22 update
Posted by Stephen Baker <st...@rmssoftwareinc.com>.
I’m not on the DBA team so I don’t know specifics but it is asynchronous
replication to a secondary server in the same datacenter and a similar
primary/secondary server to the standby datacenter.
Theoretically the mirroring in Artemis should offer the same resiliency
(synced to the disk, possibly not replicated in the event of a hard failure);
But in practice mirroring is Artemis is relatively new and SQL replication
has existed for decades with plenty of DBAs who are well versed in mitigating
common problems.
Some of the issues we have with mirroring that I can remember:
* The two sides get out of sync quickly as in: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3766
(not just the expiry queue – looking at one of our production sets right now on
two of 6 queues I see message counts of 172 and 96 on the cold side, and 0
on the hot side. Because this is Artemis 2.20 I cannot browse them or see if they
are really there but I hope to learn more when we complete our 2.22 update.)
* The stats are not tracked correctly so it’s hard to tell how out of sync
we are: https://issues.apache.org/jira/projects/ARTEMIS/issues/ARTEMIS-3743
* We use to end up with extremely large journals on the cold side that would
prevent start up. Mitigated with aggressive TTL and purging on the cold side.
* We’ve ended up with delivery of ancient messages when failing over. Mitigated
somewhat with aggressive TTL.
* More often than not when performing a graceful failover we need to restart Artemis
on the new live. Consumers connect but they don’t receive any messages.
* In some instances the mirror queues have not shown up in the Artemis console
but have functioned. No known steps to reproduce, in all cases eventually resolved
themselves.
* Messages in the logs that indicate problems we don’t know the effect of like:
2022-05-30 00:30:12,964 WARN [org.apache.activemq.artemis.protocol.amqp.connect.mirror.AMQPMirrorControllerTarget] Queue activemq.management.665d289a-520e-40a6-9233-96265817ca6c not found on mirror target, ignoring ack for queue=activemq.management.665d289a-520e-40a6-9233-96265817ca6c, messageID=68514641673, nodeID=dea32b83-efd5-11eb-b5b1-0050568fe3b2
I hope as we grow we’ll be able to devote a resource to working on Artemis.
It’s a critical and highly performant part of our infrastructure. I have personally been
advocating for it to replace our aging JbossMQ 4 (pre-hornet) infrastructure.
The JDBC replication is part of that plan, to replace a similarly JDBC replicated
JbossMQ 4 cluster that is for the first time, outside of products that I’m directly
responsible for.
A bit more than I intended to write, but I hope this helps understand our where
we are and our motivations.
From: Justin Bertram <jb...@apache.org>
Date: Wednesday, June 1, 2022 at 12:34 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Artemis MQ with JDBC persistence not starting since 2.22 update
I sent a commit to update the docs in the repo, and I also updated the
website.
Out of curiosity, how is your MySQL replication configured? Are you using
the default asynchronous, semisynchronous, or fully synchronous NDB cluster?
Justin
On Tue, May 31, 2022 at 4:09 PM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:
> Understood thank you. We (the company I work for) are definitely getting
> more value out of the product than we are contributing, so that point is
> taken. The JDBC replication route was recommended by a consultant from
> Savoir as more established/reliable than mirroring when delivery guarantees
> are important, which is why we are pursuing it.
>
>
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.