You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Roy Cohen <ro...@hotmail.com> on 2023/03/23 12:37:06 UTC

Co Located Backups Question

Hello everyone

We have a setup of three Artemis brokers (very old version don’t ask :))

We would like to configure the co located backups such that the backups are sent in this order:

Broker01 -> Broker02
Broker02 -> Broker03
Broker03 -> Broker01


I was reading on co located backups here:  https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html however not sure I fully understand how to configure the xml section to achieve that.

Shall I add excludes in each broker, i.e.

      <colocated>
         <excludes>
            <connector-ref>...</connector-ref>
         </excludes>

Any help would be appreciated.

Many thanks in advance !

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
1. Why change ?

2. What would the co located back strategy look like with three ? Will it align to my expectation below ?
Broker01 -> Broker02
Broker02 -> Broker03
Broker03 -> Broker01

3. How can I prove the co located backup strategy with three brokers ?



> On 27 Mar 2023, at 16:05, prateekjainaa@gmail.com wrote:
> 
> IMO, the only change would be changing broker (61616 to something like
> 61618) port. I would expect everything should work after that.
> 
> Regards,
> Prateek Jain
> 
> --------------------------------------------------------------
> EXPECTATION : Causes all troubles......
> --------------------------------------------------------------
> 
> 
> On Mon, Mar 27, 2023 at 3:56 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Hi Prateek
>> 
>> Thanks, however that example is for two brokers, which we already have
>> working.
>> 
>> We are now adding a third broker to the cluster and want to understand how
>> to change the co located backups according to my original post.
>> 
>> Is that possible ?
>> 
>> Regards
>> Roy
>> 
>> 
>> 
>> 
>>> On 27 Mar 2023, at 15:46, prateekjainaa@gmail.com wrote:
>>> 
>>> Hi Roy,
>>> 
>>> I dont exactly know your usecase or constraints but IMO, shared store
>>> would have been a better option. As I didnt worked/explored 1.x version
>> so,
>>> wont comment on it. But you should be able to reference examples under
>>> artemis directory:
>>> 
>>>  *examples\features\ha\colocated-failover*
>>> 
>>> Hope it helps.
>>> 
>>> Regards,
>>> Prateek Jain
>>> 
>>> --------------------------------------------------------------
>>> EXPECTATION : Causes all troubles......
>>> --------------------------------------------------------------
>>> 
>>> 
>>> On Mon, Mar 27, 2023 at 3:27 PM Roy Cohen <ro...@hotmail.com> wrote:
>>> 
>>>> Anyone has any thoughts on the below ?
>>>> 
>>>> On 23 Mar 2023, at 12:37, Roy Cohen <ro...@hotmail.com> wrote:
>>>> 
>>>>  Hello everyone
>>>> 
>>>> We have a setup of three Artemis brokers (very old version don’t ask :))
>>>> 
>>>> We would like to configure the co located backups such that the backups
>>>> are sent in this order:
>>>> 
>>>> Broker01 -> Broker02
>>>> Broker02 -> Broker03
>>>> Broker03 -> Broker01
>>>> 
>>>> 
>>>> I was reading on co located backups here:
>>>> 
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>> however not sure I fully understand how to configure the xml section to
>>>> achieve that.
>>>> 
>>>> Shall I add excludes in each broker, i.e.
>>>> 
>>>>     <colocated>
>>>>        <excludes>
>>>>           <connector-ref>...</connector-ref>
>>>>        </excludes>
>>>> 
>>>> Any help would be appreciated.
>>>> 
>>>> Many thanks in advance !
>>>> 
>>>> 
>>>> 
>> 
>>

Re: Co Located Backups Question

Posted by "prateekjainaa@gmail.com" <pr...@gmail.com>.
IMO, the only change would be changing broker (61616 to something like
61618) port. I would expect everything should work after that.

Regards,
Prateek Jain

--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------


On Mon, Mar 27, 2023 at 3:56 PM Roy Cohen <ro...@hotmail.com> wrote:

> Hi Prateek
>
> Thanks, however that example is for two brokers, which we already have
> working.
>
> We are now adding a third broker to the cluster and want to understand how
> to change the co located backups according to my original post.
>
> Is that possible ?
>
> Regards
> Roy
>
>
>
>
> > On 27 Mar 2023, at 15:46, prateekjainaa@gmail.com wrote:
> >
> > Hi Roy,
> >
> > I dont exactly know your usecase or constraints but IMO, shared store
> > would have been a better option. As I didnt worked/explored 1.x version
> so,
> > wont comment on it. But you should be able to reference examples under
> > artemis directory:
> >
> >   *examples\features\ha\colocated-failover*
> >
> > Hope it helps.
> >
> > Regards,
> > Prateek Jain
> >
> > --------------------------------------------------------------
> > EXPECTATION : Causes all troubles......
> > --------------------------------------------------------------
> >
> >
> > On Mon, Mar 27, 2023 at 3:27 PM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> Anyone has any thoughts on the below ?
> >>
> >> On 23 Mar 2023, at 12:37, Roy Cohen <ro...@hotmail.com> wrote:
> >>
> >>  Hello everyone
> >>
> >> We have a setup of three Artemis brokers (very old version don’t ask :))
> >>
> >> We would like to configure the co located backups such that the backups
> >> are sent in this order:
> >>
> >> Broker01 -> Broker02
> >> Broker02 -> Broker03
> >> Broker03 -> Broker01
> >>
> >>
> >> I was reading on co located backups here:
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >> however not sure I fully understand how to configure the xml section to
> >> achieve that.
> >>
> >> Shall I add excludes in each broker, i.e.
> >>
> >>      <colocated>
> >>         <excludes>
> >>            <connector-ref>...</connector-ref>
> >>         </excludes>
> >>
> >> Any help would be appreciated.
> >>
> >> Many thanks in advance !
> >>
> >>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Hi Prateek

Thanks, however that example is for two brokers, which we already have working.

We are now adding a third broker to the cluster and want to understand how to change the co located backups according to my original post.

Is that possible ?

Regards
Roy




> On 27 Mar 2023, at 15:46, prateekjainaa@gmail.com wrote:
> 
> Hi Roy,
> 
> I dont exactly know your usecase or constraints but IMO, shared store
> would have been a better option. As I didnt worked/explored 1.x version so,
> wont comment on it. But you should be able to reference examples under
> artemis directory:
> 
>   *examples\features\ha\colocated-failover*
> 
> Hope it helps.
> 
> Regards,
> Prateek Jain
> 
> --------------------------------------------------------------
> EXPECTATION : Causes all troubles......
> --------------------------------------------------------------
> 
> 
> On Mon, Mar 27, 2023 at 3:27 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Anyone has any thoughts on the below ?
>> 
>> On 23 Mar 2023, at 12:37, Roy Cohen <ro...@hotmail.com> wrote:
>> 
>>  Hello everyone
>> 
>> We have a setup of three Artemis brokers (very old version don’t ask :))
>> 
>> We would like to configure the co located backups such that the backups
>> are sent in this order:
>> 
>> Broker01 -> Broker02
>> Broker02 -> Broker03
>> Broker03 -> Broker01
>> 
>> 
>> I was reading on co located backups here:
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>> however not sure I fully understand how to configure the xml section to
>> achieve that.
>> 
>> Shall I add excludes in each broker, i.e.
>> 
>>      <colocated>
>>         <excludes>
>>            <connector-ref>...</connector-ref>
>>         </excludes>
>> 
>> Any help would be appreciated.
>> 
>> Many thanks in advance !
>> 
>> 
>>

Re: Co Located Backups Question

Posted by "prateekjainaa@gmail.com" <pr...@gmail.com>.
Hi Roy,

 I dont exactly know your usecase or constraints but IMO, shared store
would have been a better option. As I didnt worked/explored 1.x version so,
wont comment on it. But you should be able to reference examples under
artemis directory:

   *examples\features\ha\colocated-failover*

Hope it helps.

Regards,
Prateek Jain

--------------------------------------------------------------
EXPECTATION : Causes all troubles......
--------------------------------------------------------------


On Mon, Mar 27, 2023 at 3:27 PM Roy Cohen <ro...@hotmail.com> wrote:

> Anyone has any thoughts on the below ?
>
> On 23 Mar 2023, at 12:37, Roy Cohen <ro...@hotmail.com> wrote:
>
>  Hello everyone
>
> We have a setup of three Artemis brokers (very old version don’t ask :))
>
> We would like to configure the co located backups such that the backups
> are sent in this order:
>
> Broker01 -> Broker02
> Broker02 -> Broker03
> Broker03 -> Broker01
>
>
> I was reading on co located backups here:
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> however not sure I fully understand how to configure the xml section to
> achieve that.
>
> Shall I add excludes in each broker, i.e.
>
>       <colocated>
>          <excludes>
>             <connector-ref>...</connector-ref>
>          </excludes>
>
> Any help would be appreciated.
>
> Many thanks in advance !
>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Anyone has any thoughts on the below ?

On 23 Mar 2023, at 12:37, Roy Cohen <ro...@hotmail.com> wrote:

 Hello everyone

We have a setup of three Artemis brokers (very old version don’t ask :))

We would like to configure the co located backups such that the backups are sent in this order:

Broker01 -> Broker02
Broker02 -> Broker03
Broker03 -> Broker01


I was reading on co located backups here:  https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html however not sure I fully understand how to configure the xml section to achieve that.

Shall I add excludes in each broker, i.e.

      <colocated>
         <excludes>
            <connector-ref>...</connector-ref>
         </excludes>

Any help would be appreciated.

Many thanks in advance !

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Thank you sir :)

On 28 Mar 2023, at 16:55, Justin Bertram <jb...@apache.org> wrote:

Invitation sent.


Justin

On Tue, Mar 28, 2023 at 10:45 AM Roy Cohen <ro...@hotmail.com> wrote:

> Gotcha.
> 
> Suppose that mail group, so I hereby requesting an invite please :)
> 
> On 28 Mar 2023, at 16:40, Justin Bertram <jb...@apache.org> wrote:
> 
> You don't need an @apache.org email in order to join the ASF Slack. You
> just need to request an invitation. This is noted on the website [1] (at
> the bottom):
> 
>   If you want an invitation to the ActiveMQ Slack channel simply send a
> request to the users mailing list.
> 
> Most of the folks in there don't have an @apache.org email.
> 
> 
> Justin
> 
> [1] https://activemq.apache.org/contact
> 
> On Tue, Mar 28, 2023 at 10:31 AM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Understood and yes there’s a reason.
>> 
>> Don’t I need @apache.org email in order to join that Slack workspace ?
>> 
>> 
>>> On 28 Mar 2023, at 15:52, Justin Bertram <jb...@apache.org> wrote:
>>> Typically we like to keep everything on the list so the whole community
>> can
>>> benefit from the discussion. However, if there's a specific reason that
>>> privacy is a concern then you can email me directly or you can find me
> on
>>> the ASF Slack in #activemq.
>>> Justin
>>> On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com>
> wrote:
>>>> Hi Justin
>>>> Would it be able to reach out to you directly to discuss a couple of
>>>> points around the discussion below privately ?
>>>> Thanks
>>>> Roy
>>>>> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
>>>>> You have indeed ! :)
>>>>> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
>>>>> I wrote that on a completely different thread [1] related to MQTT
>>>> retained
>>>>> messages in a cluster. It is not related to this thread or your issue
>>>>> generally.
>>>>> Justin
>>>>> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
>>>>> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>>>>> To quote:
>>>>>> “This functionality isn't supported, and while it may be technically
>>>>>> feasible to implement I'm not sure how much sense it makes overall.”
>>>>>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org>
> wrote:
>>>>>> I'm not sure where I may have indicated that either one of those
>> things
>>>>>> isn't supported.
>>>>>> In any case, you can do either.
>>>>>> Justin
>>>>>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
>>>> wrote:
>>>>>>> Just to be clear: When you say “isn’t supported” do you mean a third
>>>>>>> broker or co located backups when running each broker on its own VM
> ?
>>>>>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>>>>>>>> Will do Justin and many thanks for all the additional details which
>> I
>>>>>>> will certainly bring forward internally, much appreciated
>>>>>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org>
>> wrote:
>>>>>>>> I recently added a new section to the clustering documentation
>>>>>> regarding
>>>>>>>> things to keep in mind regarding performance [1].
>>>>>>>> Also, it's worth noting that often the bottleneck in messaging is
>> not
>>>>>> the
>>>>>>>> broker itself but rather the consumer(s). It might be worth
> ensuring
>>>>>> that
>>>>>>>> the bottleneck really is the broker. As noted in the new
>> documentation
>>>>>>> [1],
>>>>>>>> adding brokers to a cluster can actually *reduce* throughput in
>>>> certain
>>>>>>>> circumstances.
>>>>>>>> Let me know if using group-name works for you.
>>>>>>>> Justin
>>>>>>>> [1]
>> 
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>>>>>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
>>>>>>> wrote:
>>>>>>>>> I haven’t tried he group-name yet.
>>>>>>>>> With regards to the third broker: The architects believe it’ll
>>>> improve
>>>>>>>>> performance given the amount of messages the brokers need to
>> process
>>>>>> (in
>>>>>>>>> other words “throw more resources at it…”)
>>>>>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
>>>> wrote:
>>>>>>>>>>> What would you suggest is to do ?
>>>>>>>>>> Did you try my previous suggestion already (i.e. using the
>>>>>> "group-name"
>>>>>>>>>> element in the "master" or "slave" element of "colocated")?
>>>>>>>>>> Aside from that, do you know why you were asked to add another
>>>> broker?
>>>>>>>>>> Depending on the reason it may not be a good solution.
>>>>>>>>>> Justin
>>>>>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <
> roy_cohen@hotmail.com
>>>>>>>>> wrote:
>>>>>>>>>>> Hi Justin
>>>>>>>>>>> It is a good question I honestly don’t have the answer for. I
>>>>>>> inherited
>>>>>>>>>>> this configuration and was asked to add a third broker and to
>>>> ensure
>>>>>>>>> the co
>>>>>>>>>>> located backups are being done in such a way that each broker
>>>> points
>>>>>>> on
>>>>>>>>>>> another. Perhaps those who asked for it don’t fully understand
>>>>>> Artemis
>>>>>>>>> ! :)
>>>>>>>>>>> That said, those co located backup on the existing setup with
> two
>>>>>>>>> brokers
>>>>>>>>>>> do work as we have been enabled to recover lost messages in the
>>>> past.
>>>>>>> So
>>>>>>>>>>> even not optimal, technically it does work ?
>>>>>>>>>>> I can only imagine that those who initially designed it about 5
>>>> years
>>>>>>>>> ago
>>>>>>>>>>> did not use a shared storage to avoid latency.
>>>>>>>>>>> What would you suggest is to do ?
>>>>>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
>>>>>> wrote:
>>>>>>>>>>> Your screenshot didn't come through the list.
>>>>>>>>>>> In any case, I'm pretty confused at this point. You're clearly
>>>> using
>>>>>> a
>>>>>>>>>>> colocated configuration that will request a backup from another
>>>>>> broker
>>>>>>>>> in
>>>>>>>>>>> the cluster, but you say you're not running multiple brokers in
>> the
>>>>>>> same
>>>>>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
>>>> what
>>>>>>>>> are
>>>>>>>>>>> you using the colocated configuration for? The whole point of
> the
>>>>>>>>> colocated
>>>>>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
>>>>>>> primary
>>>>>>>>>>> broker and also a backup broker for another primary in the
>>>> cluster).
>>>>>>>>>>> Justin
>>>>>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <
>> roy_cohen@hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> I don’t believe we are.
>>>>>>>>>>>> So assume three Virtual Machines on Azure.
>>>>>>>>>>>> Each VM runs one Artemis broker
>>>>>>>>>>>> All of their ha policy section on all three brokers look like
>>>> that:
>>>>>>>>>>>> <ha-policy>
>>>>>>>>>>>> <replication>
>>>>>>>>>>>> <colocated>
>>>>>>>>>>>>   <max-backups>1</max-backups>
>>>>>>>>>>>>   <request-backup>true</request-backup>
>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>>>>>>>   <excludes>
>>>>>>>>>>>>     <connector-ref>my-connector</connector-ref>
>>>>>>>>>>>>     <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>>>>>>>   </excludes>
>>>>>>>>>>>>   <master>
>>>>>>>>>>>>     <check-for-live-server>true</check-for-live-server>
>>>>>>>>>>>>   </master>
>>>>>>>>>>>>   <slave>
>>>>>>>>>>>>     <allow-failback>true</allow-failback>
>>>>>>>>>>>>     <restart-backup>true</restart-backup>
>>>>>>>>>>>>     <scale-down/>
>>>>>>>>>>>>   </slave>
>>>>>>>>>>>> </colocated>
>>>>>>>>>>>> </replication>
>>>>>>>>>>>> </ha-policy>
>>>>>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>> We are not running multiple brokers on the same JVM but a
> single
>>>>>>>>> instance
>>>>>>>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>>>>>>>> Based on your previous message I was under the impression you
>> were
>>>>>>>>> using
>>>>>>>>>>>> the "colocated" feature. *If* you're using this then you
>>>> definitely
>>>>>>> are
>>>>>>>>>>>> running multiple brokers in the same JVM because that's
>> precisely
>>>>>>> what
>>>>>>>>>>> that
>>>>>>>>>>>> feature does. It runs a primary and a backup broker in the
> *same
>>>>>>> JVM*.
>>>>>>>>> If
>>>>>>>>>>>> you aren't using a "colocated" configuration then I'm not sure
>>>> what
>>>>>>> the
>>>>>>>>>>>> original question is about. Can you clarify?
>>>>>>>>>>>> Justin
>>>>>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <
>> roy_cohen@hotmail.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi Justin
>>>>>>>>>>>> Thank you for your input.
>>>>>>>>>>>> Sorry, should have been clearer on our setup - We are not
>> running
>>>>>>>>>>> multiple
>>>>>>>>>>>> brokers on the same JVM but a single instance per VM, so each
>> one
>>>>>>> has a
>>>>>>>>>>>> dedicated JVM and VM
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Roy
>>>>>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>> I'm not entirely sure if the configuration you want is
> possible.
>>>> You
>>>>>>>>>>>> might
>>>>>>>>>>>> try using the "group-name" element in the "master" or "slave"
>>>>>> element
>>>>>>>>> of
>>>>>>>>>>>> "colocated." Only servers with the same group-name will pair
>>>>>>> together.
>>>>>>>>>>>> Aside from that I would actually recommend against using
>> colocated
>>>>>>>>>>>> brokers.
>>>>>>>>>>>> The original use-case for this functionality was very early
>> cloud
>>>>>>>>>>>> infrastructure where durable, attached storage was not readily
>>>>>>>>> available.
>>>>>>>>>>>> However, since then most (if not all) cloud environments
> support
>>>>>>>>> durable
>>>>>>>>>>>> storage separate from the broker so that if the broker goes
>> down a
>>>>>>> new,
>>>>>>>>>>>> identical broker can be spun-up relatively quickly and attached
>> to
>>>>>>> the
>>>>>>>>>>>> same
>>>>>>>>>>>> storage. This provides functional high availability without the
>>>> need
>>>>>>>>> for
>>>>>>>>>>>> any idle backups or replication of any kind which functionally
>>>>>>>>> nullifies
>>>>>>>>>>>> this feature.
>>>>>>>>>>>> Additionally, it turns out that (surprise!) configuring &
>> running
>>>>>>>>>>>> multiple
>>>>>>>>>>>> brokers in the same JVM is difficult and error-prone not to
>>>> mention
>>>>>>> the
>>>>>>>>>>>> complication of dynamically coordinating the acquisition of
>>>> backups
>>>>>>> in
>>>>>>>>> a
>>>>>>>>>>>> running cluster and protecting against split-brain.
>>>>>>>>>>>> Justin
>>>>>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <
>> roy_cohen@hotmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>>> Hello everyone
>>>>>>>>>>>> We have a setup of three Artemis brokers (very old version
> don’t
>>>> ask
>>>>>>>>> :))
>>>>>>>>>>>> We would like to configure the co located backups such that the
>>>>>>> backups
>>>>>>>>>>>> are sent in this order:
>>>>>>>>>>>> Broker01 -> Broker02
>>>>>>>>>>>> Broker02 -> Broker03
>>>>>>>>>>>> Broker03 -> Broker01
>>>>>>>>>>>> I was reading on co located backups here:
>> 
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>>>>>>>> however not sure I fully understand how to configure the xml
>>>> section
>>>>>>> to
>>>>>>>>>>>> achieve that.
>>>>>>>>>>>> Shall I add excludes in each broker, i.e.
>>>>>>>>>>>> <colocated>
>>>>>>>>>>>> <excludes>
>>>>>>>>>>>> <connector-ref>...</connector-ref>
>>>>>>>>>>>> </excludes>
>>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>>> Many thanks in advance !
>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
Invitation sent.


Justin

On Tue, Mar 28, 2023 at 10:45 AM Roy Cohen <ro...@hotmail.com> wrote:

> Gotcha.
>
> Suppose that mail group, so I hereby requesting an invite please :)
>
> On 28 Mar 2023, at 16:40, Justin Bertram <jb...@apache.org> wrote:
>
> You don't need an @apache.org email in order to join the ASF Slack. You
> just need to request an invitation. This is noted on the website [1] (at
> the bottom):
>
>    If you want an invitation to the ActiveMQ Slack channel simply send a
> request to the users mailing list.
>
> Most of the folks in there don't have an @apache.org email.
>
>
> Justin
>
> [1] https://activemq.apache.org/contact
>
> On Tue, Mar 28, 2023 at 10:31 AM Roy Cohen <ro...@hotmail.com> wrote:
>
> > Understood and yes there’s a reason.
> >
> > Don’t I need @apache.org email in order to join that Slack workspace ?
> >
> >
> >> On 28 Mar 2023, at 15:52, Justin Bertram <jb...@apache.org> wrote:
> >> Typically we like to keep everything on the list so the whole community
> > can
> >> benefit from the discussion. However, if there's a specific reason that
> >> privacy is a concern then you can email me directly or you can find me
> on
> >> the ASF Slack in #activemq.
> >> Justin
> >> On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>> Hi Justin
> >>> Would it be able to reach out to you directly to discuss a couple of
> >>> points around the discussion below privately ?
> >>> Thanks
> >>> Roy
> >>>> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
> >>>> You have indeed ! :)
> >>>> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
> >>>> I wrote that on a completely different thread [1] related to MQTT
> >>> retained
> >>>> messages in a cluster. It is not related to this thread or your issue
> >>>> generally.
> >>>> Justin
> >>>> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
> >>>> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com>
> > wrote:
> >>>>> To quote:
> >>>>> “This functionality isn't supported, and while it may be technically
> >>>>> feasible to implement I'm not sure how much sense it makes overall.”
> >>>>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>> I'm not sure where I may have indicated that either one of those
> > things
> >>>>> isn't supported.
> >>>>> In any case, you can do either.
> >>>>> Justin
> >>>>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
> >>> wrote:
> >>>>>> Just to be clear: When you say “isn’t supported” do you mean a third
> >>>>>> broker or co located backups when running each broker on its own VM
> ?
> >>>>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> >>>>>>> Will do Justin and many thanks for all the additional details which
> > I
> >>>>>> will certainly bring forward internally, much appreciated
> >>>>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org>
> > wrote:
> >>>>>>> I recently added a new section to the clustering documentation
> >>>>> regarding
> >>>>>>> things to keep in mind regarding performance [1].
> >>>>>>> Also, it's worth noting that often the bottleneck in messaging is
> > not
> >>>>> the
> >>>>>>> broker itself but rather the consumer(s). It might be worth
> ensuring
> >>>>> that
> >>>>>>> the bottleneck really is the broker. As noted in the new
> > documentation
> >>>>>> [1],
> >>>>>>> adding brokers to a cluster can actually *reduce* throughput in
> >>> certain
> >>>>>>> circumstances.
> >>>>>>> Let me know if using group-name works for you.
> >>>>>>> Justin
> >>>>>>> [1]
> >
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> >>>>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> >>>>>> wrote:
> >>>>>>>> I haven’t tried he group-name yet.
> >>>>>>>> With regards to the third broker: The architects believe it’ll
> >>> improve
> >>>>>>>> performance given the amount of messages the brokers need to
> > process
> >>>>> (in
> >>>>>>>> other words “throw more resources at it…”)
> >>>>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
> >>> wrote:
> >>>>>>>>>> What would you suggest is to do ?
> >>>>>>>>> Did you try my previous suggestion already (i.e. using the
> >>>>> "group-name"
> >>>>>>>>> element in the "master" or "slave" element of "colocated")?
> >>>>>>>>> Aside from that, do you know why you were asked to add another
> >>> broker?
> >>>>>>>>> Depending on the reason it may not be a good solution.
> >>>>>>>>> Justin
> >>>>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <
> roy_cohen@hotmail.com
> >>>>>>>> wrote:
> >>>>>>>>>> Hi Justin
> >>>>>>>>>> It is a good question I honestly don’t have the answer for. I
> >>>>>> inherited
> >>>>>>>>>> this configuration and was asked to add a third broker and to
> >>> ensure
> >>>>>>>> the co
> >>>>>>>>>> located backups are being done in such a way that each broker
> >>> points
> >>>>>> on
> >>>>>>>>>> another. Perhaps those who asked for it don’t fully understand
> >>>>> Artemis
> >>>>>>>> ! :)
> >>>>>>>>>> That said, those co located backup on the existing setup with
> two
> >>>>>>>> brokers
> >>>>>>>>>> do work as we have been enabled to recover lost messages in the
> >>> past.
> >>>>>> So
> >>>>>>>>>> even not optimal, technically it does work ?
> >>>>>>>>>> I can only imagine that those who initially designed it about 5
> >>> years
> >>>>>>>> ago
> >>>>>>>>>> did not use a shared storage to avoid latency.
> >>>>>>>>>> What would you suggest is to do ?
> >>>>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
> >>>>> wrote:
> >>>>>>>>>> Your screenshot didn't come through the list.
> >>>>>>>>>> In any case, I'm pretty confused at this point. You're clearly
> >>> using
> >>>>> a
> >>>>>>>>>> colocated configuration that will request a backup from another
> >>>>> broker
> >>>>>>>> in
> >>>>>>>>>> the cluster, but you say you're not running multiple brokers in
> > the
> >>>>>> same
> >>>>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
> >>> what
> >>>>>>>> are
> >>>>>>>>>> you using the colocated configuration for? The whole point of
> the
> >>>>>>>> colocated
> >>>>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
> >>>>>> primary
> >>>>>>>>>> broker and also a backup broker for another primary in the
> >>> cluster).
> >>>>>>>>>> Justin
> >>>>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <
> > roy_cohen@hotmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>> I don’t believe we are.
> >>>>>>>>>>> So assume three Virtual Machines on Azure.
> >>>>>>>>>>> Each VM runs one Artemis broker
> >>>>>>>>>>> All of their ha policy section on all three brokers look like
> >>> that:
> >>>>>>>>>>> <ha-policy>
> >>>>>>>>>>> <replication>
> >>>>>>>>>>>  <colocated>
> >>>>>>>>>>>    <max-backups>1</max-backups>
> >>>>>>>>>>>    <request-backup>true</request-backup>
> >>> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>>>>>>>>>    <excludes>
> >>>>>>>>>>>      <connector-ref>my-connector</connector-ref>
> >>>>>>>>>>>      <connector-ref>thishostname.mydomain</connector-ref>
> >>>>>>>>>>>    </excludes>
> >>>>>>>>>>>    <master>
> >>>>>>>>>>>      <check-for-live-server>true</check-for-live-server>
> >>>>>>>>>>>    </master>
> >>>>>>>>>>>    <slave>
> >>>>>>>>>>>      <allow-failback>true</allow-failback>
> >>>>>>>>>>>      <restart-backup>true</restart-backup>
> >>>>>>>>>>>      <scale-down/>
> >>>>>>>>>>>    </slave>
> >>>>>>>>>>>  </colocated>
> >>>>>>>>>>> </replication>
> >>>>>>>>>>> </ha-policy>
> >>>>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>> We are not running multiple brokers on the same JVM but a
> single
> >>>>>>>> instance
> >>>>>>>>>>> per VM, so each one has a dedicated JVM and VM
> >>>>>>>>>>> Based on your previous message I was under the impression you
> > were
> >>>>>>>> using
> >>>>>>>>>>> the "colocated" feature. *If* you're using this then you
> >>> definitely
> >>>>>> are
> >>>>>>>>>>> running multiple brokers in the same JVM because that's
> > precisely
> >>>>>> what
> >>>>>>>>>> that
> >>>>>>>>>>> feature does. It runs a primary and a backup broker in the
> *same
> >>>>>> JVM*.
> >>>>>>>> If
> >>>>>>>>>>> you aren't using a "colocated" configuration then I'm not sure
> >>> what
> >>>>>> the
> >>>>>>>>>>> original question is about. Can you clarify?
> >>>>>>>>>>> Justin
> >>>>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <
> > roy_cohen@hotmail.com
> >>>>>>>>>> wrote:
> >>>>>>>>>>> Hi Justin
> >>>>>>>>>>> Thank you for your input.
> >>>>>>>>>>> Sorry, should have been clearer on our setup - We are not
> > running
> >>>>>>>>>> multiple
> >>>>>>>>>>> brokers on the same JVM but a single instance per VM, so each
> > one
> >>>>>> has a
> >>>>>>>>>>> dedicated JVM and VM
> >>>>>>>>>>> Thanks
> >>>>>>>>>>> Roy
> >>>>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>> I'm not entirely sure if the configuration you want is
> possible.
> >>> You
> >>>>>>>>>>> might
> >>>>>>>>>>> try using the "group-name" element in the "master" or "slave"
> >>>>> element
> >>>>>>>> of
> >>>>>>>>>>> "colocated." Only servers with the same group-name will pair
> >>>>>> together.
> >>>>>>>>>>> Aside from that I would actually recommend against using
> > colocated
> >>>>>>>>>>> brokers.
> >>>>>>>>>>> The original use-case for this functionality was very early
> > cloud
> >>>>>>>>>>> infrastructure where durable, attached storage was not readily
> >>>>>>>> available.
> >>>>>>>>>>> However, since then most (if not all) cloud environments
> support
> >>>>>>>> durable
> >>>>>>>>>>> storage separate from the broker so that if the broker goes
> > down a
> >>>>>> new,
> >>>>>>>>>>> identical broker can be spun-up relatively quickly and attached
> > to
> >>>>>> the
> >>>>>>>>>>> same
> >>>>>>>>>>> storage. This provides functional high availability without the
> >>> need
> >>>>>>>> for
> >>>>>>>>>>> any idle backups or replication of any kind which functionally
> >>>>>>>> nullifies
> >>>>>>>>>>> this feature.
> >>>>>>>>>>> Additionally, it turns out that (surprise!) configuring &
> > running
> >>>>>>>>>>> multiple
> >>>>>>>>>>> brokers in the same JVM is difficult and error-prone not to
> >>> mention
> >>>>>> the
> >>>>>>>>>>> complication of dynamically coordinating the acquisition of
> >>> backups
> >>>>>> in
> >>>>>>>> a
> >>>>>>>>>>> running cluster and protecting against split-brain.
> >>>>>>>>>>> Justin
> >>>>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <
> > roy_cohen@hotmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>> Hello everyone
> >>>>>>>>>>> We have a setup of three Artemis brokers (very old version
> don’t
> >>> ask
> >>>>>>>> :))
> >>>>>>>>>>> We would like to configure the co located backups such that the
> >>>>>> backups
> >>>>>>>>>>> are sent in this order:
> >>>>>>>>>>> Broker01 -> Broker02
> >>>>>>>>>>> Broker02 -> Broker03
> >>>>>>>>>>> Broker03 -> Broker01
> >>>>>>>>>>> I was reading on co located backups here:
> >
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>>>>>>>>> however not sure I fully understand how to configure the xml
> >>> section
> >>>>>> to
> >>>>>>>>>>> achieve that.
> >>>>>>>>>>> Shall I add excludes in each broker, i.e.
> >>>>>>>>>>> <colocated>
> >>>>>>>>>>> <excludes>
> >>>>>>>>>>>  <connector-ref>...</connector-ref>
> >>>>>>>>>>> </excludes>
> >>>>>>>>>>> Any help would be appreciated.
> >>>>>>>>>>> Many thanks in advance !
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Gotcha.

Suppose that mail group, so I hereby requesting an invite please :)

On 28 Mar 2023, at 16:40, Justin Bertram <jb...@apache.org> wrote:

You don't need an @apache.org email in order to join the ASF Slack. You
just need to request an invitation. This is noted on the website [1] (at
the bottom):

   If you want an invitation to the ActiveMQ Slack channel simply send a
request to the users mailing list.

Most of the folks in there don't have an @apache.org email.


Justin

[1] https://activemq.apache.org/contact

On Tue, Mar 28, 2023 at 10:31 AM Roy Cohen <ro...@hotmail.com> wrote:

> Understood and yes there’s a reason.
> 
> Don’t I need @apache.org email in order to join that Slack workspace ?
> 
> 
>> On 28 Mar 2023, at 15:52, Justin Bertram <jb...@apache.org> wrote:
>> Typically we like to keep everything on the list so the whole community
> can
>> benefit from the discussion. However, if there's a specific reason that
>> privacy is a concern then you can email me directly or you can find me on
>> the ASF Slack in #activemq.
>> Justin
>> On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com> wrote:
>>> Hi Justin
>>> Would it be able to reach out to you directly to discuss a couple of
>>> points around the discussion below privately ?
>>> Thanks
>>> Roy
>>>> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
>>>> You have indeed ! :)
>>>> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
>>>> I wrote that on a completely different thread [1] related to MQTT
>>> retained
>>>> messages in a cluster. It is not related to this thread or your issue
>>>> generally.
>>>> Justin
>>>> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
>>>> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com>
> wrote:
>>>>> To quote:
>>>>> “This functionality isn't supported, and while it may be technically
>>>>> feasible to implement I'm not sure how much sense it makes overall.”
>>>>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
>>>>> I'm not sure where I may have indicated that either one of those
> things
>>>>> isn't supported.
>>>>> In any case, you can do either.
>>>>> Justin
>>>>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>>>> Just to be clear: When you say “isn’t supported” do you mean a third
>>>>>> broker or co located backups when running each broker on its own VM ?
>>>>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>>>>>>> Will do Justin and many thanks for all the additional details which
> I
>>>>>> will certainly bring forward internally, much appreciated
>>>>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org>
> wrote:
>>>>>>> I recently added a new section to the clustering documentation
>>>>> regarding
>>>>>>> things to keep in mind regarding performance [1].
>>>>>>> Also, it's worth noting that often the bottleneck in messaging is
> not
>>>>> the
>>>>>>> broker itself but rather the consumer(s). It might be worth ensuring
>>>>> that
>>>>>>> the bottleneck really is the broker. As noted in the new
> documentation
>>>>>> [1],
>>>>>>> adding brokers to a cluster can actually *reduce* throughput in
>>> certain
>>>>>>> circumstances.
>>>>>>> Let me know if using group-name works for you.
>>>>>>> Justin
>>>>>>> [1]
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>>>>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
>>>>>> wrote:
>>>>>>>> I haven’t tried he group-name yet.
>>>>>>>> With regards to the third broker: The architects believe it’ll
>>> improve
>>>>>>>> performance given the amount of messages the brokers need to
> process
>>>>> (in
>>>>>>>> other words “throw more resources at it…”)
>>>>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
>>> wrote:
>>>>>>>>>> What would you suggest is to do ?
>>>>>>>>> Did you try my previous suggestion already (i.e. using the
>>>>> "group-name"
>>>>>>>>> element in the "master" or "slave" element of "colocated")?
>>>>>>>>> Aside from that, do you know why you were asked to add another
>>> broker?
>>>>>>>>> Depending on the reason it may not be a good solution.
>>>>>>>>> Justin
>>>>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <roy_cohen@hotmail.com
>>>>>>>> wrote:
>>>>>>>>>> Hi Justin
>>>>>>>>>> It is a good question I honestly don’t have the answer for. I
>>>>>> inherited
>>>>>>>>>> this configuration and was asked to add a third broker and to
>>> ensure
>>>>>>>> the co
>>>>>>>>>> located backups are being done in such a way that each broker
>>> points
>>>>>> on
>>>>>>>>>> another. Perhaps those who asked for it don’t fully understand
>>>>> Artemis
>>>>>>>> ! :)
>>>>>>>>>> That said, those co located backup on the existing setup with two
>>>>>>>> brokers
>>>>>>>>>> do work as we have been enabled to recover lost messages in the
>>> past.
>>>>>> So
>>>>>>>>>> even not optimal, technically it does work ?
>>>>>>>>>> I can only imagine that those who initially designed it about 5
>>> years
>>>>>>>> ago
>>>>>>>>>> did not use a shared storage to avoid latency.
>>>>>>>>>> What would you suggest is to do ?
>>>>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
>>>>> wrote:
>>>>>>>>>> Your screenshot didn't come through the list.
>>>>>>>>>> In any case, I'm pretty confused at this point. You're clearly
>>> using
>>>>> a
>>>>>>>>>> colocated configuration that will request a backup from another
>>>>> broker
>>>>>>>> in
>>>>>>>>>> the cluster, but you say you're not running multiple brokers in
> the
>>>>>> same
>>>>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
>>> what
>>>>>>>> are
>>>>>>>>>> you using the colocated configuration for? The whole point of the
>>>>>>>> colocated
>>>>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
>>>>>> primary
>>>>>>>>>> broker and also a backup broker for another primary in the
>>> cluster).
>>>>>>>>>> Justin
>>>>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <
> roy_cohen@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>>>> I don’t believe we are.
>>>>>>>>>>> So assume three Virtual Machines on Azure.
>>>>>>>>>>> Each VM runs one Artemis broker
>>>>>>>>>>> All of their ha policy section on all three brokers look like
>>> that:
>>>>>>>>>>> <ha-policy>
>>>>>>>>>>> <replication>
>>>>>>>>>>>  <colocated>
>>>>>>>>>>>    <max-backups>1</max-backups>
>>>>>>>>>>>    <request-backup>true</request-backup>
>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>>>>>>    <excludes>
>>>>>>>>>>>      <connector-ref>my-connector</connector-ref>
>>>>>>>>>>>      <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>>>>>>    </excludes>
>>>>>>>>>>>    <master>
>>>>>>>>>>>      <check-for-live-server>true</check-for-live-server>
>>>>>>>>>>>    </master>
>>>>>>>>>>>    <slave>
>>>>>>>>>>>      <allow-failback>true</allow-failback>
>>>>>>>>>>>      <restart-backup>true</restart-backup>
>>>>>>>>>>>      <scale-down/>
>>>>>>>>>>>    </slave>
>>>>>>>>>>>  </colocated>
>>>>>>>>>>> </replication>
>>>>>>>>>>> </ha-policy>
>>>>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
>>>>>> wrote:
>>>>>>>>>>> We are not running multiple brokers on the same JVM but a single
>>>>>>>> instance
>>>>>>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>>>>>>> Based on your previous message I was under the impression you
> were
>>>>>>>> using
>>>>>>>>>>> the "colocated" feature. *If* you're using this then you
>>> definitely
>>>>>> are
>>>>>>>>>>> running multiple brokers in the same JVM because that's
> precisely
>>>>>> what
>>>>>>>>>> that
>>>>>>>>>>> feature does. It runs a primary and a backup broker in the *same
>>>>>> JVM*.
>>>>>>>> If
>>>>>>>>>>> you aren't using a "colocated" configuration then I'm not sure
>>> what
>>>>>> the
>>>>>>>>>>> original question is about. Can you clarify?
>>>>>>>>>>> Justin
>>>>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <
> roy_cohen@hotmail.com
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi Justin
>>>>>>>>>>> Thank you for your input.
>>>>>>>>>>> Sorry, should have been clearer on our setup - We are not
> running
>>>>>>>>>> multiple
>>>>>>>>>>> brokers on the same JVM but a single instance per VM, so each
> one
>>>>>> has a
>>>>>>>>>>> dedicated JVM and VM
>>>>>>>>>>> Thanks
>>>>>>>>>>> Roy
>>>>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
>>>>>> wrote:
>>>>>>>>>>> I'm not entirely sure if the configuration you want is possible.
>>> You
>>>>>>>>>>> might
>>>>>>>>>>> try using the "group-name" element in the "master" or "slave"
>>>>> element
>>>>>>>> of
>>>>>>>>>>> "colocated." Only servers with the same group-name will pair
>>>>>> together.
>>>>>>>>>>> Aside from that I would actually recommend against using
> colocated
>>>>>>>>>>> brokers.
>>>>>>>>>>> The original use-case for this functionality was very early
> cloud
>>>>>>>>>>> infrastructure where durable, attached storage was not readily
>>>>>>>> available.
>>>>>>>>>>> However, since then most (if not all) cloud environments support
>>>>>>>> durable
>>>>>>>>>>> storage separate from the broker so that if the broker goes
> down a
>>>>>> new,
>>>>>>>>>>> identical broker can be spun-up relatively quickly and attached
> to
>>>>>> the
>>>>>>>>>>> same
>>>>>>>>>>> storage. This provides functional high availability without the
>>> need
>>>>>>>> for
>>>>>>>>>>> any idle backups or replication of any kind which functionally
>>>>>>>> nullifies
>>>>>>>>>>> this feature.
>>>>>>>>>>> Additionally, it turns out that (surprise!) configuring &
> running
>>>>>>>>>>> multiple
>>>>>>>>>>> brokers in the same JVM is difficult and error-prone not to
>>> mention
>>>>>> the
>>>>>>>>>>> complication of dynamically coordinating the acquisition of
>>> backups
>>>>>> in
>>>>>>>> a
>>>>>>>>>>> running cluster and protecting against split-brain.
>>>>>>>>>>> Justin
>>>>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <
> roy_cohen@hotmail.com>
>>>>>>>> wrote:
>>>>>>>>>>> Hello everyone
>>>>>>>>>>> We have a setup of three Artemis brokers (very old version don’t
>>> ask
>>>>>>>> :))
>>>>>>>>>>> We would like to configure the co located backups such that the
>>>>>> backups
>>>>>>>>>>> are sent in this order:
>>>>>>>>>>> Broker01 -> Broker02
>>>>>>>>>>> Broker02 -> Broker03
>>>>>>>>>>> Broker03 -> Broker01
>>>>>>>>>>> I was reading on co located backups here:
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>>>>>>> however not sure I fully understand how to configure the xml
>>> section
>>>>>> to
>>>>>>>>>>> achieve that.
>>>>>>>>>>> Shall I add excludes in each broker, i.e.
>>>>>>>>>>> <colocated>
>>>>>>>>>>> <excludes>
>>>>>>>>>>>  <connector-ref>...</connector-ref>
>>>>>>>>>>> </excludes>
>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>> Many thanks in advance !

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
You don't need an @apache.org email in order to join the ASF Slack. You
just need to request an invitation. This is noted on the website [1] (at
the bottom):

    If you want an invitation to the ActiveMQ Slack channel simply send a
request to the users mailing list.

Most of the folks in there don't have an @apache.org email.


Justin

[1] https://activemq.apache.org/contact

On Tue, Mar 28, 2023 at 10:31 AM Roy Cohen <ro...@hotmail.com> wrote:

> Understood and yes there’s a reason.
>
> Don’t I need @apache.org email in order to join that Slack workspace ?
>
>
> > On 28 Mar 2023, at 15:52, Justin Bertram <jb...@apache.org> wrote:
> >
> > Typically we like to keep everything on the list so the whole community
> can
> > benefit from the discussion. However, if there's a specific reason that
> > privacy is a concern then you can email me directly or you can find me on
> > the ASF Slack in #activemq.
> >
> >
> > Justin
> >
> > On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> Hi Justin
> >>
> >> Would it be able to reach out to you directly to discuss a couple of
> >> points around the discussion below privately ?
> >>
> >> Thanks
> >> Roy
> >>
> >>
> >>> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
> >>>
> >>> You have indeed ! :)
> >>>
> >>> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
> >>>
> >>> I wrote that on a completely different thread [1] related to MQTT
> >> retained
> >>> messages in a cluster. It is not related to this thread or your issue
> >>> generally.
> >>>
> >>>
> >>> Justin
> >>>
> >>> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
> >>>
> >>> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>>
> >>>> To quote:
> >>>>
> >>>> “This functionality isn't supported, and while it may be technically
> >>>> feasible to implement I'm not sure how much sense it makes overall.”
> >>>>
> >>>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
> >>>>
> >>>> I'm not sure where I may have indicated that either one of those
> things
> >>>> isn't supported.
> >>>>
> >>>> In any case, you can do either.
> >>>>
> >>>>
> >>>> Justin
> >>>>
> >>>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
> >> wrote:
> >>>>
> >>>>> Just to be clear: When you say “isn’t supported” do you mean a third
> >>>>> broker or co located backups when running each broker on its own VM ?
> >>>>>
> >>>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> >>>>>>
> >>>>>> Will do Justin and many thanks for all the additional details which
> I
> >>>>> will certainly bring forward internally, much appreciated
> >>>>>>
> >>>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>>>
> >>>>>> I recently added a new section to the clustering documentation
> >>>> regarding
> >>>>>> things to keep in mind regarding performance [1].
> >>>>>>
> >>>>>> Also, it's worth noting that often the bottleneck in messaging is
> not
> >>>> the
> >>>>>> broker itself but rather the consumer(s). It might be worth ensuring
> >>>> that
> >>>>>> the bottleneck really is the broker. As noted in the new
> documentation
> >>>>> [1],
> >>>>>> adding brokers to a cluster can actually *reduce* throughput in
> >> certain
> >>>>>> circumstances.
> >>>>>>
> >>>>>> Let me know if using group-name works for you.
> >>>>>>
> >>>>>>
> >>>>>> Justin
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>>
> >>
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> >>>>>>
> >>>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> I haven’t tried he group-name yet.
> >>>>>>>
> >>>>>>> With regards to the third broker: The architects believe it’ll
> >> improve
> >>>>>>> performance given the amount of messages the brokers need to
> process
> >>>> (in
> >>>>>>> other words “throw more resources at it…”)
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
> >> wrote:
> >>>>>>>>
> >>>>>>>>> What would you suggest is to do ?
> >>>>>>>>
> >>>>>>>> Did you try my previous suggestion already (i.e. using the
> >>>> "group-name"
> >>>>>>>> element in the "master" or "slave" element of "colocated")?
> >>>>>>>>
> >>>>>>>> Aside from that, do you know why you were asked to add another
> >> broker?
> >>>>>>>> Depending on the reason it may not be a good solution.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Justin
> >>>>>>>>
> >>>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <roy_cohen@hotmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Justin
> >>>>>>>>>
> >>>>>>>>> It is a good question I honestly don’t have the answer for. I
> >>>>> inherited
> >>>>>>>>> this configuration and was asked to add a third broker and to
> >> ensure
> >>>>>>> the co
> >>>>>>>>> located backups are being done in such a way that each broker
> >> points
> >>>>> on
> >>>>>>>>> another. Perhaps those who asked for it don’t fully understand
> >>>> Artemis
> >>>>>>> ! :)
> >>>>>>>>>
> >>>>>>>>> That said, those co located backup on the existing setup with two
> >>>>>>> brokers
> >>>>>>>>> do work as we have been enabled to recover lost messages in the
> >> past.
> >>>>> So
> >>>>>>>>> even not optimal, technically it does work ?
> >>>>>>>>>
> >>>>>>>>> I can only imagine that those who initially designed it about 5
> >> years
> >>>>>>> ago
> >>>>>>>>> did not use a shared storage to avoid latency.
> >>>>>>>>>
> >>>>>>>>> What would you suggest is to do ?
> >>>>>>>>>
> >>>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
> >>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Your screenshot didn't come through the list.
> >>>>>>>>>
> >>>>>>>>> In any case, I'm pretty confused at this point. You're clearly
> >> using
> >>>> a
> >>>>>>>>> colocated configuration that will request a backup from another
> >>>> broker
> >>>>>>> in
> >>>>>>>>> the cluster, but you say you're not running multiple brokers in
> the
> >>>>> same
> >>>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
> >> what
> >>>>>>> are
> >>>>>>>>> you using the colocated configuration for? The whole point of the
> >>>>>>> colocated
> >>>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
> >>>>> primary
> >>>>>>>>> broker and also a backup broker for another primary in the
> >> cluster).
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Justin
> >>>>>>>>>
> >>>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <
> roy_cohen@hotmail.com>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I don’t believe we are.
> >>>>>>>>>>
> >>>>>>>>>> So assume three Virtual Machines on Azure.
> >>>>>>>>>>
> >>>>>>>>>> Each VM runs one Artemis broker
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> All of their ha policy section on all three brokers look like
> >> that:
> >>>>>>>>>>
> >>>>>>>>>> <ha-policy>
> >>>>>>>>>> <replication>
> >>>>>>>>>>   <colocated>
> >>>>>>>>>>     <max-backups>1</max-backups>
> >>>>>>>>>>     <request-backup>true</request-backup>
> >>>>>>>>>>
> >>>>>>>>>>
> >> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>>>>>>>>     <excludes>
> >>>>>>>>>>       <connector-ref>my-connector</connector-ref>
> >>>>>>>>>>       <connector-ref>thishostname.mydomain</connector-ref>
> >>>>>>>>>>     </excludes>
> >>>>>>>>>>     <master>
> >>>>>>>>>>       <check-for-live-server>true</check-for-live-server>
> >>>>>>>>>>     </master>
> >>>>>>>>>>     <slave>
> >>>>>>>>>>       <allow-failback>true</allow-failback>
> >>>>>>>>>>       <restart-backup>true</restart-backup>
> >>>>>>>>>>       <scale-down/>
> >>>>>>>>>>     </slave>
> >>>>>>>>>>   </colocated>
> >>>>>>>>>> </replication>
> >>>>>>>>>> </ha-policy>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> We are not running multiple brokers on the same JVM but a single
> >>>>>>> instance
> >>>>>>>>>>
> >>>>>>>>>> per VM, so each one has a dedicated JVM and VM
> >>>>>>>>>>
> >>>>>>>>>> Based on your previous message I was under the impression you
> were
> >>>>>>> using
> >>>>>>>>>> the "colocated" feature. *If* you're using this then you
> >> definitely
> >>>>> are
> >>>>>>>>>> running multiple brokers in the same JVM because that's
> precisely
> >>>>> what
> >>>>>>>>> that
> >>>>>>>>>> feature does. It runs a primary and a backup broker in the *same
> >>>>> JVM*.
> >>>>>>> If
> >>>>>>>>>> you aren't using a "colocated" configuration then I'm not sure
> >> what
> >>>>> the
> >>>>>>>>>> original question is about. Can you clarify?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Justin
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <
> roy_cohen@hotmail.com
> >>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi Justin
> >>>>>>>>>>
> >>>>>>>>>> Thank you for your input.
> >>>>>>>>>>
> >>>>>>>>>> Sorry, should have been clearer on our setup - We are not
> running
> >>>>>>>>> multiple
> >>>>>>>>>> brokers on the same JVM but a single instance per VM, so each
> one
> >>>>> has a
> >>>>>>>>>> dedicated JVM and VM
> >>>>>>>>>>
> >>>>>>>>>> Thanks
> >>>>>>>>>> Roy
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I'm not entirely sure if the configuration you want is possible.
> >> You
> >>>>>>>>>>
> >>>>>>>>>> might
> >>>>>>>>>>
> >>>>>>>>>> try using the "group-name" element in the "master" or "slave"
> >>>> element
> >>>>>>> of
> >>>>>>>>>> "colocated." Only servers with the same group-name will pair
> >>>>> together.
> >>>>>>>>>>
> >>>>>>>>>> Aside from that I would actually recommend against using
> colocated
> >>>>>>>>>>
> >>>>>>>>>> brokers.
> >>>>>>>>>>
> >>>>>>>>>> The original use-case for this functionality was very early
> cloud
> >>>>>>>>>> infrastructure where durable, attached storage was not readily
> >>>>>>> available.
> >>>>>>>>>> However, since then most (if not all) cloud environments support
> >>>>>>> durable
> >>>>>>>>>> storage separate from the broker so that if the broker goes
> down a
> >>>>> new,
> >>>>>>>>>> identical broker can be spun-up relatively quickly and attached
> to
> >>>>> the
> >>>>>>>>>>
> >>>>>>>>>> same
> >>>>>>>>>>
> >>>>>>>>>> storage. This provides functional high availability without the
> >> need
> >>>>>>> for
> >>>>>>>>>> any idle backups or replication of any kind which functionally
> >>>>>>> nullifies
> >>>>>>>>>> this feature.
> >>>>>>>>>>
> >>>>>>>>>> Additionally, it turns out that (surprise!) configuring &
> running
> >>>>>>>>>>
> >>>>>>>>>> multiple
> >>>>>>>>>>
> >>>>>>>>>> brokers in the same JVM is difficult and error-prone not to
> >> mention
> >>>>> the
> >>>>>>>>>> complication of dynamically coordinating the acquisition of
> >> backups
> >>>>> in
> >>>>>>> a
> >>>>>>>>>> running cluster and protecting against split-brain.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Justin
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <
> roy_cohen@hotmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hello everyone
> >>>>>>>>>>
> >>>>>>>>>> We have a setup of three Artemis brokers (very old version don’t
> >> ask
> >>>>>>> :))
> >>>>>>>>>>
> >>>>>>>>>> We would like to configure the co located backups such that the
> >>>>> backups
> >>>>>>>>>> are sent in this order:
> >>>>>>>>>>
> >>>>>>>>>> Broker01 -> Broker02
> >>>>>>>>>> Broker02 -> Broker03
> >>>>>>>>>> Broker03 -> Broker01
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I was reading on co located backups here:
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>>>>>>>>
> >>>>>>>>>> however not sure I fully understand how to configure the xml
> >> section
> >>>>> to
> >>>>>>>>>> achieve that.
> >>>>>>>>>>
> >>>>>>>>>> Shall I add excludes in each broker, i.e.
> >>>>>>>>>>
> >>>>>>>>>> <colocated>
> >>>>>>>>>> <excludes>
> >>>>>>>>>>   <connector-ref>...</connector-ref>
> >>>>>>>>>> </excludes>
> >>>>>>>>>>
> >>>>>>>>>> Any help would be appreciated.
> >>>>>>>>>>
> >>>>>>>>>> Many thanks in advance !
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Understood and yes there’s a reason.

Don’t I need @apache.org email in order to join that Slack workspace ?


> On 28 Mar 2023, at 15:52, Justin Bertram <jb...@apache.org> wrote:
> 
> Typically we like to keep everything on the list so the whole community can
> benefit from the discussion. However, if there's a specific reason that
> privacy is a concern then you can email me directly or you can find me on
> the ASF Slack in #activemq.
> 
> 
> Justin
> 
> On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Hi Justin
>> 
>> Would it be able to reach out to you directly to discuss a couple of
>> points around the discussion below privately ?
>> 
>> Thanks
>> Roy
>> 
>> 
>>> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
>>> 
>>> You have indeed ! :)
>>> 
>>> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>> I wrote that on a completely different thread [1] related to MQTT
>> retained
>>> messages in a cluster. It is not related to this thread or your issue
>>> generally.
>>> 
>>> 
>>> Justin
>>> 
>>> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
>>> 
>>> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com> wrote:
>>> 
>>>> To quote:
>>>> 
>>>> “This functionality isn't supported, and while it may be technically
>>>> feasible to implement I'm not sure how much sense it makes overall.”
>>>> 
>>>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>> I'm not sure where I may have indicated that either one of those things
>>>> isn't supported.
>>>> 
>>>> In any case, you can do either.
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>>> 
>>>>> Just to be clear: When you say “isn’t supported” do you mean a third
>>>>> broker or co located backups when running each broker on its own VM ?
>>>>> 
>>>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>>>>>> 
>>>>>> Will do Justin and many thanks for all the additional details which I
>>>>> will certainly bring forward internally, much appreciated
>>>>>> 
>>>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
>>>>>> 
>>>>>> I recently added a new section to the clustering documentation
>>>> regarding
>>>>>> things to keep in mind regarding performance [1].
>>>>>> 
>>>>>> Also, it's worth noting that often the bottleneck in messaging is not
>>>> the
>>>>>> broker itself but rather the consumer(s). It might be worth ensuring
>>>> that
>>>>>> the bottleneck really is the broker. As noted in the new documentation
>>>>> [1],
>>>>>> adding brokers to a cluster can actually *reduce* throughput in
>> certain
>>>>>> circumstances.
>>>>>> 
>>>>>> Let me know if using group-name works for you.
>>>>>> 
>>>>>> 
>>>>>> Justin
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>>> 
>>>> 
>> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>>>>>> 
>>>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> I haven’t tried he group-name yet.
>>>>>>> 
>>>>>>> With regards to the third broker: The architects believe it’ll
>> improve
>>>>>>> performance given the amount of messages the brokers need to process
>>>> (in
>>>>>>> other words “throw more resources at it…”)
>>>>>>> 
>>>>>>> 
>>>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
>> wrote:
>>>>>>>> 
>>>>>>>>> What would you suggest is to do ?
>>>>>>>> 
>>>>>>>> Did you try my previous suggestion already (i.e. using the
>>>> "group-name"
>>>>>>>> element in the "master" or "slave" element of "colocated")?
>>>>>>>> 
>>>>>>>> Aside from that, do you know why you were asked to add another
>> broker?
>>>>>>>> Depending on the reason it may not be a good solution.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Justin
>>>>>>>> 
>>>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Justin
>>>>>>>>> 
>>>>>>>>> It is a good question I honestly don’t have the answer for. I
>>>>> inherited
>>>>>>>>> this configuration and was asked to add a third broker and to
>> ensure
>>>>>>> the co
>>>>>>>>> located backups are being done in such a way that each broker
>> points
>>>>> on
>>>>>>>>> another. Perhaps those who asked for it don’t fully understand
>>>> Artemis
>>>>>>> ! :)
>>>>>>>>> 
>>>>>>>>> That said, those co located backup on the existing setup with two
>>>>>>> brokers
>>>>>>>>> do work as we have been enabled to recover lost messages in the
>> past.
>>>>> So
>>>>>>>>> even not optimal, technically it does work ?
>>>>>>>>> 
>>>>>>>>> I can only imagine that those who initially designed it about 5
>> years
>>>>>>> ago
>>>>>>>>> did not use a shared storage to avoid latency.
>>>>>>>>> 
>>>>>>>>> What would you suggest is to do ?
>>>>>>>>> 
>>>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
>>>> wrote:
>>>>>>>>> 
>>>>>>>>> Your screenshot didn't come through the list.
>>>>>>>>> 
>>>>>>>>> In any case, I'm pretty confused at this point. You're clearly
>> using
>>>> a
>>>>>>>>> colocated configuration that will request a backup from another
>>>> broker
>>>>>>> in
>>>>>>>>> the cluster, but you say you're not running multiple brokers in the
>>>>> same
>>>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
>> what
>>>>>>> are
>>>>>>>>> you using the colocated configuration for? The whole point of the
>>>>>>> colocated
>>>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
>>>>> primary
>>>>>>>>> broker and also a backup broker for another primary in the
>> cluster).
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Justin
>>>>>>>>> 
>>>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> I don’t believe we are.
>>>>>>>>>> 
>>>>>>>>>> So assume three Virtual Machines on Azure.
>>>>>>>>>> 
>>>>>>>>>> Each VM runs one Artemis broker
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> All of their ha policy section on all three brokers look like
>> that:
>>>>>>>>>> 
>>>>>>>>>> <ha-policy>
>>>>>>>>>> <replication>
>>>>>>>>>>   <colocated>
>>>>>>>>>>     <max-backups>1</max-backups>
>>>>>>>>>>     <request-backup>true</request-backup>
>>>>>>>>>> 
>>>>>>>>>> 
>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>>>>>     <excludes>
>>>>>>>>>>       <connector-ref>my-connector</connector-ref>
>>>>>>>>>>       <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>>>>>     </excludes>
>>>>>>>>>>     <master>
>>>>>>>>>>       <check-for-live-server>true</check-for-live-server>
>>>>>>>>>>     </master>
>>>>>>>>>>     <slave>
>>>>>>>>>>       <allow-failback>true</allow-failback>
>>>>>>>>>>       <restart-backup>true</restart-backup>
>>>>>>>>>>       <scale-down/>
>>>>>>>>>>     </slave>
>>>>>>>>>>   </colocated>
>>>>>>>>>> </replication>
>>>>>>>>>> </ha-policy>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> We are not running multiple brokers on the same JVM but a single
>>>>>>> instance
>>>>>>>>>> 
>>>>>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>>>>>> 
>>>>>>>>>> Based on your previous message I was under the impression you were
>>>>>>> using
>>>>>>>>>> the "colocated" feature. *If* you're using this then you
>> definitely
>>>>> are
>>>>>>>>>> running multiple brokers in the same JVM because that's precisely
>>>>> what
>>>>>>>>> that
>>>>>>>>>> feature does. It runs a primary and a backup broker in the *same
>>>>> JVM*.
>>>>>>> If
>>>>>>>>>> you aren't using a "colocated" configuration then I'm not sure
>> what
>>>>> the
>>>>>>>>>> original question is about. Can you clarify?
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Justin
>>>>>>>>>> 
>>>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <roy_cohen@hotmail.com
>>> 
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hi Justin
>>>>>>>>>> 
>>>>>>>>>> Thank you for your input.
>>>>>>>>>> 
>>>>>>>>>> Sorry, should have been clearer on our setup - We are not running
>>>>>>>>> multiple
>>>>>>>>>> brokers on the same JVM but a single instance per VM, so each one
>>>>> has a
>>>>>>>>>> dedicated JVM and VM
>>>>>>>>>> 
>>>>>>>>>> Thanks
>>>>>>>>>> Roy
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I'm not entirely sure if the configuration you want is possible.
>> You
>>>>>>>>>> 
>>>>>>>>>> might
>>>>>>>>>> 
>>>>>>>>>> try using the "group-name" element in the "master" or "slave"
>>>> element
>>>>>>> of
>>>>>>>>>> "colocated." Only servers with the same group-name will pair
>>>>> together.
>>>>>>>>>> 
>>>>>>>>>> Aside from that I would actually recommend against using colocated
>>>>>>>>>> 
>>>>>>>>>> brokers.
>>>>>>>>>> 
>>>>>>>>>> The original use-case for this functionality was very early cloud
>>>>>>>>>> infrastructure where durable, attached storage was not readily
>>>>>>> available.
>>>>>>>>>> However, since then most (if not all) cloud environments support
>>>>>>> durable
>>>>>>>>>> storage separate from the broker so that if the broker goes down a
>>>>> new,
>>>>>>>>>> identical broker can be spun-up relatively quickly and attached to
>>>>> the
>>>>>>>>>> 
>>>>>>>>>> same
>>>>>>>>>> 
>>>>>>>>>> storage. This provides functional high availability without the
>> need
>>>>>>> for
>>>>>>>>>> any idle backups or replication of any kind which functionally
>>>>>>> nullifies
>>>>>>>>>> this feature.
>>>>>>>>>> 
>>>>>>>>>> Additionally, it turns out that (surprise!) configuring & running
>>>>>>>>>> 
>>>>>>>>>> multiple
>>>>>>>>>> 
>>>>>>>>>> brokers in the same JVM is difficult and error-prone not to
>> mention
>>>>> the
>>>>>>>>>> complication of dynamically coordinating the acquisition of
>> backups
>>>>> in
>>>>>>> a
>>>>>>>>>> running cluster and protecting against split-brain.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Justin
>>>>>>>>>> 
>>>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Hello everyone
>>>>>>>>>> 
>>>>>>>>>> We have a setup of three Artemis brokers (very old version don’t
>> ask
>>>>>>> :))
>>>>>>>>>> 
>>>>>>>>>> We would like to configure the co located backups such that the
>>>>> backups
>>>>>>>>>> are sent in this order:
>>>>>>>>>> 
>>>>>>>>>> Broker01 -> Broker02
>>>>>>>>>> Broker02 -> Broker03
>>>>>>>>>> Broker03 -> Broker01
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> I was reading on co located backups here:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>>>>>> 
>>>>>>>>>> however not sure I fully understand how to configure the xml
>> section
>>>>> to
>>>>>>>>>> achieve that.
>>>>>>>>>> 
>>>>>>>>>> Shall I add excludes in each broker, i.e.
>>>>>>>>>> 
>>>>>>>>>> <colocated>
>>>>>>>>>> <excludes>
>>>>>>>>>>   <connector-ref>...</connector-ref>
>>>>>>>>>> </excludes>
>>>>>>>>>> 
>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>> 
>>>>>>>>>> Many thanks in advance !
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
Typically we like to keep everything on the list so the whole community can
benefit from the discussion. However, if there's a specific reason that
privacy is a concern then you can email me directly or you can find me on
the ASF Slack in #activemq.


Justin

On Tue, Mar 28, 2023 at 7:56 AM Roy Cohen <ro...@hotmail.com> wrote:

> Hi Justin
>
> Would it be able to reach out to you directly to discuss a couple of
> points around the discussion below privately ?
>
> Thanks
> Roy
>
>
> > On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
> >
> > You have indeed ! :)
> >
> > On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
> >
> > I wrote that on a completely different thread [1] related to MQTT
> retained
> > messages in a cluster. It is not related to this thread or your issue
> > generally.
> >
> >
> > Justin
> >
> > [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
> >
> > On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> To quote:
> >>
> >> “This functionality isn't supported, and while it may be technically
> >> feasible to implement I'm not sure how much sense it makes overall.”
> >>
> >> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
> >>
> >> I'm not sure where I may have indicated that either one of those things
> >> isn't supported.
> >>
> >> In any case, you can do either.
> >>
> >>
> >> Justin
> >>
> >> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>
> >>> Just to be clear: When you say “isn’t supported” do you mean a third
> >>> broker or co located backups when running each broker on its own VM ?
> >>>
> >>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> >>>>
> >>>> Will do Justin and many thanks for all the additional details which I
> >>> will certainly bring forward internally, much appreciated
> >>>>
> >>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
> >>>>
> >>>> I recently added a new section to the clustering documentation
> >> regarding
> >>>> things to keep in mind regarding performance [1].
> >>>>
> >>>> Also, it's worth noting that often the bottleneck in messaging is not
> >> the
> >>>> broker itself but rather the consumer(s). It might be worth ensuring
> >> that
> >>>> the bottleneck really is the broker. As noted in the new documentation
> >>> [1],
> >>>> adding brokers to a cluster can actually *reduce* throughput in
> certain
> >>>> circumstances.
> >>>>
> >>>> Let me know if using group-name works for you.
> >>>>
> >>>>
> >>>> Justin
> >>>>
> >>>> [1]
> >>>>
> >>>
> >>
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> >>>>
> >>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> >>> wrote:
> >>>>
> >>>>> I haven’t tried he group-name yet.
> >>>>>
> >>>>> With regards to the third broker: The architects believe it’ll
> improve
> >>>>> performance given the amount of messages the brokers need to process
> >> (in
> >>>>> other words “throw more resources at it…”)
> >>>>>
> >>>>>
> >>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> What would you suggest is to do ?
> >>>>>>
> >>>>>> Did you try my previous suggestion already (i.e. using the
> >> "group-name"
> >>>>>> element in the "master" or "slave" element of "colocated")?
> >>>>>>
> >>>>>> Aside from that, do you know why you were asked to add another
> broker?
> >>>>>> Depending on the reason it may not be a good solution.
> >>>>>>
> >>>>>>
> >>>>>> Justin
> >>>>>>
> >>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Hi Justin
> >>>>>>>
> >>>>>>> It is a good question I honestly don’t have the answer for. I
> >>> inherited
> >>>>>>> this configuration and was asked to add a third broker and to
> ensure
> >>>>> the co
> >>>>>>> located backups are being done in such a way that each broker
> points
> >>> on
> >>>>>>> another. Perhaps those who asked for it don’t fully understand
> >> Artemis
> >>>>> ! :)
> >>>>>>>
> >>>>>>> That said, those co located backup on the existing setup with two
> >>>>> brokers
> >>>>>>> do work as we have been enabled to recover lost messages in the
> past.
> >>> So
> >>>>>>> even not optimal, technically it does work ?
> >>>>>>>
> >>>>>>> I can only imagine that those who initially designed it about 5
> years
> >>>>> ago
> >>>>>>> did not use a shared storage to avoid latency.
> >>>>>>>
> >>>>>>> What would you suggest is to do ?
> >>>>>>>
> >>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>> Your screenshot didn't come through the list.
> >>>>>>>
> >>>>>>> In any case, I'm pretty confused at this point. You're clearly
> using
> >> a
> >>>>>>> colocated configuration that will request a backup from another
> >> broker
> >>>>> in
> >>>>>>> the cluster, but you say you're not running multiple brokers in the
> >>> same
> >>>>>>> JVM. If you aren't running multiple brokers in the same JVM then
> what
> >>>>> are
> >>>>>>> you using the colocated configuration for? The whole point of the
> >>>>> colocated
> >>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
> >>> primary
> >>>>>>> broker and also a backup broker for another primary in the
> cluster).
> >>>>>>>
> >>>>>>>
> >>>>>>> Justin
> >>>>>>>
> >>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> I don’t believe we are.
> >>>>>>>>
> >>>>>>>> So assume three Virtual Machines on Azure.
> >>>>>>>>
> >>>>>>>> Each VM runs one Artemis broker
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> All of their ha policy section on all three brokers look like
> that:
> >>>>>>>>
> >>>>>>>> <ha-policy>
> >>>>>>>>  <replication>
> >>>>>>>>    <colocated>
> >>>>>>>>      <max-backups>1</max-backups>
> >>>>>>>>      <request-backup>true</request-backup>
> >>>>>>>>
> >>>>>>>>
> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>>>>>>      <excludes>
> >>>>>>>>        <connector-ref>my-connector</connector-ref>
> >>>>>>>>        <connector-ref>thishostname.mydomain</connector-ref>
> >>>>>>>>      </excludes>
> >>>>>>>>      <master>
> >>>>>>>>        <check-for-live-server>true</check-for-live-server>
> >>>>>>>>      </master>
> >>>>>>>>      <slave>
> >>>>>>>>        <allow-failback>true</allow-failback>
> >>>>>>>>        <restart-backup>true</restart-backup>
> >>>>>>>>        <scale-down/>
> >>>>>>>>      </slave>
> >>>>>>>>    </colocated>
> >>>>>>>>  </replication>
> >>>>>>>> </ha-policy>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> We are not running multiple brokers on the same JVM but a single
> >>>>> instance
> >>>>>>>>
> >>>>>>>> per VM, so each one has a dedicated JVM and VM
> >>>>>>>>
> >>>>>>>> Based on your previous message I was under the impression you were
> >>>>> using
> >>>>>>>> the "colocated" feature. *If* you're using this then you
> definitely
> >>> are
> >>>>>>>> running multiple brokers in the same JVM because that's precisely
> >>> what
> >>>>>>> that
> >>>>>>>> feature does. It runs a primary and a backup broker in the *same
> >>> JVM*.
> >>>>> If
> >>>>>>>> you aren't using a "colocated" configuration then I'm not sure
> what
> >>> the
> >>>>>>>> original question is about. Can you clarify?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Justin
> >>>>>>>>
> >>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <roy_cohen@hotmail.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Justin
> >>>>>>>>
> >>>>>>>> Thank you for your input.
> >>>>>>>>
> >>>>>>>> Sorry, should have been clearer on our setup - We are not running
> >>>>>>> multiple
> >>>>>>>> brokers on the same JVM but a single instance per VM, so each one
> >>> has a
> >>>>>>>> dedicated JVM and VM
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Roy
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> >>> wrote:
> >>>>>>>>
> >>>>>>>> I'm not entirely sure if the configuration you want is possible.
> You
> >>>>>>>>
> >>>>>>>> might
> >>>>>>>>
> >>>>>>>> try using the "group-name" element in the "master" or "slave"
> >> element
> >>>>> of
> >>>>>>>> "colocated." Only servers with the same group-name will pair
> >>> together.
> >>>>>>>>
> >>>>>>>> Aside from that I would actually recommend against using colocated
> >>>>>>>>
> >>>>>>>> brokers.
> >>>>>>>>
> >>>>>>>> The original use-case for this functionality was very early cloud
> >>>>>>>> infrastructure where durable, attached storage was not readily
> >>>>> available.
> >>>>>>>> However, since then most (if not all) cloud environments support
> >>>>> durable
> >>>>>>>> storage separate from the broker so that if the broker goes down a
> >>> new,
> >>>>>>>> identical broker can be spun-up relatively quickly and attached to
> >>> the
> >>>>>>>>
> >>>>>>>> same
> >>>>>>>>
> >>>>>>>> storage. This provides functional high availability without the
> need
> >>>>> for
> >>>>>>>> any idle backups or replication of any kind which functionally
> >>>>> nullifies
> >>>>>>>> this feature.
> >>>>>>>>
> >>>>>>>> Additionally, it turns out that (surprise!) configuring & running
> >>>>>>>>
> >>>>>>>> multiple
> >>>>>>>>
> >>>>>>>> brokers in the same JVM is difficult and error-prone not to
> mention
> >>> the
> >>>>>>>> complication of dynamically coordinating the acquisition of
> backups
> >>> in
> >>>>> a
> >>>>>>>> running cluster and protecting against split-brain.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Justin
> >>>>>>>>
> >>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hello everyone
> >>>>>>>>
> >>>>>>>> We have a setup of three Artemis brokers (very old version don’t
> ask
> >>>>> :))
> >>>>>>>>
> >>>>>>>> We would like to configure the co located backups such that the
> >>> backups
> >>>>>>>> are sent in this order:
> >>>>>>>>
> >>>>>>>> Broker01 -> Broker02
> >>>>>>>> Broker02 -> Broker03
> >>>>>>>> Broker03 -> Broker01
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I was reading on co located backups here:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>>>>>>
> >>>>>>>> however not sure I fully understand how to configure the xml
> section
> >>> to
> >>>>>>>> achieve that.
> >>>>>>>>
> >>>>>>>> Shall I add excludes in each broker, i.e.
> >>>>>>>>
> >>>>>>>> <colocated>
> >>>>>>>> <excludes>
> >>>>>>>>    <connector-ref>...</connector-ref>
> >>>>>>>> </excludes>
> >>>>>>>>
> >>>>>>>> Any help would be appreciated.
> >>>>>>>>
> >>>>>>>> Many thanks in advance !
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Hi Justin

Would it be able to reach out to you directly to discuss a couple of points around the discussion below privately ?

Thanks
Roy


> On 28 Mar 2023, at 10:26, Roy Cohen <ro...@hotmail.com> wrote:
> 
> You have indeed ! :)
> 
> On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:
> 
> I wrote that on a completely different thread [1] related to MQTT retained
> messages in a cluster. It is not related to this thread or your issue
> generally.
> 
> 
> Justin
> 
> [1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf
> 
> On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> To quote:
>> 
>> “This functionality isn't supported, and while it may be technically
>> feasible to implement I'm not sure how much sense it makes overall.”
>> 
>> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
>> 
>> I'm not sure where I may have indicated that either one of those things
>> isn't supported.
>> 
>> In any case, you can do either.
>> 
>> 
>> Justin
>> 
>> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com> wrote:
>> 
>>> Just to be clear: When you say “isn’t supported” do you mean a third
>>> broker or co located backups when running each broker on its own VM ?
>>> 
>>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>>>> 
>>>> Will do Justin and many thanks for all the additional details which I
>>> will certainly bring forward internally, much appreciated
>>>> 
>>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>> I recently added a new section to the clustering documentation
>> regarding
>>>> things to keep in mind regarding performance [1].
>>>> 
>>>> Also, it's worth noting that often the bottleneck in messaging is not
>> the
>>>> broker itself but rather the consumer(s). It might be worth ensuring
>> that
>>>> the bottleneck really is the broker. As noted in the new documentation
>>> [1],
>>>> adding brokers to a cluster can actually *reduce* throughput in certain
>>>> circumstances.
>>>> 
>>>> Let me know if using group-name works for you.
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> [1]
>>>> 
>>> 
>> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>>>> 
>>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>> 
>>>>> I haven’t tried he group-name yet.
>>>>> 
>>>>> With regards to the third broker: The architects believe it’ll improve
>>>>> performance given the amount of messages the brokers need to process
>> (in
>>>>> other words “throw more resources at it…”)
>>>>> 
>>>>> 
>>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
>>>>>> 
>>>>>>> What would you suggest is to do ?
>>>>>> 
>>>>>> Did you try my previous suggestion already (i.e. using the
>> "group-name"
>>>>>> element in the "master" or "slave" element of "colocated")?
>>>>>> 
>>>>>> Aside from that, do you know why you were asked to add another broker?
>>>>>> Depending on the reason it may not be a good solution.
>>>>>> 
>>>>>> 
>>>>>> Justin
>>>>>> 
>>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> Hi Justin
>>>>>>> 
>>>>>>> It is a good question I honestly don’t have the answer for. I
>>> inherited
>>>>>>> this configuration and was asked to add a third broker and to ensure
>>>>> the co
>>>>>>> located backups are being done in such a way that each broker points
>>> on
>>>>>>> another. Perhaps those who asked for it don’t fully understand
>> Artemis
>>>>> ! :)
>>>>>>> 
>>>>>>> That said, those co located backup on the existing setup with two
>>>>> brokers
>>>>>>> do work as we have been enabled to recover lost messages in the past.
>>> So
>>>>>>> even not optimal, technically it does work ?
>>>>>>> 
>>>>>>> I can only imagine that those who initially designed it about 5 years
>>>>> ago
>>>>>>> did not use a shared storage to avoid latency.
>>>>>>> 
>>>>>>> What would you suggest is to do ?
>>>>>>> 
>>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> Your screenshot didn't come through the list.
>>>>>>> 
>>>>>>> In any case, I'm pretty confused at this point. You're clearly using
>> a
>>>>>>> colocated configuration that will request a backup from another
>> broker
>>>>> in
>>>>>>> the cluster, but you say you're not running multiple brokers in the
>>> same
>>>>>>> JVM. If you aren't running multiple brokers in the same JVM then what
>>>>> are
>>>>>>> you using the colocated configuration for? The whole point of the
>>>>> colocated
>>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
>>> primary
>>>>>>> broker and also a backup broker for another primary in the cluster).
>>>>>>> 
>>>>>>> 
>>>>>>> Justin
>>>>>>> 
>>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>>> I don’t believe we are.
>>>>>>>> 
>>>>>>>> So assume three Virtual Machines on Azure.
>>>>>>>> 
>>>>>>>> Each VM runs one Artemis broker
>>>>>>>> 
>>>>>>>> 
>>>>>>>> All of their ha policy section on all three brokers look like that:
>>>>>>>> 
>>>>>>>> <ha-policy>
>>>>>>>>  <replication>
>>>>>>>>    <colocated>
>>>>>>>>      <max-backups>1</max-backups>
>>>>>>>>      <request-backup>true</request-backup>
>>>>>>>> 
>>>>>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>>>      <excludes>
>>>>>>>>        <connector-ref>my-connector</connector-ref>
>>>>>>>>        <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>>>      </excludes>
>>>>>>>>      <master>
>>>>>>>>        <check-for-live-server>true</check-for-live-server>
>>>>>>>>      </master>
>>>>>>>>      <slave>
>>>>>>>>        <allow-failback>true</allow-failback>
>>>>>>>>        <restart-backup>true</restart-backup>
>>>>>>>>        <scale-down/>
>>>>>>>>      </slave>
>>>>>>>>    </colocated>
>>>>>>>>  </replication>
>>>>>>>> </ha-policy>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>> We are not running multiple brokers on the same JVM but a single
>>>>> instance
>>>>>>>> 
>>>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>>>> 
>>>>>>>> Based on your previous message I was under the impression you were
>>>>> using
>>>>>>>> the "colocated" feature. *If* you're using this then you definitely
>>> are
>>>>>>>> running multiple brokers in the same JVM because that's precisely
>>> what
>>>>>>> that
>>>>>>>> feature does. It runs a primary and a backup broker in the *same
>>> JVM*.
>>>>> If
>>>>>>>> you aren't using a "colocated" configuration then I'm not sure what
>>> the
>>>>>>>> original question is about. Can you clarify?
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Justin
>>>>>>>> 
>>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi Justin
>>>>>>>> 
>>>>>>>> Thank you for your input.
>>>>>>>> 
>>>>>>>> Sorry, should have been clearer on our setup - We are not running
>>>>>>> multiple
>>>>>>>> brokers on the same JVM but a single instance per VM, so each one
>>> has a
>>>>>>>> dedicated JVM and VM
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> Roy
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
>>> wrote:
>>>>>>>> 
>>>>>>>> I'm not entirely sure if the configuration you want is possible. You
>>>>>>>> 
>>>>>>>> might
>>>>>>>> 
>>>>>>>> try using the "group-name" element in the "master" or "slave"
>> element
>>>>> of
>>>>>>>> "colocated." Only servers with the same group-name will pair
>>> together.
>>>>>>>> 
>>>>>>>> Aside from that I would actually recommend against using colocated
>>>>>>>> 
>>>>>>>> brokers.
>>>>>>>> 
>>>>>>>> The original use-case for this functionality was very early cloud
>>>>>>>> infrastructure where durable, attached storage was not readily
>>>>> available.
>>>>>>>> However, since then most (if not all) cloud environments support
>>>>> durable
>>>>>>>> storage separate from the broker so that if the broker goes down a
>>> new,
>>>>>>>> identical broker can be spun-up relatively quickly and attached to
>>> the
>>>>>>>> 
>>>>>>>> same
>>>>>>>> 
>>>>>>>> storage. This provides functional high availability without the need
>>>>> for
>>>>>>>> any idle backups or replication of any kind which functionally
>>>>> nullifies
>>>>>>>> this feature.
>>>>>>>> 
>>>>>>>> Additionally, it turns out that (surprise!) configuring & running
>>>>>>>> 
>>>>>>>> multiple
>>>>>>>> 
>>>>>>>> brokers in the same JVM is difficult and error-prone not to mention
>>> the
>>>>>>>> complication of dynamically coordinating the acquisition of backups
>>> in
>>>>> a
>>>>>>>> running cluster and protecting against split-brain.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Justin
>>>>>>>> 
>>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hello everyone
>>>>>>>> 
>>>>>>>> We have a setup of three Artemis brokers (very old version don’t ask
>>>>> :))
>>>>>>>> 
>>>>>>>> We would like to configure the co located backups such that the
>>> backups
>>>>>>>> are sent in this order:
>>>>>>>> 
>>>>>>>> Broker01 -> Broker02
>>>>>>>> Broker02 -> Broker03
>>>>>>>> Broker03 -> Broker01
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I was reading on co located backups here:
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>>>> 
>>>>>>>> however not sure I fully understand how to configure the xml section
>>> to
>>>>>>>> achieve that.
>>>>>>>> 
>>>>>>>> Shall I add excludes in each broker, i.e.
>>>>>>>> 
>>>>>>>> <colocated>
>>>>>>>> <excludes>
>>>>>>>>    <connector-ref>...</connector-ref>
>>>>>>>> </excludes>
>>>>>>>> 
>>>>>>>> Any help would be appreciated.
>>>>>>>> 
>>>>>>>> Many thanks in advance !
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
You have indeed ! :)

On 27 Mar 2023, at 19:35, Justin Bertram <jb...@apache.org> wrote:

I wrote that on a completely different thread [1] related to MQTT retained
messages in a cluster. It is not related to this thread or your issue
generally.


Justin

[1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf

On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com> wrote:

> To quote:
> 
> “This functionality isn't supported, and while it may be technically
> feasible to implement I'm not sure how much sense it makes overall.”
> 
> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
> 
> I'm not sure where I may have indicated that either one of those things
> isn't supported.
> 
> In any case, you can do either.
> 
> 
> Justin
> 
> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Just to be clear: When you say “isn’t supported” do you mean a third
>> broker or co located backups when running each broker on its own VM ?
>> 
>>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>>> 
>>> Will do Justin and many thanks for all the additional details which I
>> will certainly bring forward internally, much appreciated
>>> 
>>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>> I recently added a new section to the clustering documentation
> regarding
>>> things to keep in mind regarding performance [1].
>>> 
>>> Also, it's worth noting that often the bottleneck in messaging is not
> the
>>> broker itself but rather the consumer(s). It might be worth ensuring
> that
>>> the bottleneck really is the broker. As noted in the new documentation
>> [1],
>>> adding brokers to a cluster can actually *reduce* throughput in certain
>>> circumstances.
>>> 
>>> Let me know if using group-name works for you.
>>> 
>>> 
>>> Justin
>>> 
>>> [1]
>>> 
>> 
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>>> 
>>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>> 
>>>> I haven’t tried he group-name yet.
>>>> 
>>>> With regards to the third broker: The architects believe it’ll improve
>>>> performance given the amount of messages the brokers need to process
> (in
>>>> other words “throw more resources at it…”)
>>>> 
>>>> 
>>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
>>>>> 
>>>>>> What would you suggest is to do ?
>>>>> 
>>>>> Did you try my previous suggestion already (i.e. using the
> "group-name"
>>>>> element in the "master" or "slave" element of "colocated")?
>>>>> 
>>>>> Aside from that, do you know why you were asked to add another broker?
>>>>> Depending on the reason it may not be a good solution.
>>>>> 
>>>>> 
>>>>> Justin
>>>>> 
>>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
>>>> wrote:
>>>>> 
>>>>>> Hi Justin
>>>>>> 
>>>>>> It is a good question I honestly don’t have the answer for. I
>> inherited
>>>>>> this configuration and was asked to add a third broker and to ensure
>>>> the co
>>>>>> located backups are being done in such a way that each broker points
>> on
>>>>>> another. Perhaps those who asked for it don’t fully understand
> Artemis
>>>> ! :)
>>>>>> 
>>>>>> That said, those co located backup on the existing setup with two
>>>> brokers
>>>>>> do work as we have been enabled to recover lost messages in the past.
>> So
>>>>>> even not optimal, technically it does work ?
>>>>>> 
>>>>>> I can only imagine that those who initially designed it about 5 years
>>>> ago
>>>>>> did not use a shared storage to avoid latency.
>>>>>> 
>>>>>> What would you suggest is to do ?
>>>>>> 
>>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
> wrote:
>>>>>> 
>>>>>> Your screenshot didn't come through the list.
>>>>>> 
>>>>>> In any case, I'm pretty confused at this point. You're clearly using
> a
>>>>>> colocated configuration that will request a backup from another
> broker
>>>> in
>>>>>> the cluster, but you say you're not running multiple brokers in the
>> same
>>>>>> JVM. If you aren't running multiple brokers in the same JVM then what
>>>> are
>>>>>> you using the colocated configuration for? The whole point of the
>>>> colocated
>>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
>> primary
>>>>>> broker and also a backup broker for another primary in the cluster).
>>>>>> 
>>>>>> 
>>>>>> Justin
>>>>>> 
>>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
>>>> wrote:
>>>>>> 
>>>>>>> I don’t believe we are.
>>>>>>> 
>>>>>>> So assume three Virtual Machines on Azure.
>>>>>>> 
>>>>>>> Each VM runs one Artemis broker
>>>>>>> 
>>>>>>> 
>>>>>>> All of their ha policy section on all three brokers look like that:
>>>>>>> 
>>>>>>> <ha-policy>
>>>>>>>   <replication>
>>>>>>>     <colocated>
>>>>>>>       <max-backups>1</max-backups>
>>>>>>>       <request-backup>true</request-backup>
>>>>>>> 
>>>>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>>       <excludes>
>>>>>>>         <connector-ref>my-connector</connector-ref>
>>>>>>>         <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>>       </excludes>
>>>>>>>       <master>
>>>>>>>         <check-for-live-server>true</check-for-live-server>
>>>>>>>       </master>
>>>>>>>       <slave>
>>>>>>>         <allow-failback>true</allow-failback>
>>>>>>>         <restart-backup>true</restart-backup>
>>>>>>>         <scale-down/>
>>>>>>>       </slave>
>>>>>>>     </colocated>
>>>>>>>   </replication>
>>>>>>> </ha-policy>
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> We are not running multiple brokers on the same JVM but a single
>>>> instance
>>>>>>> 
>>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>>> 
>>>>>>> Based on your previous message I was under the impression you were
>>>> using
>>>>>>> the "colocated" feature. *If* you're using this then you definitely
>> are
>>>>>>> running multiple brokers in the same JVM because that's precisely
>> what
>>>>>> that
>>>>>>> feature does. It runs a primary and a backup broker in the *same
>> JVM*.
>>>> If
>>>>>>> you aren't using a "colocated" configuration then I'm not sure what
>> the
>>>>>>> original question is about. Can you clarify?
>>>>>>> 
>>>>>>> 
>>>>>>> Justin
>>>>>>> 
>>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi Justin
>>>>>>> 
>>>>>>> Thank you for your input.
>>>>>>> 
>>>>>>> Sorry, should have been clearer on our setup - We are not running
>>>>>> multiple
>>>>>>> brokers on the same JVM but a single instance per VM, so each one
>> has a
>>>>>>> dedicated JVM and VM
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Roy
>>>>>>> 
>>>>>>> 
>>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
>> wrote:
>>>>>>> 
>>>>>>> I'm not entirely sure if the configuration you want is possible. You
>>>>>>> 
>>>>>>> might
>>>>>>> 
>>>>>>> try using the "group-name" element in the "master" or "slave"
> element
>>>> of
>>>>>>> "colocated." Only servers with the same group-name will pair
>> together.
>>>>>>> 
>>>>>>> Aside from that I would actually recommend against using colocated
>>>>>>> 
>>>>>>> brokers.
>>>>>>> 
>>>>>>> The original use-case for this functionality was very early cloud
>>>>>>> infrastructure where durable, attached storage was not readily
>>>> available.
>>>>>>> However, since then most (if not all) cloud environments support
>>>> durable
>>>>>>> storage separate from the broker so that if the broker goes down a
>> new,
>>>>>>> identical broker can be spun-up relatively quickly and attached to
>> the
>>>>>>> 
>>>>>>> same
>>>>>>> 
>>>>>>> storage. This provides functional high availability without the need
>>>> for
>>>>>>> any idle backups or replication of any kind which functionally
>>>> nullifies
>>>>>>> this feature.
>>>>>>> 
>>>>>>> Additionally, it turns out that (surprise!) configuring & running
>>>>>>> 
>>>>>>> multiple
>>>>>>> 
>>>>>>> brokers in the same JVM is difficult and error-prone not to mention
>> the
>>>>>>> complication of dynamically coordinating the acquisition of backups
>> in
>>>> a
>>>>>>> running cluster and protecting against split-brain.
>>>>>>> 
>>>>>>> 
>>>>>>> Justin
>>>>>>> 
>>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
>>>> wrote:
>>>>>>> 
>>>>>>> Hello everyone
>>>>>>> 
>>>>>>> We have a setup of three Artemis brokers (very old version don’t ask
>>>> :))
>>>>>>> 
>>>>>>> We would like to configure the co located backups such that the
>> backups
>>>>>>> are sent in this order:
>>>>>>> 
>>>>>>> Broker01 -> Broker02
>>>>>>> Broker02 -> Broker03
>>>>>>> Broker03 -> Broker01
>>>>>>> 
>>>>>>> 
>>>>>>> I was reading on co located backups here:
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>> 
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>>> 
>>>>>>> however not sure I fully understand how to configure the xml section
>> to
>>>>>>> achieve that.
>>>>>>> 
>>>>>>> Shall I add excludes in each broker, i.e.
>>>>>>> 
>>>>>>> <colocated>
>>>>>>>  <excludes>
>>>>>>>     <connector-ref>...</connector-ref>
>>>>>>>  </excludes>
>>>>>>> 
>>>>>>> Any help would be appreciated.
>>>>>>> 
>>>>>>> Many thanks in advance !
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 
>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
I wrote that on a completely different thread [1] related to MQTT retained
messages in a cluster. It is not related to this thread or your issue
generally.


Justin

[1] https://lists.apache.org/thread/oq41shfpv108m739km3rhs4tfj76c1zf

On Mon, Mar 27, 2023 at 1:28 PM Roy Cohen <ro...@hotmail.com> wrote:

> To quote:
>
> “This functionality isn't supported, and while it may be technically
> feasible to implement I'm not sure how much sense it makes overall.”
>
> On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:
>
> I'm not sure where I may have indicated that either one of those things
> isn't supported.
>
> In any case, you can do either.
>
>
> Justin
>
> On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com> wrote:
>
> > Just to be clear: When you say “isn’t supported” do you mean a third
> > broker or co located backups when running each broker on its own VM ?
> >
> >> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> >>
> >> Will do Justin and many thanks for all the additional details which I
> > will certainly bring forward internally, much appreciated
> >>
> >> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
> >>
> >> I recently added a new section to the clustering documentation
> regarding
> >> things to keep in mind regarding performance [1].
> >>
> >> Also, it's worth noting that often the bottleneck in messaging is not
> the
> >> broker itself but rather the consumer(s). It might be worth ensuring
> that
> >> the bottleneck really is the broker. As noted in the new documentation
> > [1],
> >> adding brokers to a cluster can actually *reduce* throughput in certain
> >> circumstances.
> >>
> >> Let me know if using group-name works for you.
> >>
> >>
> >> Justin
> >>
> >> [1]
> >>
> >
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> >>
> >> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> > wrote:
> >>
> >>> I haven’t tried he group-name yet.
> >>>
> >>> With regards to the third broker: The architects believe it’ll improve
> >>> performance given the amount of messages the brokers need to process
> (in
> >>> other words “throw more resources at it…”)
> >>>
> >>>
> >>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
> >>>>
> >>>>> What would you suggest is to do ?
> >>>>
> >>>> Did you try my previous suggestion already (i.e. using the
> "group-name"
> >>>> element in the "master" or "slave" element of "colocated")?
> >>>>
> >>>> Aside from that, do you know why you were asked to add another broker?
> >>>> Depending on the reason it may not be a good solution.
> >>>>
> >>>>
> >>>> Justin
> >>>>
> >>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
> >>> wrote:
> >>>>
> >>>>> Hi Justin
> >>>>>
> >>>>> It is a good question I honestly don’t have the answer for. I
> > inherited
> >>>>> this configuration and was asked to add a third broker and to ensure
> >>> the co
> >>>>> located backups are being done in such a way that each broker points
> > on
> >>>>> another. Perhaps those who asked for it don’t fully understand
> Artemis
> >>> ! :)
> >>>>>
> >>>>> That said, those co located backup on the existing setup with two
> >>> brokers
> >>>>> do work as we have been enabled to recover lost messages in the past.
> > So
> >>>>> even not optimal, technically it does work ?
> >>>>>
> >>>>> I can only imagine that those who initially designed it about 5 years
> >>> ago
> >>>>> did not use a shared storage to avoid latency.
> >>>>>
> >>>>> What would you suggest is to do ?
> >>>>>
> >>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>>
> >>>>> Your screenshot didn't come through the list.
> >>>>>
> >>>>> In any case, I'm pretty confused at this point. You're clearly using
> a
> >>>>> colocated configuration that will request a backup from another
> broker
> >>> in
> >>>>> the cluster, but you say you're not running multiple brokers in the
> > same
> >>>>> JVM. If you aren't running multiple brokers in the same JVM then what
> >>> are
> >>>>> you using the colocated configuration for? The whole point of the
> >>> colocated
> >>>>> configuration is to run multiple brokers in the same JVM (i.e. a
> > primary
> >>>>> broker and also a backup broker for another primary in the cluster).
> >>>>>
> >>>>>
> >>>>> Justin
> >>>>>
> >>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
> >>> wrote:
> >>>>>
> >>>>>> I don’t believe we are.
> >>>>>>
> >>>>>> So assume three Virtual Machines on Azure.
> >>>>>>
> >>>>>> Each VM runs one Artemis broker
> >>>>>>
> >>>>>>
> >>>>>> All of their ha policy section on all three brokers look like that:
> >>>>>>
> >>>>>> <ha-policy>
> >>>>>>    <replication>
> >>>>>>      <colocated>
> >>>>>>        <max-backups>1</max-backups>
> >>>>>>        <request-backup>true</request-backup>
> >>>>>>
> >>>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>>>>        <excludes>
> >>>>>>          <connector-ref>my-connector</connector-ref>
> >>>>>>          <connector-ref>thishostname.mydomain</connector-ref>
> >>>>>>        </excludes>
> >>>>>>        <master>
> >>>>>>          <check-for-live-server>true</check-for-live-server>
> >>>>>>        </master>
> >>>>>>        <slave>
> >>>>>>          <allow-failback>true</allow-failback>
> >>>>>>          <restart-backup>true</restart-backup>
> >>>>>>          <scale-down/>
> >>>>>>        </slave>
> >>>>>>      </colocated>
> >>>>>>    </replication>
> >>>>>>  </ha-policy>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> > wrote:
> >>>>>>
> >>>>>> We are not running multiple brokers on the same JVM but a single
> >>> instance
> >>>>>>
> >>>>>> per VM, so each one has a dedicated JVM and VM
> >>>>>>
> >>>>>> Based on your previous message I was under the impression you were
> >>> using
> >>>>>> the "colocated" feature. *If* you're using this then you definitely
> > are
> >>>>>> running multiple brokers in the same JVM because that's precisely
> > what
> >>>>> that
> >>>>>> feature does. It runs a primary and a backup broker in the *same
> > JVM*.
> >>> If
> >>>>>> you aren't using a "colocated" configuration then I'm not sure what
> > the
> >>>>>> original question is about. Can you clarify?
> >>>>>>
> >>>>>>
> >>>>>> Justin
> >>>>>>
> >>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hi Justin
> >>>>>>
> >>>>>> Thank you for your input.
> >>>>>>
> >>>>>> Sorry, should have been clearer on our setup - We are not running
> >>>>> multiple
> >>>>>> brokers on the same JVM but a single instance per VM, so each one
> > has a
> >>>>>> dedicated JVM and VM
> >>>>>>
> >>>>>> Thanks
> >>>>>> Roy
> >>>>>>
> >>>>>>
> >>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> > wrote:
> >>>>>>
> >>>>>> I'm not entirely sure if the configuration you want is possible. You
> >>>>>>
> >>>>>> might
> >>>>>>
> >>>>>> try using the "group-name" element in the "master" or "slave"
> element
> >>> of
> >>>>>> "colocated." Only servers with the same group-name will pair
> > together.
> >>>>>>
> >>>>>> Aside from that I would actually recommend against using colocated
> >>>>>>
> >>>>>> brokers.
> >>>>>>
> >>>>>> The original use-case for this functionality was very early cloud
> >>>>>> infrastructure where durable, attached storage was not readily
> >>> available.
> >>>>>> However, since then most (if not all) cloud environments support
> >>> durable
> >>>>>> storage separate from the broker so that if the broker goes down a
> > new,
> >>>>>> identical broker can be spun-up relatively quickly and attached to
> > the
> >>>>>>
> >>>>>> same
> >>>>>>
> >>>>>> storage. This provides functional high availability without the need
> >>> for
> >>>>>> any idle backups or replication of any kind which functionally
> >>> nullifies
> >>>>>> this feature.
> >>>>>>
> >>>>>> Additionally, it turns out that (surprise!) configuring & running
> >>>>>>
> >>>>>> multiple
> >>>>>>
> >>>>>> brokers in the same JVM is difficult and error-prone not to mention
> > the
> >>>>>> complication of dynamically coordinating the acquisition of backups
> > in
> >>> a
> >>>>>> running cluster and protecting against split-brain.
> >>>>>>
> >>>>>>
> >>>>>> Justin
> >>>>>>
> >>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
> >>> wrote:
> >>>>>>
> >>>>>> Hello everyone
> >>>>>>
> >>>>>> We have a setup of three Artemis brokers (very old version don’t ask
> >>> :))
> >>>>>>
> >>>>>> We would like to configure the co located backups such that the
> > backups
> >>>>>> are sent in this order:
> >>>>>>
> >>>>>> Broker01 -> Broker02
> >>>>>> Broker02 -> Broker03
> >>>>>> Broker03 -> Broker01
> >>>>>>
> >>>>>>
> >>>>>> I was reading on co located backups here:
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>>>>
> >>>>>> however not sure I fully understand how to configure the xml section
> > to
> >>>>>> achieve that.
> >>>>>>
> >>>>>> Shall I add excludes in each broker, i.e.
> >>>>>>
> >>>>>> <colocated>
> >>>>>>   <excludes>
> >>>>>>      <connector-ref>...</connector-ref>
> >>>>>>   </excludes>
> >>>>>>
> >>>>>> Any help would be appreciated.
> >>>>>>
> >>>>>> Many thanks in advance !
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >
> >
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
To quote:

“This functionality isn't supported, and while it may be technically feasible to implement I'm not sure how much sense it makes overall.”

On 27 Mar 2023, at 19:16, Justin Bertram <jb...@apache.org> wrote:

I'm not sure where I may have indicated that either one of those things
isn't supported.

In any case, you can do either.


Justin

On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com> wrote:

> Just to be clear: When you say “isn’t supported” do you mean a third
> broker or co located backups when running each broker on its own VM ?
> 
>> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
>> 
>> Will do Justin and many thanks for all the additional details which I
> will certainly bring forward internally, much appreciated
>> 
>> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
>> 
>> I recently added a new section to the clustering documentation regarding
>> things to keep in mind regarding performance [1].
>> 
>> Also, it's worth noting that often the bottleneck in messaging is not the
>> broker itself but rather the consumer(s). It might be worth ensuring that
>> the bottleneck really is the broker. As noted in the new documentation
> [1],
>> adding brokers to a cluster can actually *reduce* throughput in certain
>> circumstances.
>> 
>> Let me know if using group-name works for you.
>> 
>> 
>> Justin
>> 
>> [1]
>> 
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
>> 
>> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> wrote:
>> 
>>> I haven’t tried he group-name yet.
>>> 
>>> With regards to the third broker: The architects believe it’ll improve
>>> performance given the amount of messages the brokers need to process (in
>>> other words “throw more resources at it…”)
>>> 
>>> 
>>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>>> What would you suggest is to do ?
>>>> 
>>>> Did you try my previous suggestion already (i.e. using the "group-name"
>>>> element in the "master" or "slave" element of "colocated")?
>>>> 
>>>> Aside from that, do you know why you were asked to add another broker?
>>>> Depending on the reason it may not be a good solution.
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>> 
>>>>> Hi Justin
>>>>> 
>>>>> It is a good question I honestly don’t have the answer for. I
> inherited
>>>>> this configuration and was asked to add a third broker and to ensure
>>> the co
>>>>> located backups are being done in such a way that each broker points
> on
>>>>> another. Perhaps those who asked for it don’t fully understand Artemis
>>> ! :)
>>>>> 
>>>>> That said, those co located backup on the existing setup with two
>>> brokers
>>>>> do work as we have been enabled to recover lost messages in the past.
> So
>>>>> even not optimal, technically it does work ?
>>>>> 
>>>>> I can only imagine that those who initially designed it about 5 years
>>> ago
>>>>> did not use a shared storage to avoid latency.
>>>>> 
>>>>> What would you suggest is to do ?
>>>>> 
>>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
>>>>> 
>>>>> Your screenshot didn't come through the list.
>>>>> 
>>>>> In any case, I'm pretty confused at this point. You're clearly using a
>>>>> colocated configuration that will request a backup from another broker
>>> in
>>>>> the cluster, but you say you're not running multiple brokers in the
> same
>>>>> JVM. If you aren't running multiple brokers in the same JVM then what
>>> are
>>>>> you using the colocated configuration for? The whole point of the
>>> colocated
>>>>> configuration is to run multiple brokers in the same JVM (i.e. a
> primary
>>>>> broker and also a backup broker for another primary in the cluster).
>>>>> 
>>>>> 
>>>>> Justin
>>>>> 
>>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>>> 
>>>>>> I don’t believe we are.
>>>>>> 
>>>>>> So assume three Virtual Machines on Azure.
>>>>>> 
>>>>>> Each VM runs one Artemis broker
>>>>>> 
>>>>>> 
>>>>>> All of their ha policy section on all three brokers look like that:
>>>>>> 
>>>>>> <ha-policy>
>>>>>>    <replication>
>>>>>>      <colocated>
>>>>>>        <max-backups>1</max-backups>
>>>>>>        <request-backup>true</request-backup>
>>>>>> 
>>>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>>        <excludes>
>>>>>>          <connector-ref>my-connector</connector-ref>
>>>>>>          <connector-ref>thishostname.mydomain</connector-ref>
>>>>>>        </excludes>
>>>>>>        <master>
>>>>>>          <check-for-live-server>true</check-for-live-server>
>>>>>>        </master>
>>>>>>        <slave>
>>>>>>          <allow-failback>true</allow-failback>
>>>>>>          <restart-backup>true</restart-backup>
>>>>>>          <scale-down/>
>>>>>>        </slave>
>>>>>>      </colocated>
>>>>>>    </replication>
>>>>>>  </ha-policy>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> wrote:
>>>>>> 
>>>>>> We are not running multiple brokers on the same JVM but a single
>>> instance
>>>>>> 
>>>>>> per VM, so each one has a dedicated JVM and VM
>>>>>> 
>>>>>> Based on your previous message I was under the impression you were
>>> using
>>>>>> the "colocated" feature. *If* you're using this then you definitely
> are
>>>>>> running multiple brokers in the same JVM because that's precisely
> what
>>>>> that
>>>>>> feature does. It runs a primary and a backup broker in the *same
> JVM*.
>>> If
>>>>>> you aren't using a "colocated" configuration then I'm not sure what
> the
>>>>>> original question is about. Can you clarify?
>>>>>> 
>>>>>> 
>>>>>> Justin
>>>>>> 
>>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> Hi Justin
>>>>>> 
>>>>>> Thank you for your input.
>>>>>> 
>>>>>> Sorry, should have been clearer on our setup - We are not running
>>>>> multiple
>>>>>> brokers on the same JVM but a single instance per VM, so each one
> has a
>>>>>> dedicated JVM and VM
>>>>>> 
>>>>>> Thanks
>>>>>> Roy
>>>>>> 
>>>>>> 
>>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> wrote:
>>>>>> 
>>>>>> I'm not entirely sure if the configuration you want is possible. You
>>>>>> 
>>>>>> might
>>>>>> 
>>>>>> try using the "group-name" element in the "master" or "slave" element
>>> of
>>>>>> "colocated." Only servers with the same group-name will pair
> together.
>>>>>> 
>>>>>> Aside from that I would actually recommend against using colocated
>>>>>> 
>>>>>> brokers.
>>>>>> 
>>>>>> The original use-case for this functionality was very early cloud
>>>>>> infrastructure where durable, attached storage was not readily
>>> available.
>>>>>> However, since then most (if not all) cloud environments support
>>> durable
>>>>>> storage separate from the broker so that if the broker goes down a
> new,
>>>>>> identical broker can be spun-up relatively quickly and attached to
> the
>>>>>> 
>>>>>> same
>>>>>> 
>>>>>> storage. This provides functional high availability without the need
>>> for
>>>>>> any idle backups or replication of any kind which functionally
>>> nullifies
>>>>>> this feature.
>>>>>> 
>>>>>> Additionally, it turns out that (surprise!) configuring & running
>>>>>> 
>>>>>> multiple
>>>>>> 
>>>>>> brokers in the same JVM is difficult and error-prone not to mention
> the
>>>>>> complication of dynamically coordinating the acquisition of backups
> in
>>> a
>>>>>> running cluster and protecting against split-brain.
>>>>>> 
>>>>>> 
>>>>>> Justin
>>>>>> 
>>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>>>> 
>>>>>> Hello everyone
>>>>>> 
>>>>>> We have a setup of three Artemis brokers (very old version don’t ask
>>> :))
>>>>>> 
>>>>>> We would like to configure the co located backups such that the
> backups
>>>>>> are sent in this order:
>>>>>> 
>>>>>> Broker01 -> Broker02
>>>>>> Broker02 -> Broker03
>>>>>> Broker03 -> Broker01
>>>>>> 
>>>>>> 
>>>>>> I was reading on co located backups here:
>>>>>> 
>>>>>> 
>>>>> 
>>> 
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>>> 
>>>>>> however not sure I fully understand how to configure the xml section
> to
>>>>>> achieve that.
>>>>>> 
>>>>>> Shall I add excludes in each broker, i.e.
>>>>>> 
>>>>>> <colocated>
>>>>>>   <excludes>
>>>>>>      <connector-ref>...</connector-ref>
>>>>>>   </excludes>
>>>>>> 
>>>>>> Any help would be appreciated.
>>>>>> 
>>>>>> Many thanks in advance !
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
> 
>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
I'm not sure where I may have indicated that either one of those things
isn't supported.

In any case, you can do either.


Justin

On Mon, Mar 27, 2023 at 1:07 PM Roy Cohen <ro...@hotmail.com> wrote:

> Just to be clear: When you say “isn’t supported” do you mean a third
> broker or co located backups when running each broker on its own VM ?
>
> > On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> >
> > Will do Justin and many thanks for all the additional details which I
> will certainly bring forward internally, much appreciated
> >
> > On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
> >
> > I recently added a new section to the clustering documentation regarding
> > things to keep in mind regarding performance [1].
> >
> > Also, it's worth noting that often the bottleneck in messaging is not the
> > broker itself but rather the consumer(s). It might be worth ensuring that
> > the bottleneck really is the broker. As noted in the new documentation
> [1],
> > adding brokers to a cluster can actually *reduce* throughput in certain
> > circumstances.
> >
> > Let me know if using group-name works for you.
> >
> >
> > Justin
> >
> > [1]
> >
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> >
> > On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com>
> wrote:
> >
> >> I haven’t tried he group-name yet.
> >>
> >> With regards to the third broker: The architects believe it’ll improve
> >> performance given the amount of messages the brokers need to process (in
> >> other words “throw more resources at it…”)
> >>
> >>
> >>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
> >>>
> >>>> What would you suggest is to do ?
> >>>
> >>> Did you try my previous suggestion already (i.e. using the "group-name"
> >>> element in the "master" or "slave" element of "colocated")?
> >>>
> >>> Aside from that, do you know why you were asked to add another broker?
> >>> Depending on the reason it may not be a good solution.
> >>>
> >>>
> >>> Justin
> >>>
> >>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
> >> wrote:
> >>>
> >>>> Hi Justin
> >>>>
> >>>> It is a good question I honestly don’t have the answer for. I
> inherited
> >>>> this configuration and was asked to add a third broker and to ensure
> >> the co
> >>>> located backups are being done in such a way that each broker points
> on
> >>>> another. Perhaps those who asked for it don’t fully understand Artemis
> >> ! :)
> >>>>
> >>>> That said, those co located backup on the existing setup with two
> >> brokers
> >>>> do work as we have been enabled to recover lost messages in the past.
> So
> >>>> even not optimal, technically it does work ?
> >>>>
> >>>> I can only imagine that those who initially designed it about 5 years
> >> ago
> >>>> did not use a shared storage to avoid latency.
> >>>>
> >>>> What would you suggest is to do ?
> >>>>
> >>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
> >>>>
> >>>> Your screenshot didn't come through the list.
> >>>>
> >>>> In any case, I'm pretty confused at this point. You're clearly using a
> >>>> colocated configuration that will request a backup from another broker
> >> in
> >>>> the cluster, but you say you're not running multiple brokers in the
> same
> >>>> JVM. If you aren't running multiple brokers in the same JVM then what
> >> are
> >>>> you using the colocated configuration for? The whole point of the
> >> colocated
> >>>> configuration is to run multiple brokers in the same JVM (i.e. a
> primary
> >>>> broker and also a backup broker for another primary in the cluster).
> >>>>
> >>>>
> >>>> Justin
> >>>>
> >>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
> >> wrote:
> >>>>
> >>>>> I don’t believe we are.
> >>>>>
> >>>>> So assume three Virtual Machines on Azure.
> >>>>>
> >>>>> Each VM runs one Artemis broker
> >>>>>
> >>>>>
> >>>>> All of their ha policy section on all three brokers look like that:
> >>>>>
> >>>>> <ha-policy>
> >>>>>     <replication>
> >>>>>       <colocated>
> >>>>>         <max-backups>1</max-backups>
> >>>>>         <request-backup>true</request-backup>
> >>>>>
> >>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>>>         <excludes>
> >>>>>           <connector-ref>my-connector</connector-ref>
> >>>>>           <connector-ref>thishostname.mydomain</connector-ref>
> >>>>>         </excludes>
> >>>>>         <master>
> >>>>>           <check-for-live-server>true</check-for-live-server>
> >>>>>         </master>
> >>>>>         <slave>
> >>>>>           <allow-failback>true</allow-failback>
> >>>>>           <restart-backup>true</restart-backup>
> >>>>>           <scale-down/>
> >>>>>         </slave>
> >>>>>       </colocated>
> >>>>>     </replication>
> >>>>>   </ha-policy>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>>
> >>>>> We are not running multiple brokers on the same JVM but a single
> >> instance
> >>>>>
> >>>>> per VM, so each one has a dedicated JVM and VM
> >>>>>
> >>>>> Based on your previous message I was under the impression you were
> >> using
> >>>>> the "colocated" feature. *If* you're using this then you definitely
> are
> >>>>> running multiple brokers in the same JVM because that's precisely
> what
> >>>> that
> >>>>> feature does. It runs a primary and a backup broker in the *same
> JVM*.
> >> If
> >>>>> you aren't using a "colocated" configuration then I'm not sure what
> the
> >>>>> original question is about. Can you clarify?
> >>>>>
> >>>>>
> >>>>> Justin
> >>>>>
> >>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi Justin
> >>>>>
> >>>>> Thank you for your input.
> >>>>>
> >>>>> Sorry, should have been clearer on our setup - We are not running
> >>>> multiple
> >>>>> brokers on the same JVM but a single instance per VM, so each one
> has a
> >>>>> dedicated JVM and VM
> >>>>>
> >>>>> Thanks
> >>>>> Roy
> >>>>>
> >>>>>
> >>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>
> wrote:
> >>>>>
> >>>>> I'm not entirely sure if the configuration you want is possible. You
> >>>>>
> >>>>> might
> >>>>>
> >>>>> try using the "group-name" element in the "master" or "slave" element
> >> of
> >>>>> "colocated." Only servers with the same group-name will pair
> together.
> >>>>>
> >>>>> Aside from that I would actually recommend against using colocated
> >>>>>
> >>>>> brokers.
> >>>>>
> >>>>> The original use-case for this functionality was very early cloud
> >>>>> infrastructure where durable, attached storage was not readily
> >> available.
> >>>>> However, since then most (if not all) cloud environments support
> >> durable
> >>>>> storage separate from the broker so that if the broker goes down a
> new,
> >>>>> identical broker can be spun-up relatively quickly and attached to
> the
> >>>>>
> >>>>> same
> >>>>>
> >>>>> storage. This provides functional high availability without the need
> >> for
> >>>>> any idle backups or replication of any kind which functionally
> >> nullifies
> >>>>> this feature.
> >>>>>
> >>>>> Additionally, it turns out that (surprise!) configuring & running
> >>>>>
> >>>>> multiple
> >>>>>
> >>>>> brokers in the same JVM is difficult and error-prone not to mention
> the
> >>>>> complication of dynamically coordinating the acquisition of backups
> in
> >> a
> >>>>> running cluster and protecting against split-brain.
> >>>>>
> >>>>>
> >>>>> Justin
> >>>>>
> >>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
> >> wrote:
> >>>>>
> >>>>> Hello everyone
> >>>>>
> >>>>> We have a setup of three Artemis brokers (very old version don’t ask
> >> :))
> >>>>>
> >>>>> We would like to configure the co located backups such that the
> backups
> >>>>> are sent in this order:
> >>>>>
> >>>>> Broker01 -> Broker02
> >>>>> Broker02 -> Broker03
> >>>>> Broker03 -> Broker01
> >>>>>
> >>>>>
> >>>>> I was reading on co located backups here:
> >>>>>
> >>>>>
> >>>>
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>>>
> >>>>> however not sure I fully understand how to configure the xml section
> to
> >>>>> achieve that.
> >>>>>
> >>>>> Shall I add excludes in each broker, i.e.
> >>>>>
> >>>>> <colocated>
> >>>>>    <excludes>
> >>>>>       <connector-ref>...</connector-ref>
> >>>>>    </excludes>
> >>>>>
> >>>>> Any help would be appreciated.
> >>>>>
> >>>>> Many thanks in advance !
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Just to be clear: When you say “isn’t supported” do you mean a third broker or co located backups when running each broker on its own VM ?

> On 27 Mar 2023, at 19:04, Roy Cohen <ro...@hotmail.com> wrote:
> 
> Will do Justin and many thanks for all the additional details which I will certainly bring forward internally, much appreciated 
> 
> On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:
> 
> I recently added a new section to the clustering documentation regarding
> things to keep in mind regarding performance [1].
> 
> Also, it's worth noting that often the bottleneck in messaging is not the
> broker itself but rather the consumer(s). It might be worth ensuring that
> the bottleneck really is the broker. As noted in the new documentation [1],
> adding brokers to a cluster can actually *reduce* throughput in certain
> circumstances.
> 
> Let me know if using group-name works for you.
> 
> 
> Justin
> 
> [1]
> https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations
> 
> On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> I haven’t tried he group-name yet.
>> 
>> With regards to the third broker: The architects believe it’ll improve
>> performance given the amount of messages the brokers need to process (in
>> other words “throw more resources at it…”)
>> 
>> 
>>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>>> What would you suggest is to do ?
>>> 
>>> Did you try my previous suggestion already (i.e. using the "group-name"
>>> element in the "master" or "slave" element of "colocated")?
>>> 
>>> Aside from that, do you know why you were asked to add another broker?
>>> Depending on the reason it may not be a good solution.
>>> 
>>> 
>>> Justin
>>> 
>>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>> 
>>>> Hi Justin
>>>> 
>>>> It is a good question I honestly don’t have the answer for. I inherited
>>>> this configuration and was asked to add a third broker and to ensure
>> the co
>>>> located backups are being done in such a way that each broker points on
>>>> another. Perhaps those who asked for it don’t fully understand Artemis
>> ! :)
>>>> 
>>>> That said, those co located backup on the existing setup with two
>> brokers
>>>> do work as we have been enabled to recover lost messages in the past. So
>>>> even not optimal, technically it does work ?
>>>> 
>>>> I can only imagine that those who initially designed it about 5 years
>> ago
>>>> did not use a shared storage to avoid latency.
>>>> 
>>>> What would you suggest is to do ?
>>>> 
>>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>> Your screenshot didn't come through the list.
>>>> 
>>>> In any case, I'm pretty confused at this point. You're clearly using a
>>>> colocated configuration that will request a backup from another broker
>> in
>>>> the cluster, but you say you're not running multiple brokers in the same
>>>> JVM. If you aren't running multiple brokers in the same JVM then what
>> are
>>>> you using the colocated configuration for? The whole point of the
>> colocated
>>>> configuration is to run multiple brokers in the same JVM (i.e. a primary
>>>> broker and also a backup broker for another primary in the cluster).
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>>> 
>>>>> I don’t believe we are.
>>>>> 
>>>>> So assume three Virtual Machines on Azure.
>>>>> 
>>>>> Each VM runs one Artemis broker
>>>>> 
>>>>> 
>>>>> All of their ha policy section on all three brokers look like that:
>>>>> 
>>>>> <ha-policy>
>>>>>     <replication>
>>>>>       <colocated>
>>>>>         <max-backups>1</max-backups>
>>>>>         <request-backup>true</request-backup>
>>>>> 
>>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>>         <excludes>
>>>>>           <connector-ref>my-connector</connector-ref>
>>>>>           <connector-ref>thishostname.mydomain</connector-ref>
>>>>>         </excludes>
>>>>>         <master>
>>>>>           <check-for-live-server>true</check-for-live-server>
>>>>>         </master>
>>>>>         <slave>
>>>>>           <allow-failback>true</allow-failback>
>>>>>           <restart-backup>true</restart-backup>
>>>>>           <scale-down/>
>>>>>         </slave>
>>>>>       </colocated>
>>>>>     </replication>
>>>>>   </ha-policy>
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
>>>>> 
>>>>> We are not running multiple brokers on the same JVM but a single
>> instance
>>>>> 
>>>>> per VM, so each one has a dedicated JVM and VM
>>>>> 
>>>>> Based on your previous message I was under the impression you were
>> using
>>>>> the "colocated" feature. *If* you're using this then you definitely are
>>>>> running multiple brokers in the same JVM because that's precisely what
>>>> that
>>>>> feature does. It runs a primary and a backup broker in the *same JVM*.
>> If
>>>>> you aren't using a "colocated" configuration then I'm not sure what the
>>>>> original question is about. Can you clarify?
>>>>> 
>>>>> 
>>>>> Justin
>>>>> 
>>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>>>> wrote:
>>>>> 
>>>>> Hi Justin
>>>>> 
>>>>> Thank you for your input.
>>>>> 
>>>>> Sorry, should have been clearer on our setup - We are not running
>>>> multiple
>>>>> brokers on the same JVM but a single instance per VM, so each one has a
>>>>> dedicated JVM and VM
>>>>> 
>>>>> Thanks
>>>>> Roy
>>>>> 
>>>>> 
>>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
>>>>> 
>>>>> I'm not entirely sure if the configuration you want is possible. You
>>>>> 
>>>>> might
>>>>> 
>>>>> try using the "group-name" element in the "master" or "slave" element
>> of
>>>>> "colocated." Only servers with the same group-name will pair together.
>>>>> 
>>>>> Aside from that I would actually recommend against using colocated
>>>>> 
>>>>> brokers.
>>>>> 
>>>>> The original use-case for this functionality was very early cloud
>>>>> infrastructure where durable, attached storage was not readily
>> available.
>>>>> However, since then most (if not all) cloud environments support
>> durable
>>>>> storage separate from the broker so that if the broker goes down a new,
>>>>> identical broker can be spun-up relatively quickly and attached to the
>>>>> 
>>>>> same
>>>>> 
>>>>> storage. This provides functional high availability without the need
>> for
>>>>> any idle backups or replication of any kind which functionally
>> nullifies
>>>>> this feature.
>>>>> 
>>>>> Additionally, it turns out that (surprise!) configuring & running
>>>>> 
>>>>> multiple
>>>>> 
>>>>> brokers in the same JVM is difficult and error-prone not to mention the
>>>>> complication of dynamically coordinating the acquisition of backups in
>> a
>>>>> running cluster and protecting against split-brain.
>>>>> 
>>>>> 
>>>>> Justin
>>>>> 
>>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>>>> 
>>>>> Hello everyone
>>>>> 
>>>>> We have a setup of three Artemis brokers (very old version don’t ask
>> :))
>>>>> 
>>>>> We would like to configure the co located backups such that the backups
>>>>> are sent in this order:
>>>>> 
>>>>> Broker01 -> Broker02
>>>>> Broker02 -> Broker03
>>>>> Broker03 -> Broker01
>>>>> 
>>>>> 
>>>>> I was reading on co located backups here:
>>>>> 
>>>>> 
>>>> 
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>>> 
>>>>> however not sure I fully understand how to configure the xml section to
>>>>> achieve that.
>>>>> 
>>>>> Shall I add excludes in each broker, i.e.
>>>>> 
>>>>> <colocated>
>>>>>    <excludes>
>>>>>       <connector-ref>...</connector-ref>
>>>>>    </excludes>
>>>>> 
>>>>> Any help would be appreciated.
>>>>> 
>>>>> Many thanks in advance !
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>> 
>>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Will do Justin and many thanks for all the additional details which I will certainly bring forward internally, much appreciated 

On 27 Mar 2023, at 18:58, Justin Bertram <jb...@apache.org> wrote:

I recently added a new section to the clustering documentation regarding
things to keep in mind regarding performance [1].

Also, it's worth noting that often the bottleneck in messaging is not the
broker itself but rather the consumer(s). It might be worth ensuring that
the bottleneck really is the broker. As noted in the new documentation [1],
adding brokers to a cluster can actually *reduce* throughput in certain
circumstances.

Let me know if using group-name works for you.


Justin

[1]
https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations

On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com> wrote:

> I haven’t tried he group-name yet.
> 
> With regards to the third broker: The architects believe it’ll improve
> performance given the amount of messages the brokers need to process (in
> other words “throw more resources at it…”)
> 
> 
>> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
>> 
>>> What would you suggest is to do ?
>> 
>> Did you try my previous suggestion already (i.e. using the "group-name"
>> element in the "master" or "slave" element of "colocated")?
>> 
>> Aside from that, do you know why you were asked to add another broker?
>> Depending on the reason it may not be a good solution.
>> 
>> 
>> Justin
>> 
>> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
> wrote:
>> 
>>> Hi Justin
>>> 
>>> It is a good question I honestly don’t have the answer for. I inherited
>>> this configuration and was asked to add a third broker and to ensure
> the co
>>> located backups are being done in such a way that each broker points on
>>> another. Perhaps those who asked for it don’t fully understand Artemis
> ! :)
>>> 
>>> That said, those co located backup on the existing setup with two
> brokers
>>> do work as we have been enabled to recover lost messages in the past. So
>>> even not optimal, technically it does work ?
>>> 
>>> I can only imagine that those who initially designed it about 5 years
> ago
>>> did not use a shared storage to avoid latency.
>>> 
>>> What would you suggest is to do ?
>>> 
>>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>> Your screenshot didn't come through the list.
>>> 
>>> In any case, I'm pretty confused at this point. You're clearly using a
>>> colocated configuration that will request a backup from another broker
> in
>>> the cluster, but you say you're not running multiple brokers in the same
>>> JVM. If you aren't running multiple brokers in the same JVM then what
> are
>>> you using the colocated configuration for? The whole point of the
> colocated
>>> configuration is to run multiple brokers in the same JVM (i.e. a primary
>>> broker and also a backup broker for another primary in the cluster).
>>> 
>>> 
>>> Justin
>>> 
>>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
> wrote:
>>> 
>>>> I don’t believe we are.
>>>> 
>>>> So assume three Virtual Machines on Azure.
>>>> 
>>>> Each VM runs one Artemis broker
>>>> 
>>>> 
>>>> All of their ha policy section on all three brokers look like that:
>>>> 
>>>> <ha-policy>
>>>>      <replication>
>>>>        <colocated>
>>>>          <max-backups>1</max-backups>
>>>>          <request-backup>true</request-backup>
>>>> 
>>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>>          <excludes>
>>>>            <connector-ref>my-connector</connector-ref>
>>>>            <connector-ref>thishostname.mydomain</connector-ref>
>>>>          </excludes>
>>>>          <master>
>>>>            <check-for-live-server>true</check-for-live-server>
>>>>          </master>
>>>>          <slave>
>>>>            <allow-failback>true</allow-failback>
>>>>            <restart-backup>true</restart-backup>
>>>>            <scale-down/>
>>>>          </slave>
>>>>        </colocated>
>>>>      </replication>
>>>>    </ha-policy>
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>> We are not running multiple brokers on the same JVM but a single
> instance
>>>> 
>>>> per VM, so each one has a dedicated JVM and VM
>>>> 
>>>> Based on your previous message I was under the impression you were
> using
>>>> the "colocated" feature. *If* you're using this then you definitely are
>>>> running multiple brokers in the same JVM because that's precisely what
>>> that
>>>> feature does. It runs a primary and a backup broker in the *same JVM*.
> If
>>>> you aren't using a "colocated" configuration then I'm not sure what the
>>>> original question is about. Can you clarify?
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>>> wrote:
>>>> 
>>>> Hi Justin
>>>> 
>>>> Thank you for your input.
>>>> 
>>>> Sorry, should have been clearer on our setup - We are not running
>>> multiple
>>>> brokers on the same JVM but a single instance per VM, so each one has a
>>>> dedicated JVM and VM
>>>> 
>>>> Thanks
>>>> Roy
>>>> 
>>>> 
>>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
>>>> 
>>>> I'm not entirely sure if the configuration you want is possible. You
>>>> 
>>>> might
>>>> 
>>>> try using the "group-name" element in the "master" or "slave" element
> of
>>>> "colocated." Only servers with the same group-name will pair together.
>>>> 
>>>> Aside from that I would actually recommend against using colocated
>>>> 
>>>> brokers.
>>>> 
>>>> The original use-case for this functionality was very early cloud
>>>> infrastructure where durable, attached storage was not readily
> available.
>>>> However, since then most (if not all) cloud environments support
> durable
>>>> storage separate from the broker so that if the broker goes down a new,
>>>> identical broker can be spun-up relatively quickly and attached to the
>>>> 
>>>> same
>>>> 
>>>> storage. This provides functional high availability without the need
> for
>>>> any idle backups or replication of any kind which functionally
> nullifies
>>>> this feature.
>>>> 
>>>> Additionally, it turns out that (surprise!) configuring & running
>>>> 
>>>> multiple
>>>> 
>>>> brokers in the same JVM is difficult and error-prone not to mention the
>>>> complication of dynamically coordinating the acquisition of backups in
> a
>>>> running cluster and protecting against split-brain.
>>>> 
>>>> 
>>>> Justin
>>>> 
>>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
> wrote:
>>>> 
>>>> Hello everyone
>>>> 
>>>> We have a setup of three Artemis brokers (very old version don’t ask
> :))
>>>> 
>>>> We would like to configure the co located backups such that the backups
>>>> are sent in this order:
>>>> 
>>>> Broker01 -> Broker02
>>>> Broker02 -> Broker03
>>>> Broker03 -> Broker01
>>>> 
>>>> 
>>>> I was reading on co located backups here:
>>>> 
>>>> 
>>> 
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>>> 
>>>> however not sure I fully understand how to configure the xml section to
>>>> achieve that.
>>>> 
>>>> Shall I add excludes in each broker, i.e.
>>>> 
>>>>  <colocated>
>>>>     <excludes>
>>>>        <connector-ref>...</connector-ref>
>>>>     </excludes>
>>>> 
>>>> Any help would be appreciated.
>>>> 
>>>> Many thanks in advance !
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
> 
>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
I recently added a new section to the clustering documentation regarding
things to keep in mind regarding performance [1].

Also, it's worth noting that often the bottleneck in messaging is not the
broker itself but rather the consumer(s). It might be worth ensuring that
the bottleneck really is the broker. As noted in the new documentation [1],
adding brokers to a cluster can actually *reduce* throughput in certain
circumstances.

Let me know if using group-name works for you.


Justin

[1]
https://github.com/apache/activemq-artemis/blob/main/docs/user-manual/en/clusters.md#performance-considerations

On Mon, Mar 27, 2023 at 12:44 PM Roy Cohen <ro...@hotmail.com> wrote:

> I haven’t tried he group-name yet.
>
> With regards to the third broker: The architects believe it’ll improve
> performance given the amount of messages the brokers need to process (in
> other words “throw more resources at it…”)
>
>
> > On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
> >
> >> What would you suggest is to do ?
> >
> > Did you try my previous suggestion already (i.e. using the "group-name"
> > element in the "master" or "slave" element of "colocated")?
> >
> > Aside from that, do you know why you were asked to add another broker?
> > Depending on the reason it may not be a good solution.
> >
> >
> > Justin
> >
> > On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com>
> wrote:
> >
> >> Hi Justin
> >>
> >> It is a good question I honestly don’t have the answer for. I inherited
> >> this configuration and was asked to add a third broker and to ensure
> the co
> >> located backups are being done in such a way that each broker points on
> >> another. Perhaps those who asked for it don’t fully understand Artemis
> ! :)
> >>
> >> That said, those co located backup on the existing setup with two
> brokers
> >> do work as we have been enabled to recover lost messages in the past. So
> >> even not optimal, technically it does work ?
> >>
> >> I can only imagine that those who initially designed it about 5 years
> ago
> >> did not use a shared storage to avoid latency.
> >>
> >> What would you suggest is to do ?
> >>
> >> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
> >>
> >> Your screenshot didn't come through the list.
> >>
> >> In any case, I'm pretty confused at this point. You're clearly using a
> >> colocated configuration that will request a backup from another broker
> in
> >> the cluster, but you say you're not running multiple brokers in the same
> >> JVM. If you aren't running multiple brokers in the same JVM then what
> are
> >> you using the colocated configuration for? The whole point of the
> colocated
> >> configuration is to run multiple brokers in the same JVM (i.e. a primary
> >> broker and also a backup broker for another primary in the cluster).
> >>
> >>
> >> Justin
> >>
> >> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>
> >>> I don’t believe we are.
> >>>
> >>> So assume three Virtual Machines on Azure.
> >>>
> >>> Each VM runs one Artemis broker
> >>>
> >>>
> >>> All of their ha policy section on all three brokers look like that:
> >>>
> >>> <ha-policy>
> >>>       <replication>
> >>>         <colocated>
> >>>           <max-backups>1</max-backups>
> >>>           <request-backup>true</request-backup>
> >>>
> >>> <backup-request-retry-interval>1000</backup-request-retry-interval>
> >>>           <excludes>
> >>>             <connector-ref>my-connector</connector-ref>
> >>>             <connector-ref>thishostname.mydomain</connector-ref>
> >>>           </excludes>
> >>>           <master>
> >>>             <check-for-live-server>true</check-for-live-server>
> >>>           </master>
> >>>           <slave>
> >>>             <allow-failback>true</allow-failback>
> >>>             <restart-backup>true</restart-backup>
> >>>             <scale-down/>
> >>>           </slave>
> >>>         </colocated>
> >>>       </replication>
> >>>     </ha-policy>
> >>>
> >>>
> >>>
> >>>
> >>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
> >>>
> >>> We are not running multiple brokers on the same JVM but a single
> instance
> >>>
> >>> per VM, so each one has a dedicated JVM and VM
> >>>
> >>> Based on your previous message I was under the impression you were
> using
> >>> the "colocated" feature. *If* you're using this then you definitely are
> >>> running multiple brokers in the same JVM because that's precisely what
> >> that
> >>> feature does. It runs a primary and a backup broker in the *same JVM*.
> If
> >>> you aren't using a "colocated" configuration then I'm not sure what the
> >>> original question is about. Can you clarify?
> >>>
> >>>
> >>> Justin
> >>>
> >>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
> >> wrote:
> >>>
> >>> Hi Justin
> >>>
> >>> Thank you for your input.
> >>>
> >>> Sorry, should have been clearer on our setup - We are not running
> >> multiple
> >>> brokers on the same JVM but a single instance per VM, so each one has a
> >>> dedicated JVM and VM
> >>>
> >>> Thanks
> >>> Roy
> >>>
> >>>
> >>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
> >>>
> >>> I'm not entirely sure if the configuration you want is possible. You
> >>>
> >>> might
> >>>
> >>> try using the "group-name" element in the "master" or "slave" element
> of
> >>> "colocated." Only servers with the same group-name will pair together.
> >>>
> >>> Aside from that I would actually recommend against using colocated
> >>>
> >>> brokers.
> >>>
> >>> The original use-case for this functionality was very early cloud
> >>> infrastructure where durable, attached storage was not readily
> available.
> >>> However, since then most (if not all) cloud environments support
> durable
> >>> storage separate from the broker so that if the broker goes down a new,
> >>> identical broker can be spun-up relatively quickly and attached to the
> >>>
> >>> same
> >>>
> >>> storage. This provides functional high availability without the need
> for
> >>> any idle backups or replication of any kind which functionally
> nullifies
> >>> this feature.
> >>>
> >>> Additionally, it turns out that (surprise!) configuring & running
> >>>
> >>> multiple
> >>>
> >>> brokers in the same JVM is difficult and error-prone not to mention the
> >>> complication of dynamically coordinating the acquisition of backups in
> a
> >>> running cluster and protecting against split-brain.
> >>>
> >>>
> >>> Justin
> >>>
> >>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>
> wrote:
> >>>
> >>> Hello everyone
> >>>
> >>> We have a setup of three Artemis brokers (very old version don’t ask
> :))
> >>>
> >>> We would like to configure the co located backups such that the backups
> >>> are sent in this order:
> >>>
> >>> Broker01 -> Broker02
> >>> Broker02 -> Broker03
> >>> Broker03 -> Broker01
> >>>
> >>>
> >>> I was reading on co located backups here:
> >>>
> >>>
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >>>
> >>> however not sure I fully understand how to configure the xml section to
> >>> achieve that.
> >>>
> >>> Shall I add excludes in each broker, i.e.
> >>>
> >>>   <colocated>
> >>>      <excludes>
> >>>         <connector-ref>...</connector-ref>
> >>>      </excludes>
> >>>
> >>> Any help would be appreciated.
> >>>
> >>> Many thanks in advance !
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
I haven’t tried he group-name yet.

With regards to the third broker: The architects believe it’ll improve performance given the amount of messages the brokers need to process (in other words “throw more resources at it…”)


> On 27 Mar 2023, at 18:28, Justin Bertram <jb...@apache.org> wrote:
> 
>> What would you suggest is to do ?
> 
> Did you try my previous suggestion already (i.e. using the "group-name"
> element in the "master" or "slave" element of "colocated")?
> 
> Aside from that, do you know why you were asked to add another broker?
> Depending on the reason it may not be a good solution.
> 
> 
> Justin
> 
> On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Hi Justin
>> 
>> It is a good question I honestly don’t have the answer for. I inherited
>> this configuration and was asked to add a third broker and to ensure the co
>> located backups are being done in such a way that each broker points on
>> another. Perhaps those who asked for it don’t fully understand Artemis ! :)
>> 
>> That said, those co located backup on the existing setup with two brokers
>> do work as we have been enabled to recover lost messages in the past. So
>> even not optimal, technically it does work ?
>> 
>> I can only imagine that those who initially designed it about 5 years ago
>> did not use a shared storage to avoid latency.
>> 
>> What would you suggest is to do ?
>> 
>> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
>> 
>> Your screenshot didn't come through the list.
>> 
>> In any case, I'm pretty confused at this point. You're clearly using a
>> colocated configuration that will request a backup from another broker in
>> the cluster, but you say you're not running multiple brokers in the same
>> JVM. If you aren't running multiple brokers in the same JVM then what are
>> you using the colocated configuration for? The whole point of the colocated
>> configuration is to run multiple brokers in the same JVM (i.e. a primary
>> broker and also a backup broker for another primary in the cluster).
>> 
>> 
>> Justin
>> 
>> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com> wrote:
>> 
>>> I don’t believe we are.
>>> 
>>> So assume three Virtual Machines on Azure.
>>> 
>>> Each VM runs one Artemis broker
>>> 
>>> 
>>> All of their ha policy section on all three brokers look like that:
>>> 
>>> <ha-policy>
>>>       <replication>
>>>         <colocated>
>>>           <max-backups>1</max-backups>
>>>           <request-backup>true</request-backup>
>>> 
>>> <backup-request-retry-interval>1000</backup-request-retry-interval>
>>>           <excludes>
>>>             <connector-ref>my-connector</connector-ref>
>>>             <connector-ref>thishostname.mydomain</connector-ref>
>>>           </excludes>
>>>           <master>
>>>             <check-for-live-server>true</check-for-live-server>
>>>           </master>
>>>           <slave>
>>>             <allow-failback>true</allow-failback>
>>>             <restart-backup>true</restart-backup>
>>>             <scale-down/>
>>>           </slave>
>>>         </colocated>
>>>       </replication>
>>>     </ha-policy>
>>> 
>>> 
>>> 
>>> 
>>> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>> We are not running multiple brokers on the same JVM but a single instance
>>> 
>>> per VM, so each one has a dedicated JVM and VM
>>> 
>>> Based on your previous message I was under the impression you were using
>>> the "colocated" feature. *If* you're using this then you definitely are
>>> running multiple brokers in the same JVM because that's precisely what
>> that
>>> feature does. It runs a primary and a backup broker in the *same JVM*. If
>>> you aren't using a "colocated" configuration then I'm not sure what the
>>> original question is about. Can you clarify?
>>> 
>>> 
>>> Justin
>>> 
>>> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
>> wrote:
>>> 
>>> Hi Justin
>>> 
>>> Thank you for your input.
>>> 
>>> Sorry, should have been clearer on our setup - We are not running
>> multiple
>>> brokers on the same JVM but a single instance per VM, so each one has a
>>> dedicated JVM and VM
>>> 
>>> Thanks
>>> Roy
>>> 
>>> 
>>> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
>>> 
>>> I'm not entirely sure if the configuration you want is possible. You
>>> 
>>> might
>>> 
>>> try using the "group-name" element in the "master" or "slave" element of
>>> "colocated." Only servers with the same group-name will pair together.
>>> 
>>> Aside from that I would actually recommend against using colocated
>>> 
>>> brokers.
>>> 
>>> The original use-case for this functionality was very early cloud
>>> infrastructure where durable, attached storage was not readily available.
>>> However, since then most (if not all) cloud environments support durable
>>> storage separate from the broker so that if the broker goes down a new,
>>> identical broker can be spun-up relatively quickly and attached to the
>>> 
>>> same
>>> 
>>> storage. This provides functional high availability without the need for
>>> any idle backups or replication of any kind which functionally nullifies
>>> this feature.
>>> 
>>> Additionally, it turns out that (surprise!) configuring & running
>>> 
>>> multiple
>>> 
>>> brokers in the same JVM is difficult and error-prone not to mention the
>>> complication of dynamically coordinating the acquisition of backups in a
>>> running cluster and protecting against split-brain.
>>> 
>>> 
>>> Justin
>>> 
>>> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
>>> 
>>> Hello everyone
>>> 
>>> We have a setup of three Artemis brokers (very old version don’t ask :))
>>> 
>>> We would like to configure the co located backups such that the backups
>>> are sent in this order:
>>> 
>>> Broker01 -> Broker02
>>> Broker02 -> Broker03
>>> Broker03 -> Broker01
>>> 
>>> 
>>> I was reading on co located backups here:
>>> 
>>> 
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>>> 
>>> however not sure I fully understand how to configure the xml section to
>>> achieve that.
>>> 
>>> Shall I add excludes in each broker, i.e.
>>> 
>>>   <colocated>
>>>      <excludes>
>>>         <connector-ref>...</connector-ref>
>>>      </excludes>
>>> 
>>> Any help would be appreciated.
>>> 
>>> Many thanks in advance !
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
> What would you suggest is to do ?

Did you try my previous suggestion already (i.e. using the "group-name"
element in the "master" or "slave" element of "colocated")?

Aside from that, do you know why you were asked to add another broker?
Depending on the reason it may not be a good solution.


Justin

On Mon, Mar 27, 2023 at 12:07 PM Roy Cohen <ro...@hotmail.com> wrote:

> Hi Justin
>
> It is a good question I honestly don’t have the answer for. I inherited
> this configuration and was asked to add a third broker and to ensure the co
> located backups are being done in such a way that each broker points on
> another. Perhaps those who asked for it don’t fully understand Artemis ! :)
>
> That said, those co located backup on the existing setup with two brokers
> do work as we have been enabled to recover lost messages in the past. So
> even not optimal, technically it does work ?
>
> I can only imagine that those who initially designed it about 5 years ago
> did not use a shared storage to avoid latency.
>
> What would you suggest is to do ?
>
> On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:
>
> Your screenshot didn't come through the list.
>
> In any case, I'm pretty confused at this point. You're clearly using a
> colocated configuration that will request a backup from another broker in
> the cluster, but you say you're not running multiple brokers in the same
> JVM. If you aren't running multiple brokers in the same JVM then what are
> you using the colocated configuration for? The whole point of the colocated
> configuration is to run multiple brokers in the same JVM (i.e. a primary
> broker and also a backup broker for another primary in the cluster).
>
>
> Justin
>
> On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com> wrote:
>
> > I don’t believe we are.
> >
> > So assume three Virtual Machines on Azure.
> >
> > Each VM runs one Artemis broker
> >
> >
> > All of their ha policy section on all three brokers look like that:
> >
> > <ha-policy>
> >        <replication>
> >          <colocated>
> >            <max-backups>1</max-backups>
> >            <request-backup>true</request-backup>
> >
> > <backup-request-retry-interval>1000</backup-request-retry-interval>
> >            <excludes>
> >              <connector-ref>my-connector</connector-ref>
> >              <connector-ref>thishostname.mydomain</connector-ref>
> >            </excludes>
> >            <master>
> >              <check-for-live-server>true</check-for-live-server>
> >            </master>
> >            <slave>
> >              <allow-failback>true</allow-failback>
> >              <restart-backup>true</restart-backup>
> >              <scale-down/>
> >            </slave>
> >          </colocated>
> >        </replication>
> >      </ha-policy>
> >
> >
> >
> >
> > On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
> >
> > We are not running multiple brokers on the same JVM but a single instance
> >
> > per VM, so each one has a dedicated JVM and VM
> >
> > Based on your previous message I was under the impression you were using
> > the "colocated" feature. *If* you're using this then you definitely are
> > running multiple brokers in the same JVM because that's precisely what
> that
> > feature does. It runs a primary and a backup broker in the *same JVM*. If
> > you aren't using a "colocated" configuration then I'm not sure what the
> > original question is about. Can you clarify?
> >
> >
> > Justin
> >
> > On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>
> wrote:
> >
> > Hi Justin
> >
> > Thank you for your input.
> >
> > Sorry, should have been clearer on our setup - We are not running
> multiple
> > brokers on the same JVM but a single instance per VM, so each one has a
> > dedicated JVM and VM
> >
> > Thanks
> > Roy
> >
> >
> > On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
> >
> > I'm not entirely sure if the configuration you want is possible. You
> >
> > might
> >
> > try using the "group-name" element in the "master" or "slave" element of
> > "colocated." Only servers with the same group-name will pair together.
> >
> > Aside from that I would actually recommend against using colocated
> >
> > brokers.
> >
> > The original use-case for this functionality was very early cloud
> > infrastructure where durable, attached storage was not readily available.
> > However, since then most (if not all) cloud environments support durable
> > storage separate from the broker so that if the broker goes down a new,
> > identical broker can be spun-up relatively quickly and attached to the
> >
> > same
> >
> > storage. This provides functional high availability without the need for
> > any idle backups or replication of any kind which functionally nullifies
> > this feature.
> >
> > Additionally, it turns out that (surprise!) configuring & running
> >
> > multiple
> >
> > brokers in the same JVM is difficult and error-prone not to mention the
> > complication of dynamically coordinating the acquisition of backups in a
> > running cluster and protecting against split-brain.
> >
> >
> > Justin
> >
> > On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
> >
> > Hello everyone
> >
> > We have a setup of three Artemis brokers (very old version don’t ask :))
> >
> > We would like to configure the co located backups such that the backups
> > are sent in this order:
> >
> > Broker01 -> Broker02
> > Broker02 -> Broker03
> > Broker03 -> Broker01
> >
> >
> > I was reading on co located backups here:
> >
> >
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >
> > however not sure I fully understand how to configure the xml section to
> > achieve that.
> >
> > Shall I add excludes in each broker, i.e.
> >
> >    <colocated>
> >       <excludes>
> >          <connector-ref>...</connector-ref>
> >       </excludes>
> >
> > Any help would be appreciated.
> >
> > Many thanks in advance !
> >
> >
> >
> >
> >
> >
> >
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Hi Justin

It is a good question I honestly don’t have the answer for. I inherited this configuration and was asked to add a third broker and to ensure the co located backups are being done in such a way that each broker points on another. Perhaps those who asked for it don’t fully understand Artemis ! :)

That said, those co located backup on the existing setup with two brokers do work as we have been enabled to recover lost messages in the past. So even not optimal, technically it does work ?

I can only imagine that those who initially designed it about 5 years ago did not use a shared storage to avoid latency.

What would you suggest is to do ?

On 27 Mar 2023, at 18:00, Justin Bertram <jb...@apache.org> wrote:

Your screenshot didn't come through the list.

In any case, I'm pretty confused at this point. You're clearly using a
colocated configuration that will request a backup from another broker in
the cluster, but you say you're not running multiple brokers in the same
JVM. If you aren't running multiple brokers in the same JVM then what are
you using the colocated configuration for? The whole point of the colocated
configuration is to run multiple brokers in the same JVM (i.e. a primary
broker and also a backup broker for another primary in the cluster).


Justin

On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com> wrote:

> I don’t believe we are.
> 
> So assume three Virtual Machines on Azure.
> 
> Each VM runs one Artemis broker
> 
> 
> All of their ha policy section on all three brokers look like that:
> 
> <ha-policy>
>        <replication>
>          <colocated>
>            <max-backups>1</max-backups>
>            <request-backup>true</request-backup>
> 
> <backup-request-retry-interval>1000</backup-request-retry-interval>
>            <excludes>
>              <connector-ref>my-connector</connector-ref>
>              <connector-ref>thishostname.mydomain</connector-ref>
>            </excludes>
>            <master>
>              <check-for-live-server>true</check-for-live-server>
>            </master>
>            <slave>
>              <allow-failback>true</allow-failback>
>              <restart-backup>true</restart-backup>
>              <scale-down/>
>            </slave>
>          </colocated>
>        </replication>
>      </ha-policy>
> 
> 
> 
> 
> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
> 
> We are not running multiple brokers on the same JVM but a single instance
> 
> per VM, so each one has a dedicated JVM and VM
> 
> Based on your previous message I was under the impression you were using
> the "colocated" feature. *If* you're using this then you definitely are
> running multiple brokers in the same JVM because that's precisely what that
> feature does. It runs a primary and a backup broker in the *same JVM*. If
> you aren't using a "colocated" configuration then I'm not sure what the
> original question is about. Can you clarify?
> 
> 
> Justin
> 
> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com> wrote:
> 
> Hi Justin
> 
> Thank you for your input.
> 
> Sorry, should have been clearer on our setup - We are not running multiple
> brokers on the same JVM but a single instance per VM, so each one has a
> dedicated JVM and VM
> 
> Thanks
> Roy
> 
> 
> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
> 
> I'm not entirely sure if the configuration you want is possible. You
> 
> might
> 
> try using the "group-name" element in the "master" or "slave" element of
> "colocated." Only servers with the same group-name will pair together.
> 
> Aside from that I would actually recommend against using colocated
> 
> brokers.
> 
> The original use-case for this functionality was very early cloud
> infrastructure where durable, attached storage was not readily available.
> However, since then most (if not all) cloud environments support durable
> storage separate from the broker so that if the broker goes down a new,
> identical broker can be spun-up relatively quickly and attached to the
> 
> same
> 
> storage. This provides functional high availability without the need for
> any idle backups or replication of any kind which functionally nullifies
> this feature.
> 
> Additionally, it turns out that (surprise!) configuring & running
> 
> multiple
> 
> brokers in the same JVM is difficult and error-prone not to mention the
> complication of dynamically coordinating the acquisition of backups in a
> running cluster and protecting against split-brain.
> 
> 
> Justin
> 
> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
> 
> Hello everyone
> 
> We have a setup of three Artemis brokers (very old version don’t ask :))
> 
> We would like to configure the co located backups such that the backups
> are sent in this order:
> 
> Broker01 -> Broker02
> Broker02 -> Broker03
> Broker03 -> Broker01
> 
> 
> I was reading on co located backups here:
> 
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> 
> however not sure I fully understand how to configure the xml section to
> achieve that.
> 
> Shall I add excludes in each broker, i.e.
> 
>    <colocated>
>       <excludes>
>          <connector-ref>...</connector-ref>
>       </excludes>
> 
> Any help would be appreciated.
> 
> Many thanks in advance !
> 
> 
> 
> 
> 
> 
>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
Your screenshot didn't come through the list.

In any case, I'm pretty confused at this point. You're clearly using a
colocated configuration that will request a backup from another broker in
the cluster, but you say you're not running multiple brokers in the same
JVM. If you aren't running multiple brokers in the same JVM then what are
you using the colocated configuration for? The whole point of the colocated
configuration is to run multiple brokers in the same JVM (i.e. a primary
broker and also a backup broker for another primary in the cluster).


Justin

On Mon, Mar 27, 2023 at 11:42 AM Roy Cohen <ro...@hotmail.com> wrote:

> I don’t believe we are.
>
> So assume three Virtual Machines on Azure.
>
> Each VM runs one Artemis broker
>
>
> All of their ha policy section on all three brokers look like that:
>
>  <ha-policy>
>         <replication>
>           <colocated>
>             <max-backups>1</max-backups>
>             <request-backup>true</request-backup>
>
> <backup-request-retry-interval>1000</backup-request-retry-interval>
>             <excludes>
>               <connector-ref>my-connector</connector-ref>
>               <connector-ref>thishostname.mydomain</connector-ref>
>             </excludes>
>             <master>
>               <check-for-live-server>true</check-for-live-server>
>             </master>
>             <slave>
>               <allow-failback>true</allow-failback>
>               <restart-backup>true</restart-backup>
>               <scale-down/>
>             </slave>
>           </colocated>
>         </replication>
>       </ha-policy>
>
>
>
>
> On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org> wrote:
>
> We are not running multiple brokers on the same JVM but a single instance
>
> per VM, so each one has a dedicated JVM and VM
>
> Based on your previous message I was under the impression you were using
> the "colocated" feature. *If* you're using this then you definitely are
> running multiple brokers in the same JVM because that's precisely what that
> feature does. It runs a primary and a backup broker in the *same JVM*. If
> you aren't using a "colocated" configuration then I'm not sure what the
> original question is about. Can you clarify?
>
>
> Justin
>
> On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com> wrote:
>
> Hi Justin
>
> Thank you for your input.
>
> Sorry, should have been clearer on our setup - We are not running multiple
> brokers on the same JVM but a single instance per VM, so each one has a
> dedicated JVM and VM
>
> Thanks
> Roy
>
>
> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
>
> I'm not entirely sure if the configuration you want is possible. You
>
> might
>
> try using the "group-name" element in the "master" or "slave" element of
> "colocated." Only servers with the same group-name will pair together.
>
> Aside from that I would actually recommend against using colocated
>
> brokers.
>
> The original use-case for this functionality was very early cloud
> infrastructure where durable, attached storage was not readily available.
> However, since then most (if not all) cloud environments support durable
> storage separate from the broker so that if the broker goes down a new,
> identical broker can be spun-up relatively quickly and attached to the
>
> same
>
> storage. This provides functional high availability without the need for
> any idle backups or replication of any kind which functionally nullifies
> this feature.
>
> Additionally, it turns out that (surprise!) configuring & running
>
> multiple
>
> brokers in the same JVM is difficult and error-prone not to mention the
> complication of dynamically coordinating the acquisition of backups in a
> running cluster and protecting against split-brain.
>
>
> Justin
>
> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
>
> Hello everyone
>
> We have a setup of three Artemis brokers (very old version don’t ask :))
>
> We would like to configure the co located backups such that the backups
> are sent in this order:
>
> Broker01 -> Broker02
> Broker02 -> Broker03
> Broker03 -> Broker01
>
>
> I was reading on co located backups here:
>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>
> however not sure I fully understand how to configure the xml section to
> achieve that.
>
> Shall I add excludes in each broker, i.e.
>
>     <colocated>
>        <excludes>
>           <connector-ref>...</connector-ref>
>        </excludes>
>
> Any help would be appreciated.
>
> Many thanks in advance !
>
>
>
>
>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
I don’t believe we are.

So assume three Virtual Machines on Azure.

Each VM runs one Artemis broker

[cid:2B4DF021-281F-4CFB-B5B5-E94DA3967299]

All of their ha policy section on all three brokers look like that:

 <ha-policy>
        <replication>
          <colocated>
            <max-backups>1</max-backups>
            <request-backup>true</request-backup>
            <backup-request-retry-interval>1000</backup-request-retry-interval>
            <excludes>
              <connector-ref>my-connector</connector-ref>
              <connector-ref>thishostname.mydomain</connector-ref>
            </excludes>
            <master>
              <check-for-live-server>true</check-for-live-server>
            </master>
            <slave>
              <allow-failback>true</allow-failback>
              <restart-backup>true</restart-backup>
              <scale-down/>
            </slave>
          </colocated>
        </replication>
      </ha-policy>




On 27 Mar 2023, at 17:26, Justin Bertram <jb...@apache.org>> wrote:

We are not running multiple brokers on the same JVM but a single instance
per VM, so each one has a dedicated JVM and VM

Based on your previous message I was under the impression you were using
the "colocated" feature. *If* you're using this then you definitely are
running multiple brokers in the same JVM because that's precisely what that
feature does. It runs a primary and a backup broker in the *same JVM*. If
you aren't using a "colocated" configuration then I'm not sure what the
original question is about. Can you clarify?


Justin

On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com>> wrote:

Hi Justin

Thank you for your input.

Sorry, should have been clearer on our setup - We are not running multiple
brokers on the same JVM but a single instance per VM, so each one has a
dedicated JVM and VM

Thanks
Roy


On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org>> wrote:

I'm not entirely sure if the configuration you want is possible. You
might
try using the "group-name" element in the "master" or "slave" element of
"colocated." Only servers with the same group-name will pair together.

Aside from that I would actually recommend against using colocated
brokers.
The original use-case for this functionality was very early cloud
infrastructure where durable, attached storage was not readily available.
However, since then most (if not all) cloud environments support durable
storage separate from the broker so that if the broker goes down a new,
identical broker can be spun-up relatively quickly and attached to the
same
storage. This provides functional high availability without the need for
any idle backups or replication of any kind which functionally nullifies
this feature.

Additionally, it turns out that (surprise!) configuring & running
multiple
brokers in the same JVM is difficult and error-prone not to mention the
complication of dynamically coordinating the acquisition of backups in a
running cluster and protecting against split-brain.


Justin

On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com>> wrote:

Hello everyone

We have a setup of three Artemis brokers (very old version don’t ask :))

We would like to configure the co located backups such that the backups
are sent in this order:

Broker01 -> Broker02
Broker02 -> Broker03
Broker03 -> Broker01


I was reading on co located backups here:

https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
however not sure I fully understand how to configure the xml section to
achieve that.

Shall I add excludes in each broker, i.e.

    <colocated>
       <excludes>
          <connector-ref>...</connector-ref>
       </excludes>

Any help would be appreciated.

Many thanks in advance !

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
> We are not running multiple brokers on the same JVM but a single instance
per VM, so each one has a dedicated JVM and VM

Based on your previous message I was under the impression you were using
the "colocated" feature. *If* you're using this then you definitely are
running multiple brokers in the same JVM because that's precisely what that
feature does. It runs a primary and a backup broker in the *same JVM*. If
you aren't using a "colocated" configuration then I'm not sure what the
original question is about. Can you clarify?


Justin

On Mon, Mar 27, 2023 at 11:07 AM Roy Cohen <ro...@hotmail.com> wrote:

> Hi Justin
>
> Thank you for your input.
>
> Sorry, should have been clearer on our setup - We are not running multiple
> brokers on the same JVM but a single instance per VM, so each one has a
> dedicated JVM and VM
>
> Thanks
> Roy
>
>
> > On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
> >
> > I'm not entirely sure if the configuration you want is possible. You
> might
> > try using the "group-name" element in the "master" or "slave" element of
> > "colocated." Only servers with the same group-name will pair together.
> >
> > Aside from that I would actually recommend against using colocated
> brokers.
> > The original use-case for this functionality was very early cloud
> > infrastructure where durable, attached storage was not readily available.
> > However, since then most (if not all) cloud environments support durable
> > storage separate from the broker so that if the broker goes down a new,
> > identical broker can be spun-up relatively quickly and attached to the
> same
> > storage. This provides functional high availability without the need for
> > any idle backups or replication of any kind which functionally nullifies
> > this feature.
> >
> > Additionally, it turns out that (surprise!) configuring & running
> multiple
> > brokers in the same JVM is difficult and error-prone not to mention the
> > complication of dynamically coordinating the acquisition of backups in a
> > running cluster and protecting against split-brain.
> >
> >
> > Justin
> >
> > On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
> >
> >> Hello everyone
> >>
> >> We have a setup of three Artemis brokers (very old version don’t ask :))
> >>
> >> We would like to configure the co located backups such that the backups
> >> are sent in this order:
> >>
> >> Broker01 -> Broker02
> >> Broker02 -> Broker03
> >> Broker03 -> Broker01
> >>
> >>
> >> I was reading on co located backups here:
> >>
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> >> however not sure I fully understand how to configure the xml section to
> >> achieve that.
> >>
> >> Shall I add excludes in each broker, i.e.
> >>
> >>      <colocated>
> >>         <excludes>
> >>            <connector-ref>...</connector-ref>
> >>         </excludes>
> >>
> >> Any help would be appreciated.
> >>
> >> Many thanks in advance !
> >>
> >>
> >>
>
>

Re: Co Located Backups Question

Posted by Roy Cohen <ro...@hotmail.com>.
Hi Justin

Thank you for your input.

Sorry, should have been clearer on our setup - We are not running multiple brokers on the same JVM but a single instance per VM, so each one has a dedicated JVM and VM

Thanks
Roy


> On 27 Mar 2023, at 16:59, Justin Bertram <jb...@apache.org> wrote:
> 
> I'm not entirely sure if the configuration you want is possible. You might
> try using the "group-name" element in the "master" or "slave" element of
> "colocated." Only servers with the same group-name will pair together.
> 
> Aside from that I would actually recommend against using colocated brokers.
> The original use-case for this functionality was very early cloud
> infrastructure where durable, attached storage was not readily available.
> However, since then most (if not all) cloud environments support durable
> storage separate from the broker so that if the broker goes down a new,
> identical broker can be spun-up relatively quickly and attached to the same
> storage. This provides functional high availability without the need for
> any idle backups or replication of any kind which functionally nullifies
> this feature.
> 
> Additionally, it turns out that (surprise!) configuring & running multiple
> brokers in the same JVM is difficult and error-prone not to mention the
> complication of dynamically coordinating the acquisition of backups in a
> running cluster and protecting against split-brain.
> 
> 
> Justin
> 
> On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:
> 
>> Hello everyone
>> 
>> We have a setup of three Artemis brokers (very old version don’t ask :))
>> 
>> We would like to configure the co located backups such that the backups
>> are sent in this order:
>> 
>> Broker01 -> Broker02
>> Broker02 -> Broker03
>> Broker03 -> Broker01
>> 
>> 
>> I was reading on co located backups here:
>> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
>> however not sure I fully understand how to configure the xml section to
>> achieve that.
>> 
>> Shall I add excludes in each broker, i.e.
>> 
>>      <colocated>
>>         <excludes>
>>            <connector-ref>...</connector-ref>
>>         </excludes>
>> 
>> Any help would be appreciated.
>> 
>> Many thanks in advance !
>> 
>> 
>>

Re: Co Located Backups Question

Posted by Justin Bertram <jb...@apache.org>.
I'm not entirely sure if the configuration you want is possible. You might
try using the "group-name" element in the "master" or "slave" element of
"colocated." Only servers with the same group-name will pair together.

Aside from that I would actually recommend against using colocated brokers.
The original use-case for this functionality was very early cloud
infrastructure where durable, attached storage was not readily available.
However, since then most (if not all) cloud environments support durable
storage separate from the broker so that if the broker goes down a new,
identical broker can be spun-up relatively quickly and attached to the same
storage. This provides functional high availability without the need for
any idle backups or replication of any kind which functionally nullifies
this feature.

Additionally, it turns out that (surprise!) configuring & running multiple
brokers in the same JVM is difficult and error-prone not to mention the
complication of dynamically coordinating the acquisition of backups in a
running cluster and protecting against split-brain.


Justin

On Thu, Mar 23, 2023 at 7:37 AM Roy Cohen <ro...@hotmail.com> wrote:

> Hello everyone
>
> We have a setup of three Artemis brokers (very old version don’t ask :))
>
> We would like to configure the co located backups such that the backups
> are sent in this order:
>
> Broker01 -> Broker02
> Broker02 -> Broker03
> Broker03 -> Broker01
>
>
> I was reading on co located backups here:
> https://activemq.apache.org/components/artemis/documentation/1.0.0/ha.html
> however not sure I fully understand how to configure the xml section to
> achieve that.
>
> Shall I add excludes in each broker, i.e.
>
>       <colocated>
>          <excludes>
>             <connector-ref>...</connector-ref>
>          </excludes>
>
> Any help would be appreciated.
>
> Many thanks in advance !
>
>
>