You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by artemisn00b <pr...@nokia.com> on 2019/03/15 17:49:46 UTC
Not receiving all messages when paging
So, I was testing my clustered setup (I have 3 nodes in a symmetric cluster
and I want all messages to be delivered to all nodes which have consumers),
and I was testing out large messages with a slow consumer. The situation I
ran into was -
1. I send lot of messages fast, and large to broker 1.
2. My client is listening on broker 3, receiving the messages, but slow. I
added a sleep there.
3. Now, a lot of messages get pages on broker 1, and I ungracefully shutdown
broker1.
4. My producer reconnects, and moves to broker2.
5. The client keeps receiving messages on broker3, but only the new ones.
The ones that were pages on broker1, won't be received.
6. Now, even after I bring broker1 up live, the paged messages won't be
delivered to the consumer.
I had to tear down the client and make it connect to broker1, to receive all
the paged messages. This is not something I can do in a production
environment.
Do you guys have any solution to this as well? Would federated address
handle this in a different way?
Thank you so much!
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
broker.xml <http://activemq.2283324.n4.nabble.com/file/t379404/broker.xml>
How can I configure HA on a master - master configuration? I'm using a
cluster only, without HA.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
Awesome. Let me go through it!
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by Clebert Suconic <cl...@gmail.com>.
Right. I suggested federation as it would always redistributed as long as
there are no consumers.
On Fri, Mar 15, 2019 at 3:21 PM Justin Bertram <jb...@apache.org> wrote:
> It looks to me like your problem is just a misconfiguration. You haven't
> configured a redistribution-delay on your cluster-connection which means
> it's using the default of -1 which means messages will never be
> redistributed. If you set this to something >= 0 I believe you'll see the
> behavior you expect.
>
>
> Justin
>
> On Fri, Mar 15, 2019 at 2:15 PM artemisn00b <pr...@nokia.com>
> wrote:
>
> > "Federation has built in resilience to failure so if the target server
> > connection is lost, e.g. due to network failure, federation will retry
> > connecting to the target until it comes back online. When it comes back
> > online it will resume operation as normal."
> >
> >
> > How does that help in my situation? I don't want to be waiting
> indefinitely
> > for a broker to come up before it should start sending messages. Instead,
> > on
> > a broker failure, I want to switch over to an active one right away and
> > start forwarding messages to consumers. Similarly, on a broker failure,
> my
> > consumers also switch to a broker that is live and running.
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
--
Clebert Suconic
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
Thanks!
When a cluster is configured, and a message is being sent to broker1, and
there is no consumer on broker1, but there is one on broker2, will the
messages be paged only on broker2, and not broker1? or both?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by Clebert Suconic <cl...@gmail.com>.
The message is sent into that broker. And later transferred through store
and forward queue.
On Fri, Mar 15, 2019 at 3:52 PM artemisn00b <pr...@nokia.com> wrote:
> Hmm, this is interesting. I tried and I didn't have to bring my broker1 up
> at
> all, to receive all the messages which is strange, as the messages on the
> broker should be lost. So, in my case, when exactly does redistribution
> take
> place ?
>
> For reference -
>
> 1. I send lot of messages fast, and large to broker 1.
> 2. My client is listening on broker 3, receiving the messages, but slow. I
> added a sleep there.
> 3. Now, a lot of messages get pages on broker 1, and I ungracefully
> shutdown
> broker1.
> 4. My producer reconnects, and moves to broker2.
> 5. The client keeps receiving messages on broker3, but only the new ones.
> The ones that were pages on broker1, won't be received.
> 6. Now, even after I bring broker1 up live, the paged messages won't be
> delivered to the consumer.
>
>
> When a cluster is configured, and a message is being sent to broker1, and
> there is no consumer on broker1, but there is one on broker2, will the
> messages be paged only on broker2, and not broker1?
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
--
Clebert Suconic
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
Hmm, this is interesting. I tried and I didn't have to bring my broker1 up at
all, to receive all the messages which is strange, as the messages on the
broker should be lost. So, in my case, when exactly does redistribution take
place ?
For reference -
1. I send lot of messages fast, and large to broker 1.
2. My client is listening on broker 3, receiving the messages, but slow. I
added a sleep there.
3. Now, a lot of messages get pages on broker 1, and I ungracefully shutdown
broker1.
4. My producer reconnects, and moves to broker2.
5. The client keeps receiving messages on broker3, but only the new ones.
The ones that were pages on broker1, won't be received.
6. Now, even after I bring broker1 up live, the paged messages won't be
delivered to the consumer.
When a cluster is configured, and a message is being sent to broker1, and
there is no consumer on broker1, but there is one on broker2, will the
messages be paged only on broker2, and not broker1?
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by Justin Bertram <jb...@apache.org>.
It looks to me like your problem is just a misconfiguration. You haven't
configured a redistribution-delay on your cluster-connection which means
it's using the default of -1 which means messages will never be
redistributed. If you set this to something >= 0 I believe you'll see the
behavior you expect.
Justin
On Fri, Mar 15, 2019 at 2:15 PM artemisn00b <pr...@nokia.com> wrote:
> "Federation has built in resilience to failure so if the target server
> connection is lost, e.g. due to network failure, federation will retry
> connecting to the target until it comes back online. When it comes back
> online it will resume operation as normal."
>
>
> How does that help in my situation? I don't want to be waiting indefinitely
> for a broker to come up before it should start sending messages. Instead,
> on
> a broker failure, I want to switch over to an active one right away and
> start forwarding messages to consumers. Similarly, on a broker failure, my
> consumers also switch to a broker that is live and running.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
"Federation has built in resilience to failure so if the target server
connection is lost, e.g. due to network failure, federation will retry
connecting to the target until it comes back online. When it comes back
online it will resume operation as normal."
How does that help in my situation? I don't want to be waiting indefinitely
for a broker to come up before it should start sending messages. Instead, on
a broker failure, I want to switch over to an active one right away and
start forwarding messages to consumers. Similarly, on a broker failure, my
consumers also switch to a broker that is live and running.
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by Clebert Suconic <cl...@gmail.com>.
Since it's not released yet.. the documentation is here:
https://github.com/apache/activemq-artemis/blob/master/docs/user-manual/en/federation.md
https://github.com/apache/activemq-artemis/blob/master/docs/user-manual/en/federation-queue.md
https://github.com/apache/activemq-artemis/blob/master/docs/user-manual/en/federation-address.md
It will be on the website as soon as the release passed.
On Fri, Mar 15, 2019 at 2:54 PM artemisn00b <pr...@nokia.com> wrote:
>
> Is there documentation for federation? I need to know how it works to find
> out, if it fits in my scenario or not!
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
--
Clebert Suconic
Re: Not receiving all messages when paging
Posted by artemisn00b <pr...@nokia.com>.
Is there documentation for federation? I need to know how it works to find
out, if it fits in my scenario or not!
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: Not receiving all messages when paging
Posted by Clebert Suconic <cl...@gmail.com>.
Perhaps federation would work instead of clustering?
On Fri, Mar 15, 2019 at 1:54 PM Justin Bertram <jb...@apache.org> wrote:
> #5 is expected. If a broker goes down then all the messages on the broker
> are no longer available. You'd need to configure HA to mitigate this.
>
> #6 may be expected depending on your configuration. Can you provide your
> broker.xml?
>
>
> Justin
>
> On Fri, Mar 15, 2019 at 12:49 PM artemisn00b <pr...@nokia.com>
> wrote:
>
> > So, I was testing my clustered setup (I have 3 nodes in a symmetric
> cluster
> > and I want all messages to be delivered to all nodes which have
> consumers),
> > and I was testing out large messages with a slow consumer. The situation
> I
> > ran into was -
> >
> > 1. I send lot of messages fast, and large to broker 1.
> > 2. My client is listening on broker 3, receiving the messages, but slow.
> I
> > added a sleep there.
> > 3. Now, a lot of messages get pages on broker 1, and I ungracefully
> > shutdown
> > broker1.
> > 4. My producer reconnects, and moves to broker2.
> > 5. The client keeps receiving messages on broker3, but only the new ones.
> > The ones that were pages on broker1, won't be received.
> > 6. Now, even after I bring broker1 up live, the paged messages won't be
> > delivered to the consumer.
> >
> > I had to tear down the client and make it connect to broker1, to receive
> > all
> > the paged messages. This is not something I can do in a production
> > environment.
> >
> > Do you guys have any solution to this as well? Would federated address
> > handle this in a different way?
> >
> > Thank you so much!
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
--
Clebert Suconic
Re: Not receiving all messages when paging
Posted by Justin Bertram <jb...@apache.org>.
#5 is expected. If a broker goes down then all the messages on the broker
are no longer available. You'd need to configure HA to mitigate this.
#6 may be expected depending on your configuration. Can you provide your
broker.xml?
Justin
On Fri, Mar 15, 2019 at 12:49 PM artemisn00b <pr...@nokia.com>
wrote:
> So, I was testing my clustered setup (I have 3 nodes in a symmetric cluster
> and I want all messages to be delivered to all nodes which have consumers),
> and I was testing out large messages with a slow consumer. The situation I
> ran into was -
>
> 1. I send lot of messages fast, and large to broker 1.
> 2. My client is listening on broker 3, receiving the messages, but slow. I
> added a sleep there.
> 3. Now, a lot of messages get pages on broker 1, and I ungracefully
> shutdown
> broker1.
> 4. My producer reconnects, and moves to broker2.
> 5. The client keeps receiving messages on broker3, but only the new ones.
> The ones that were pages on broker1, won't be received.
> 6. Now, even after I bring broker1 up live, the paged messages won't be
> delivered to the consumer.
>
> I had to tear down the client and make it connect to broker1, to receive
> all
> the paged messages. This is not something I can do in a production
> environment.
>
> Do you guys have any solution to this as well? Would federated address
> handle this in a different way?
>
> Thank you so much!
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Re: Not receiving all messages when paging
Posted by Clebert Suconic <cl...@gmail.com>.
It's probably waiting message redistribution to kick in...
I would recommend you to as much as you can to have a local consumer
with clustering.
Federation will play with consumer priorities and move the messages
when you don't have enough load at the box and consumers in other
nodes. I think it would play better.
And ...if you give it a try... give it a feedback on the release
please. There's a vote open for 2.7.0 on the dev list.
On Fri, Mar 15, 2019 at 1:49 PM artemisn00b <pr...@nokia.com> wrote:
>
> So, I was testing my clustered setup (I have 3 nodes in a symmetric cluster
> and I want all messages to be delivered to all nodes which have consumers),
> and I was testing out large messages with a slow consumer. The situation I
> ran into was -
>
> 1. I send lot of messages fast, and large to broker 1.
> 2. My client is listening on broker 3, receiving the messages, but slow. I
> added a sleep there.
> 3. Now, a lot of messages get pages on broker 1, and I ungracefully shutdown
> broker1.
> 4. My producer reconnects, and moves to broker2.
> 5. The client keeps receiving messages on broker3, but only the new ones.
> The ones that were pages on broker1, won't be received.
> 6. Now, even after I bring broker1 up live, the paged messages won't be
> delivered to the consumer.
>
> I had to tear down the client and make it connect to broker1, to receive all
> the paged messages. This is not something I can do in a production
> environment.
>
> Do you guys have any solution to this as well? Would federated address
> handle this in a different way?
>
> Thank you so much!
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
--
Clebert Suconic