Pure Master/Slave Bug? Expired messages not replicated to Slave - update

[An...@attensity.com]

Update: Same problem with ActiveMQ 5.3.0, current 5.3.3- and current 5.4-SNAPSHOT.

Any ideas?

 Andreas

-----Ursprüngliche Nachricht-----
Von: Weber, Andreas, M-ED 
Gesendet: Montag, 5. Juli 2010 09:47
An: 'users@activemq.apache.org'
Betreff: Pure Master/Slave Bug? Expired messages not replicated to Slave

Hi,

I use a Pure Master/Slave configuration with ActiveMQ 5.3.2.
Master/Slave both use the same DLQ configuration with: processExpired="true" processNonPersistent="true"

The normal Master/Slave processing seems to work correctly, actions on Master are always replicated to the Slave.
But there's a problem: Expired messages do go to the Master's DLQ, but this is not adapted in the Slave.

I debugged in the (Slave's) Code and found the appropriate send-to-DLQ-Command/Message arriving at the MasterConnector.
But in further processing this message is filtered out as a duplicate(?) (TransactionBroker.send() resp. ActiveMQMessageAudit.isDuplicate()). It seems that this ProducerSequenceBit, which is checked there, was already set... but here I'm a little bit lost in the code.

So, any ideas why this happens?

Some further note/question: A Slave also seems to have its own Timer to look for expired messages in the queues, even if the Master is still alive. Is that really intended? 
So in the szenario above, the Slave also generates a message to the DLQ for its expired item. But this is filtered out as a duplicate, too. That may be ok here if "normal replication" would work, but I'm generally not sure if a Slave should really run it's own timeout checking of expired messages at all, as long as he's not become a Master.

Best regards,
 Andreas

AW: Pure Master/Slave Bug? Expired messages not replicated to Slave - update

[An...@attensity.com]

Hi Gary,

thanks for your reply.
I opened a JIRA issue: https://issues.apache.org/activemq/browse/AMQ-2810

It's important for us, because in our application design we use the (individual) DLQs like "normal" data queues, which are processed by consumers, and amongst others are filled by expired messages from other queues. And if these are not replicated to the Slave, then we can't handle a Master node failure correctly.

Concerning the limitations you mentioned below, well, I don't really see a direct relation to the current issue. And btw, these are open issues, too - right? So I would hope to have them also fixed in next ActiveMQ release.

BTW, is it correct what I wrote about a Slave having its own handling (Timer) of expired messages?

 Andreas

> -----Ursprüngliche Nachricht-----
> Von: Gary Tully [mailto:gary.tully@gmail.com]
> Gesendet: Dienstag, 6. Juli 2010 10:52
> An: users@activemq.apache.org
> Betreff: Re: Pure Master/Slave Bug? Expired messages not replicated to Slave - update
> 
> This looks like a bug, can u open a jira issue to track this. I think
> that audit on TransactionBroker has caused a problem before, not sure
> what duplicates it is intended to suppress.
> 
> Out of interest, what is your use case for pure master slave?
> Due to the limitations of restarting master/slave pairs (both need to
> be restarted in order) and the fact that only the master can fail, it
> tends not be get that much use in production and as a result not that
> much attention w.r.t bug fixes. Having said that, what is there should
> work as expected with its limitations.
> 
> You may want to consider using a shared file system master/slave setup.
> 
> On 5 July 2010 14:59, <An...@attensity.com> wrote:
> >
> > Update: Same problem with ActiveMQ 5.3.0, current 5.3.3- and current 5.4-SNAPSHOT.
> >
> > Any ideas?
> >
> >  Andreas
> >
> > -----Ursprüngliche Nachricht-----
> > Von: Weber, Andreas, M-ED
> > Gesendet: Montag, 5. Juli 2010 09:47
> > An: 'users@activemq.apache.org'
> > Betreff: Pure Master/Slave Bug? Expired messages not replicated to Slave
> >
> > Hi,
> >
> > I use a Pure Master/Slave configuration with ActiveMQ 5.3.2.
> > Master/Slave both use the same DLQ configuration with: processExpired="true"
> processNonPersistent="true"
> >
> > The normal Master/Slave processing seems to work correctly, actions on Master are always replicated
> to the Slave.
> > But there's a problem: Expired messages do go to the Master's DLQ, but this is not adapted in the
> Slave.
> >
> > I debugged in the (Slave's) Code and found the appropriate send-to-DLQ-Command/Message arriving at
> the MasterConnector.
> > But in further processing this message is filtered out as a duplicate(?) (TransactionBroker.send()
> resp. ActiveMQMessageAudit.isDuplicate()). It seems that this ProducerSequenceBit, which is checked
> there, was already set... but here I'm a little bit lost in the code.
> >
> > So, any ideas why this happens?
> >
> > Some further note/question: A Slave also seems to have its own Timer to look for expired messages in
> the queues, even if the Master is still alive. Is that really intended?
> > So in the szenario above, the Slave also generates a message to the DLQ for its expired item. But
> this is filtered out as a duplicate, too. That may be ok here if "normal replication" would work, but
> I'm generally not sure if a Slave should really run it's own timeout checking of expired messages at
> all, as long as he's not become a Master.
> >
> > Best regards,
> >  Andreas
> 
> 
> 
> --
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com

Re: Pure Master/Slave Bug? Expired messages not replicated to Slave - update

[Gary Tully <ga...@gmail.com>]

This looks like a bug, can u open a jira issue to track this. I think
that audit on TransactionBroker has caused a problem before, not sure
what duplicates it is intended to suppress.

Out of interest, what is your use case for pure master slave?
Due to the limitations of restarting master/slave pairs (both need to
be restarted in order) and the fact that only the master can fail, it
tends not be get that much use in production and as a result not that
much attention w.r.t bug fixes. Having said that, what is there should
work as expected with its limitations.

You may want to consider using a shared file system master/slave setup.

On 5 July 2010 14:59, <An...@attensity.com> wrote:
>
> Update: Same problem with ActiveMQ 5.3.0, current 5.3.3- and current 5.4-SNAPSHOT.
>
> Any ideas?
>
>  Andreas
>
> -----Ursprüngliche Nachricht-----
> Von: Weber, Andreas, M-ED
> Gesendet: Montag, 5. Juli 2010 09:47
> An: 'users@activemq.apache.org'
> Betreff: Pure Master/Slave Bug? Expired messages not replicated to Slave
>
> Hi,
>
> I use a Pure Master/Slave configuration with ActiveMQ 5.3.2.
> Master/Slave both use the same DLQ configuration with: processExpired="true" processNonPersistent="true"
>
> The normal Master/Slave processing seems to work correctly, actions on Master are always replicated to the Slave.
> But there's a problem: Expired messages do go to the Master's DLQ, but this is not adapted in the Slave.
>
> I debugged in the (Slave's) Code and found the appropriate send-to-DLQ-Command/Message arriving at the MasterConnector.
> But in further processing this message is filtered out as a duplicate(?) (TransactionBroker.send() resp. ActiveMQMessageAudit.isDuplicate()). It seems that this ProducerSequenceBit, which is checked there, was already set... but here I'm a little bit lost in the code.
>
> So, any ideas why this happens?
>
> Some further note/question: A Slave also seems to have its own Timer to look for expired messages in the queues, even if the Master is still alive. Is that really intended?
> So in the szenario above, the Slave also generates a message to the DLQ for its expired item. But this is filtered out as a duplicate, too. That may be ok here if "normal replication" would work, but I'm generally not sure if a Slave should really run it's own timeout checking of expired messages at all, as long as he's not become a Master.
>
> Best regards,
>  Andreas



--
http://blog.garytully.com

Open Source Integration
http://fusesource.com