[jira] Resolved: (AMQ-2803) 'Zombie' messages created in KahaDB after failover, with warning 'Duplicate message add attempt rejected.'

["Gary Tully (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/activemq/browse/AMQ-2803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gary Tully resolved AMQ-2803.
-----------------------------

         Assignee: Gary Tully
    Fix Version/s: 5.4.0
       Resolution: Fixed

resolved in r961783

> 'Zombie' messages created in KahaDB after failover, with warning 'Duplicate message add attempt rejected.'
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: AMQ-2803
>                 URL: https://issues.apache.org/activemq/browse/AMQ-2803
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Message Store
>    Affects Versions: 5.3.2
>            Reporter: Adrian Trenaman
>            Assignee: Gary Tully
>             Fix For: 5.4.0
>
>
> Am doing some failover testing of ActiveMQ, sending 10,000 messages to the broker with a live consumer; the producer and consumer are transactional, and the broker is configured to use KahaDB. On failover of the broker, I see log warnings like this: 
> {code}
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:29:1:1:240
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:24:1:1:241
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:23:1:1:242
>  WARN | Duplicate message add attempt rejected. Message id: ID:Ade's-54024-1277715585703-0:0:1:1:237
> {code}
> When the test has run its course, I discover that there are no messages lost - all my messages have gone through successfully. However, I notice that there are in fact a number of 'zombie' messages on the queue: by 'zombie' I mean that the messages are in the store, and the JMX queueSize says they're there. However, if you browse the queue you don't see them, and, if you have a consumer on the queue, you won't receive them either. 
> Looking into the code, in {{org.apache.activemq.store.kahadb.MessageDatabase}} you can see the following very interesting comments concerning the log warning:
> {code}
>                // If the message ID as indexed, then the broker asked us to store a DUP
>                 // message.  Bad BOY!  Don't do it, and log a warning.
>                 LOG.warn("Duplicate message add attempt rejected. Message id: "+command.getMessageId());
>                 // TODO: consider just rolling back the tx.
>                 sd.messageIdIndex.put(tx, command.getMessageId(), previous);
> {code} 
> So. It seems to me that while we're detecting the condition of the broker trying to reinsert a message that's already in the KahaDB correctly, we're not handling this case very well. The outcome is that while we do not get any message loss, we do get worrying warning messages *and* we get erroneous reporting of the queueSize in JMX which will mislead and create the perception of undelivered messages.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.