duplicate messages

[me really <zm...@yahoo.com>]

Hello,
We are running a two-broker federation. We have a Java producer (0.5) sending messages to Broker 1, which is configured to push to Broker 2, from which a consumer is reading the data. The push link is configured with: 'qpid-route -d -s queue add broker2 broker1 myexchange myqueue'
We are experiencing duplicate messages.
Here's what happens. The producer sends a steady stream of messages, but we cansee that occasionally, the consumer receives duplicate messages. What we see in the logs is that the duplicate messages appear to be transmitted when there is a failover recovery. But the duplicated messages are duplicates of messagesthat were sent on the order of an hour ago.
We are running a version of the C++ broker that has the fix mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=510301(inactive federation links timeout occasionally) -- we are seeing the timeouts at the 8 minute intervals of this bug (we're running 0.5.752581-34, specifically), though not continuously, only when idle enough time.
Can anyone offer any insight into this rather strange behavior?
Thanks,Simon


      

Re: duplicate messages

[me really <zm...@yahoo.com>]

OK, so seems like there's at least a partial resolution to this, which is to use the latest trunk build, due to the bug QPID-1893. This bug relates to the java client not generating heartbeats, fixed 12-Jan-2010.
I'm not entirely sure how not properly responding to heartbeats causes the resend on failover, but with heartbeating, a failover doesn't occur, so in one sense, it's moot. 
But if anyone can offer any insight into why, if no heartbeat occurs, that a message sent before the connection timeout and failover, gets re-sent, would still be appreciated.


      

Re: duplicate messages

[me really <zm...@yahoo.com>]

As a follow-on...
 
The problem appears to be between the Java producer and the initial federated broker.
 
The scenario is that after 8 mins, the connection times out on the broker, and the producer
does failover, and resends a message that was sent previously, and received by a 
consumer.
 
The log files show that the broker is attempting to send heartbeat messages to the
producer, but there is no response. Specifically, the broker records a SENT event, but
there is no corresponding RECV. Conversely, the heartbeat to the other federated broker
does have the SENT/RECV pattern. After the 4th failed heartbeat, the connection is
timed out and the failover happens.
 
So this leads me to think that my producer is (a) either set up incorrectly in terms of
heartbeats, or (2) the messages are not being ack'ed properly, and are hence resent
upon failover.
 
The producer is being started with -Damqj.heartbeat.delay=30, but the log shows a
heartbeat only every two minutes (i.e. the broker default).
 
For the ack, the producer is setting up the session to be auto-ack (and non-transactional):
_session = _connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
 
and the message is sent as
_producer.send(message, DeliveryMode.PERSISTENT, Message.DEFAULT_PRIORITY, Message.DEFAULT_TIME_TO_LIVE);
 
The producer is essentially just the example producer as per the qpid documentation, 
with adjustments for hosts and queue names, where connection params are:
connectionfactory.broker=amqp://guest:guest@/QPID?brokerlist='tcp://localhost:5672?retries='10'&connectdelay='10''
destination.mydest=direct://amq.direct/myqueue?durable='true'&exclusive='false'&autodelete='false'&routingkey='myroute'
 
So, can anyone provide some insight into these more specific questions:
 
(1) what is the proper configuration to set up a Java producer to heartbeat properly with
a target broker?
(1a) Or, to put it another way, what can cause a Java producer to not answer a heartbeat
message from its broker?
(3) What could cause a messages sent in a session set up as "auto-ack" to
not be ack'd?
(4) This is an additional bit of strange behavior. As you can see from the broker connect
string above, it specifies a connectdelay. However, on the producer's failover, the log
shows:
 
FailoverSingleServer:114 - No delay between connect retries, use tcp://host:port?connectdelay='value' to enable.
 
It seems that the connectdelay value has not been picked up. Can anyone conjecture why?
 
Thanks for any help!
-Simon


 
 


--- On Sun, 2/28/10, me really <zm...@yahoo.com> wrote:


From: me really <zm...@yahoo.com>
Subject: duplicate messages
To: users@qpid.apache.org
Date: Sunday, February 28, 2010, 9:37 PM


Hello,
We are running a two-broker federation. We have a Java producer (0.5) sending messages to Broker 1, which is configured to push to Broker 2, from which a consumer is reading the data. The push link is configured with: 'qpid-route -d -s queue add broker2 broker1 myexchange myqueue'
We are experiencing duplicate messages.
Here's what happens. The producer sends a steady stream of messages, but we cansee that occasionally, the consumer receives duplicate messages. What we see in the logs is that the duplicate messages appear to be transmitted when there is a failover recovery. But the duplicated messages are duplicates of messagesthat were sent on the order of an hour ago.
We are running a version of the C++ broker that has the fix mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=510301(inactive federation links timeout occasionally) -- we are seeing the timeouts at the 8 minute intervals of this bug (we're running 0.5.752581-34, specifically), though not continuously, only when idle enough time.
Can anyone offer any insight into this rather strange behavior?
Thanks,Simon