You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by tpavelka <to...@ca.com> on 2017/07/18 12:16:48 UTC
Duplicate messages received with ActiveMQ 5.13.2
I have a problem where I produce a TextMessage to a topic and the consumer
receives it twice. I only have one system where I can reproduce this and I
would appreciate any tips on how to debug this further.
I have four JVMs, let's call them A, B, C and D. Each one runs an ActiveMQ
broker. The connection topology looks like this:
A -> B
C -> B
D -> B
I.e. B is the only one with broker that runs transport connectors.
Let's say that I have topic T, where C and D have subscribers with
AUTO_ACKNOWLEDGE.
- If A sends a text message to T, then C receives it twice
- If D is not connected then C receives it only once.
I put some debug logs in C and so far I have learned the following about the
two instances received:
1) The messages arrive about 200 ms apart.
2) Both messages have the same id as returned by
javax.jms.Message#getJMSMessageID
3) Both have the same connection as reported by
org.apache.activemq.command.Message#getConnection
4) Both have redelivered as false as reported by
org.apache.activemq.command.ActiveMQMessage#getJMSRedelivered
5) Both have redeliver counter = 0 as reported by
org.apache.activemq.command.Message#getRedeliveryCounter
6) The messages have different broker intime, as reported by
org.apache.activemq.command.Message#getBrokerInTime - The first has
1500305481165 and the second has 1500305481166.
I have tried to reproduce this elsewhere without any success and I am near
the end of my wits.
Thanks for any debugging tips.
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Rob Davies <ra...@gmail.com>.
> On 1 Aug 2017, at 15:36, tpavelka <to...@ca.com> wrote:
>
> It looks like the check for networkSubscription is there for a reason,
> described in https://issues.apache.org/jira/browse/AMQ-2327
> The commit 6c5732b also mentions:
>
> /key is not to conduit proxy/proxy consumers b/c the dependencies cannot be
> easily resolved without more network traffic on additions to demand subs.
> Maintaining the order of consumer advisories fixes duplicate suppression.
> thanks for the easymock test. while brittle it did help focus on checkpaths
> which was key
> /
>
> So the question remains, is the way I am doing it a proper way to do hub and
> spoke? The current behavior means that if each spoke has an embedded broker
> and if there are multiple spokes consuming a topic, then you can get
> duplicate messages. Embedded brokers are useful on both producer and
> consumer, because they allow me to not have to deal with reconnects (the
> embedded broker takes care of that).
>
> There is also a workaround, if I add any policy, then audit processing is
> turned on and duplicate messages are removed. Is such a workaround
> sufficient?
Thomas - for your use case - this is sufficient :)
>
> Thanks,
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729132.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I have opened an issue for this and added a unit test that reproduces the
problem to it:
https://issues.apache.org/jira/browse/AMQ-6784
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729221.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
It looks like the check for networkSubscription is there for a reason,
described in https://issues.apache.org/jira/browse/AMQ-2327
The commit 6c5732b also mentions:
/key is not to conduit proxy/proxy consumers b/c the dependencies cannot be
easily resolved without more network traffic on additions to demand subs.
Maintaining the order of consumer advisories fixes duplicate suppression.
thanks for the easymock test. while brittle it did help focus on checkpaths
which was key
/
So the question remains, is the way I am doing it a proper way to do hub and
spoke? The current behavior means that if each spoke has an embedded broker
and if there are multiple spokes consuming a topic, then you can get
duplicate messages. Embedded brokers are useful on both producer and
consumer, because they allow me to not have to deal with reconnects (the
embedded broker takes care of that).
There is also a workaround, if I add any policy, then audit processing is
turned on and duplicate messages are removed. Is such a workaround
sufficient?
Thanks,
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729132.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I think I am confusing things by describing the problem with words, so here
is a plain ActiveMQ implementation that reproduces the problem. If I run
this, I send one message and receive four instead of the expected two.
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.activemq.broker.BrokerService;
import org.apache.activemq.broker.TransportConnector;
import org.apache.activemq.network.NetworkConnector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.jms.*;
import java.lang.IllegalStateException;
public class DuplicationPlainAMQ {
public static final Logger LOG =
LoggerFactory.getLogger(DuplicationPlainAMQ.class);
public static final String HOST = "localhost";
public static final int PORT = 61616;
public static final String TOPIC = "DTT";
public static void main(String[] args) throws Exception {
try(Hub b = new Hub();
Spoke a = new Spoke("A");
Spoke c = new Spoke("C");
Spoke d = new Spoke("D")) {
c.registerListener();
d.registerListener();
Thread.sleep(1000); // wait for broker and consumer info to
propagate
a.produce();
Thread.sleep(1000); // wait for messages to arrive
}
}
private static class Spoke implements AutoCloseable, ExceptionListener {
private final BrokerService embeddedBroker;
private final String name;
private final NetworkConnector networkConnector;
private final Connection connection;
private final Session session;
private final Destination destination;
private MessageConsumer consumer = null;
private Spoke(String name) throws Exception {
this.name = name;
embeddedBroker = new BrokerService();
embeddedBroker.setBrokerName(name);
embeddedBroker.start();
if(!embeddedBroker.waitUntilStarted()) {
throw new IllegalStateException("Broker did not start");
}
networkConnector = embeddedBroker
.addNetworkConnector("static:(tcp://"+HOST+":"+PORT+")");
networkConnector.setName("connector"+name);
networkConnector.setDuplex(true);
networkConnector.setNetworkTTL(3);
networkConnector.start();
ActiveMQConnectionFactory connectionFactory = new
ActiveMQConnectionFactory("vm://"+name);
connection = connectionFactory.createConnection();
connection.start();
connection.setExceptionListener(this);
session = connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);
destination = session.createTopic(TOPIC);
}
private void produce() throws JMSException {
MessageProducer producer = session.createProducer(destination);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
TextMessage message = session.createTextMessage("test");
producer.send(message);
}
private void registerListener() throws JMSException {
consumer = session.createConsumer(destination);
consumer.setMessageListener(new MessageListener() {
@Override
public void onMessage(Message message) {
try {
LOG.info(name+": message received with id
"+message.getJMSMessageID());
} catch (JMSException e) {
LOG.error(e.getLocalizedMessage(), e);
}
}
});
}
@Override
public void close() throws Exception {
if(consumer != null) {
consumer.close();
}
networkConnector.stop();
embeddedBroker.stop();
}
@Override
public void onException(JMSException exception) {
LOG.error(exception.getLocalizedMessage(), exception);
}
}
private static class Hub implements AutoCloseable {
private final BrokerService broker;
private final TransportConnector transportConnector;
private Hub() throws Exception {
broker = new BrokerService();
broker.setBrokerName("B");
broker.start();
if(!broker.waitUntilStarted()) {
throw new IllegalStateException("Broker did not start");
}
transportConnector = broker
.addConnector("tcp://"+HOST+":"+PORT);
transportConnector.setName("transport");
transportConnector.setBrokerService(broker);
transportConnector.start();
}
@Override
public void close() throws Exception {
transportConnector.stop();
broker.stop();
}
}
}
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729100.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I finally understand where the conduit should happen. I was of the impression
that the conduit happens in broker B (hub) when subscriptions come from C
and D. But that can't happen because there are two bridges (B-C and B-D) and
conduit needs to happen within one bridge.
The conduit needs to happen in the bridge A-B. The reason it does not happen
is that the two ConsumerInfos that come (over one transport connection) have
isNetworkSubscription() returning true.
I suspect the networkSubscription is true because the subscription passed
through two brokers (B, C or B, D). Moreover, if I skip the embedded broker
in C and D then networkSubscription is false and conduit happens within the
A-B bridge and there is no message duplication.
Moreover, if I don't use the embedded broker in A and use it in C and D,
then there is no conduit but also no duplication.
So this topology:
spoke [broker] -> hub [broker] <- spoke [broker]
<- spoke [broker]
Leads to message duplication,
But those two don't:
spoke -> hub [broker] <- spoke [broker]
<- spoke [broker]
spoke [broker] -> hub [broker] <- spoke
<- spoke
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729096.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
Wait, there are multiple transport connections between A and B? Why? That
right there seems like a problem, and might be the root cause of the
behavior you're seeing.
The code I see in that processBrokerInfo() method creates the inverse side
of the duplex network connection (under the hood it's not a single two-way
connection, but rather two one-way connections), but I don't see anything
that seems concerning about what it's doing.
Tim
On Mon, Jul 31, 2017 at 8:10 AM, tpavelka <to...@ca.com> wrote:
> I got a little further today:
> The reason that conduit does not happen is that even though ConsumerInfos
> come twice for the same topic, they come over different transport
> connections and thus come to different bridges. For conduit to happen it
> must happen within one bridge.
> Also, I noticed that when A,C and D (spokes) connect to B (hub) they do so
> over duplex connections. I noticed some special processing for duplex
> connections, namely that it creates additional bridges, in
> org.apache.activemq.broker.TransportConnection#processBrokerInfo.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4729068.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I got a little further today:
The reason that conduit does not happen is that even though ConsumerInfos
come twice for the same topic, they come over different transport
connections and thus come to different bridges. For conduit to happen it
must happen within one bridge.
Also, I noticed that when A,C and D (spokes) connect to B (hub) they do so
over duplex connections. I noticed some special processing for duplex
connections, namely that it creates additional bridges, in
org.apache.activemq.broker.TransportConnection#processBrokerInfo.
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729068.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
BTW here is where I am getting stuck:
I was assuming that brokers must somehow learn about consumers on other
brokers. I looked at the code of org.apache.activemq.network.ConduitBridge
and later org.apache.activemq.network.DemandForwardingBridge and it seemed
to me that the means by which the brokers learn about consumers is by
sending and receiving instances of org.apache.activemq.command.ConsumerInfo.
So I put a breakpoint in
org.apache.activemq.transport.TransportSupport#doConsume and I log every
instance of ConsumerInfo. But I am only seeing ConsumerInfo's for some
advisory topics and not for the topic which I am sending messages to.
So either I don't have the tracing setup right or there is some other
channel via which brokers exchange information about consumers.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728963.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
In my broker "B" which is the hub in the hub & spoke topology, and in
org.apache.activemq.network.ConduitBridge#addToAlreadyInterestedConsumers
There is this code:
protected boolean addToAlreadyInterestedConsumers(ConsumerInfo info) {
// search through existing subscriptions and see if we have a match
if (info.isNetworkSubscription()) {
return false;
}
boolean matched = false;
for (DemandSubscription ds : subscriptionMapByLocalId.values()) {
DestinationFilter filter =
DestinationFilter.parseFilter(ds.getLocalInfo().getDestination());
if (canConduit(ds) && filter.matches(info.getDestination())) {
I get two ConsumerInfo instances (as expected, from "C" and "D"), each of
these have isNetworkSubscription = false
On top of that, canConduit(ds) is true but
filter.matches(info.getDestination()) evaluates to false in both cases.
I would expect that when the first one comes, the subscriptionMapByLocalId
would get updated and the second one would result in a conduit. But that
does not happen.
I suspect that the glitch is in subscriptionMapByLocalId not getting updated
when it should...
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4729005.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
Can you set a breakpoint at ConduitBridge:60 and see whether
info.isNetworkSubscription() evaluates to true (i.e. whether you go into
the then block, and therefore return false)? If I'm understanding that code
correctly, I think that may be the reason why the connections aren't being
conduited.
Tim
On Thu, Jul 27, 2017 at 8:54 AM, tpavelka <to...@ca.com> wrote:
> I figured out why I am not seeing ConsumerInfo coming: it comes in wrapped
> in
> ActiveMQMessage (an advisory message) and the ConsumerInfo is contained
> within as a dataStructure.
>
> I don't know why conduit is not happening but I can now trace the code flow
> to within ConduitBridge.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728966.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I figured out why I am not seeing ConsumerInfo coming: it comes in wrapped in
ActiveMQMessage (an advisory message) and the ConsumerInfo is contained
within as a dataStructure.
I don't know why conduit is not happening but I can now trace the code flow
to within ConduitBridge.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728966.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
Sorry, I was so focused on the difference between the enqueue and dequeue
counts that I missed that row in your earlier post.
So that confirms that despite using a DurableConduitBridge, the
subscriptions are not actually being combined, which is indeed what I was
asking. (And yes, your understanding of what happens when subscriptions are
conduited is exactly right.)
The only way that brokers know to forward messages across a dynamically
included networkConnector is by getting messages on the appropriate
advisory topic for a given destination. I've never looked at what type
those messages have, but ConsumerInfo sounds plausible. But if you see the
consumer count increasing without hitting your breakpoint, then that
doesn't match my mental model of how the network bridges work.
I probably won't have time to go through the code before the weekend, but
if you don't have it figured out by then, I can see if I can pinpoint where
you'd want to set your breakpoint to catch the decision about whether to
conduit or not.
Tim
On Jul 27, 2017 7:37 AM, "tpavelka" <to...@ca.com> wrote:
> The tables I sent before have a row named "Number of consumers". These are
> from org.apache.activemq.broker.jmx.DestinationViewMBean#getConsumerCount
>
> I also put a breakpoint in org.apache.activemq.broker.
> region.Topic#dispatch
> And logged the number of consumers
> (org.apache.activemq.broker.region.Topic#consumers.length()), on every
> broker (including A) it was always 2 (of type
> org.apache.activemq.broker.region.TopicSubscription).
>
> Is this what you mean when you say "we have two subscriptions on A"?
>
> Also, can you confirm that my understanding of conduitSubscriptions is
> correct?
> The way I understand it is that if there is a broker (X) who sends a
> message
> to another broker (Y) that has multiple subscriptions for it then X tracks
> it as just one subscription so that it does not have to send the message
> multiple times.
>
> I am trying to debug where exactly does the decision not to conduit get
> made
> but I am still kind of lost in the code.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728962.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
The tables I sent before have a row named "Number of consumers". These are
from org.apache.activemq.broker.jmx.DestinationViewMBean#getConsumerCount
I also put a breakpoint in org.apache.activemq.broker.region.Topic#dispatch
And logged the number of consumers
(org.apache.activemq.broker.region.Topic#consumers.length()), on every
broker (including A) it was always 2 (of type
org.apache.activemq.broker.region.TopicSubscription).
Is this what you mean when you say "we have two subscriptions on A"?
Also, can you confirm that my understanding of conduitSubscriptions is
correct?
The way I understand it is that if there is a broker (X) who sends a message
to another broker (Y) that has multiple subscriptions for it then X tracks
it as just one subscription so that it does not have to send the message
multiple times.
I am trying to debug where exactly does the decision not to conduit get made
but I am still kind of lost in the code.
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728962.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
I don't know of another place where it would be configured, so I think the
more likely explanation is that there's a bug causing the second
subscription to not be conduited onto the first.
But let's first let's confirm that we actually have two subscriptions on A
(because if we don't, then something else is going on). Can you please
attach a JMX viewer such as JConsole to broker A, and navigate the MBeans
to find your topic, and drill into it within the tree to see how many
subscriptions are associated with it? You should see a single subscription
if conduitSubscriptions is working properly, but I expect that you'll see
two.
Tim
On Jul 26, 2017 8:04 AM, "tpavelka" <to...@ca.com> wrote:
> I double checked just to make sure:
>
> There is this code in org.apache.activemq.network.NetworkBridgeFactory
>
> public static DemandForwardingBridge
> createBridge(NetworkBridgeConfiguration configuration,
> Transport
> localTransport, Transport remoteTransport,
> final
> NetworkBridgeListener listener) {
> DemandForwardingBridge result = null;
> if (configuration.isConduitSubscriptions()) {
> // dynamicOnly determines whether durables are auto bridged
> result = new DurableConduitBridge(configuration,
> localTransport,
> remoteTransport);
> } else {
> result = new DemandForwardingBridge(configuration,
> localTransport, remoteTransport);
> }
> if (listener != null) {
> result.setNetworkBridgeListener(listener);
> }
> return result;
> }
>
> I put a breakpoint in and made it display the result of
> configuration.isConduitSubscriptions(), it shows as true for every call
> for
> every end point.
>
> Can you think of any other place that I may have missed that could change
> the conduitSubscribers behavior?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728931.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I double checked just to make sure:
There is this code in org.apache.activemq.network.NetworkBridgeFactory
public static DemandForwardingBridge
createBridge(NetworkBridgeConfiguration configuration,
Transport
localTransport, Transport remoteTransport,
final
NetworkBridgeListener listener) {
DemandForwardingBridge result = null;
if (configuration.isConduitSubscriptions()) {
// dynamicOnly determines whether durables are auto bridged
result = new DurableConduitBridge(configuration, localTransport,
remoteTransport);
} else {
result = new DemandForwardingBridge(configuration,
localTransport, remoteTransport);
}
if (listener != null) {
result.setNetworkBridgeListener(listener);
}
return result;
}
I put a breakpoint in and made it display the result of
configuration.isConduitSubscriptions(), it shows as true for every call for
every end point.
Can you think of any other place that I may have missed that could change
the conduitSubscribers behavior?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728931.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
Those stats clearly show that the duplication occurs when forwarding from A
to B, so my question of whether conduitSubscriptions is in use for that
link is even more pertinent. My guess is that you'll find that it's not,
and that enabling it fixes this behavior.
Tim
On Jul 26, 2017 7:13 AM, "tpavelka" <to...@ca.com> wrote:
> I plan to debug further, but right now I have statistics for one message
> sent
> from A to topic T. This is best shown in a table, but I don't want to mess
> up formatting in email clients, so here it is in CSV:
>
> With audit on:
>
> audit on,A,B,C,D,,
> Enqueue count,1,2,1,1
> Dequeue count,2,2,1,1
> Dispatch count,2,2,1,1
> Forward count,2,2,0,0
> Inflight count,0,0,0,0
> Consumer count,2,2,2,2
>
> With audit off:
>
> audit off,A,B,C,D
> Enqueue count,1,2,2,2
> Dequeue count,2,4,2,2
> Dispatch count,2,4,2,2
> Forward count,2,4,0,0
> Inflight count,0,0,0,0
> Consumer count,2,2,2,2
>
> I can reliably reproduce this now, but I am using some proprietary wrappers
> which I can't share. If I can't find the root cause in reasonable time I
> can
> try to reproduce this in plain ActiveMQ.
>
> Thanks for the help!
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728926.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I plan to debug further, but right now I have statistics for one message sent
from A to topic T. This is best shown in a table, but I don't want to mess
up formatting in email clients, so here it is in CSV:
With audit on:
audit on,A,B,C,D,,
Enqueue count,1,2,1,1
Dequeue count,2,2,1,1
Dispatch count,2,2,1,1
Forward count,2,2,0,0
Inflight count,0,0,0,0
Consumer count,2,2,2,2
With audit off:
audit off,A,B,C,D
Enqueue count,1,2,2,2
Dequeue count,2,4,2,2
Dispatch count,2,4,2,2
Forward count,2,4,0,0
Inflight count,0,0,0,0
Consumer count,2,2,2,2
I can reliably reproduce this now, but I am using some proprietary wrappers
which I can't share. If I can't find the root cause in reasonable time I can
try to reproduce this in plain ActiveMQ.
Thanks for the help!
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728926.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
I can't answer question #2, so I asked the dev list if someone could answer
it.
For #1, what you're seeing is not expected when conduitSubscriptions is in
use. In fact, this behavior is expected when someone fails to use
conduitSubscriptions, so it's odd that you're seeing it with
conduitSubscriptions enabled. Your earlier messages sounded like you
debugged the C-B and D-B links to confirm that conduitSubscriptions was in
used there, but have you confirmed that conduitSubscriptions is also being
applied on the A-B link? That's the only one where it matters.
Tim
Did you confirm which broker is the one where the duplication occurs? Also,
does broker D also get two copies of the message, or does it get one or
zero?
On Jul 25, 2017 6:40 AM, "tpavelka" <to...@ca.com> wrote:
> I can now reliably reproduce the duplication and I can prevent it. However,
> the behavior leaves me a bit puzzled:
>
> The issue is that the broker adds the message twice to the subscription,
> via
> org.apache.activemq.broker.region.TopicSubscription#add
>
> There is a check for duplicate messages,
> org.apache.activemq.broker.region.TopicSubscription#isDuplicate which only
> works if org.apache.activemq.broker.region.TopicSubscription#enableAudit
> is
> enabled (set to true).
>
> enableAutdit is false by default and thus I am getting duplicate messages.
> However, if I add a policy that applies to the topic, then policies have
> org.apache.activemq.broker.region.policy.PolicyEntry#enableAudit which is
> true by default, so any policy applied to the destination will turn audit
> on
> and prevent duplicated messages.
>
> I have a couple of questions:
>
> 1) Is getting duplicate messages expected behavior, if conduitSubscribers
> is
> set to true? (i.e. when I am using
> org.apache.activemq.network.DurableConduitBridge)?
>
> 2) Why does the auditing default change when policies are applied? Or, what
> is the reason that TopicSubscription has audit off by default?
>
> Thanks,
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728871.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I can now reliably reproduce the duplication and I can prevent it. However,
the behavior leaves me a bit puzzled:
The issue is that the broker adds the message twice to the subscription, via
org.apache.activemq.broker.region.TopicSubscription#add
There is a check for duplicate messages,
org.apache.activemq.broker.region.TopicSubscription#isDuplicate which only
works if org.apache.activemq.broker.region.TopicSubscription#enableAudit is
enabled (set to true).
enableAutdit is false by default and thus I am getting duplicate messages.
However, if I add a policy that applies to the topic, then policies have
org.apache.activemq.broker.region.policy.PolicyEntry#enableAudit which is
true by default, so any policy applied to the destination will turn audit on
and prevent duplicated messages.
I have a couple of questions:
1) Is getting duplicate messages expected behavior, if conduitSubscribers is
set to true? (i.e. when I am using
org.apache.activemq.network.DurableConduitBridge)?
2) Why does the auditing default change when policies are applied? Or, what
is the reason that TopicSubscription has audit off by default?
Thanks,
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728871.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
Sorry, I messed up my boolean logic when I wrote my reply. I meant to write
that I have verified that my conduitSubscriptions is set to TRUE so this
should not be the cause. By mistake I have typed false...
I'll try to watch the the enqueue counts to see if I can figure out the
origin.
Thanks,
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728675.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
If you set conduitSubscriptions to true, does the behavior become the
expected behavior? If so, you can read a description of why having it false
would cause this to happen halfway down
http://activemq.apache.org/networks-of-brokers.html.
You can determine where the duplication occurs by watching the enqueue
counts on each broker when you send a single message. You can see those
counts in the web console or via JMX.
Tim
On Jul 19, 2017 1:17 AM, "tpavelka" <to...@ca.com> wrote:
> I am using just regular topics. In my example above C and D connect to
> their
> embedded brokers via VM transport, create a consumer and register a
> listener.
>
> I have debugged it just to make sure and conduitSubscriptions are set to
> false, i.e. org.apache.activemq.network.NetworkBridgeFactory#createBridge
> returns an instance of org.apache.activemq.network.DurableConduitBridge.
>
> Is there anything I could do to trace the flow to see where the message
> gets
> duplicated?
>
> Thanks,
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728671.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by tpavelka <to...@ca.com>.
I am using just regular topics. In my example above C and D connect to their
embedded brokers via VM transport, create a consumer and register a
listener.
I have debugged it just to make sure and conduitSubscriptions are set to
false, i.e. org.apache.activemq.network.NetworkBridgeFactory#createBridge
returns an instance of org.apache.activemq.network.DurableConduitBridge.
Is there anything I could do to trace the flow to see where the message gets
duplicated?
Thanks,
Tomas
--
View this message in context: http://activemq.2283324.n4.nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627p4728671.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Duplicate messages received with ActiveMQ 5.13.2
Posted by Tim Bain <tb...@alumni.duke.edu>.
Do you have conduitSubscriptions set to false for your networkConnectors?
Or are your topics virtual and you're dynamically including both the topic
and the consumer queues?
Both of those could result in duplicate messages on a topic.
Tim
On Jul 18, 2017 6:36 AM, "tpavelka" <to...@ca.com> wrote:
> I have a problem where I produce a TextMessage to a topic and the consumer
> receives it twice. I only have one system where I can reproduce this and I
> would appreciate any tips on how to debug this further.
>
> I have four JVMs, let's call them A, B, C and D. Each one runs an ActiveMQ
> broker. The connection topology looks like this:
>
> A -> B
> C -> B
> D -> B
>
> I.e. B is the only one with broker that runs transport connectors.
>
> Let's say that I have topic T, where C and D have subscribers with
> AUTO_ACKNOWLEDGE.
>
> - If A sends a text message to T, then C receives it twice
> - If D is not connected then C receives it only once.
>
> I put some debug logs in C and so far I have learned the following about
> the
> two instances received:
>
> 1) The messages arrive about 200 ms apart.
> 2) Both messages have the same id as returned by
> javax.jms.Message#getJMSMessageID
> 3) Both have the same connection as reported by
> org.apache.activemq.command.Message#getConnection
> 4) Both have redelivered as false as reported by
> org.apache.activemq.command.ActiveMQMessage#getJMSRedelivered
> 5) Both have redeliver counter = 0 as reported by
> org.apache.activemq.command.Message#getRedeliveryCounter
> 6) The messages have different broker intime, as reported by
> org.apache.activemq.command.Message#getBrokerInTime - The first has
> 1500305481165 and the second has 1500305481166.
>
> I have tried to reproduce this elsewhere without any success and I am near
> the end of my wits.
>
> Thanks for any debugging tips.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>