message duplicates on federated routes

[Fraser Adams <fr...@blueyonder.co.uk>]

Hi all,
we appear to be seeing some fairly large numbers of message duplicates, 
we've had a few other issues that have caused us to be more observant of 
what's passing through the system so we're not sure if this is recent of 
we've simply not noticed before.

We've got quite a heavily federated system with producers delivering to 
multiple brokers (generally three) providing increasing levels of 
consolidation.

when we see dupes we've seen two three or four instances of an item but 
no more and when they occur they appear to happen pretty much 
immediately after each other a few tens of milliseconds apart.

The routes are "push" queue routes (using qpid-route -s) and delivering 
to headers exchanges. There's nothing fancy about the route, so they are 
using normal unreliable links as per the default on federated routes.

can anyone shed any light on this sort of behaviour? Any thoughts on 
what might be causing it and how to minimise the issue.

BTW we've not proven that it is definitely any issue with qpid or the 
routes so we're investigating message counts and sampling messages at 
each hop, but we want to cover off all angles so any thoughts on 
scenarios that may cause significant dupes would be really useful.

MTIA,
Frase

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: message duplicates on federated routes

[Fraser Adams <fr...@blueyonder.co.uk>]

Hi all,
I think we've got to the bottom of this now, the good new is that it 
looks like it's not a qpid issue per se.
We've got a pretty big federated network and we've recently looked into 
partitioning it up better to make management and administration easier 
unfortunately..... during that exercise, and despite having quite a 
detailed transition plan, it looks like a number of the original queue 
routes were left in place instead of being cleaned up when the new paths 
were established.

If I didn't laugh I'd cry :-(

I'd be interested in hearing of how others manage large deployments, the 
tools are good for what they are and QMF gives a lot of good stats, but 
visualising large networks is still pretty hard. I really must crack on 
and get my QMF2 web UI finished, it's my spare time project but it'd 
probably have saved us a load of grief had I actually had it in a state 
I was happy to release :-(


On 29/05/12 23:17, Fraser Adams wrote:
> Hi all,
> we appear to be seeing some fairly large numbers of message 
> duplicates, we've had a few other issues that have caused us to be 
> more observant of what's passing through the system so we're not sure 
> if this is recent of we've simply not noticed before.
>
> We've got quite a heavily federated system with producers delivering 
> to multiple brokers (generally three) providing increasing levels of 
> consolidation.
>
> when we see dupes we've seen two three or four instances of an item 
> but no more and when they occur they appear to happen pretty much 
> immediately after each other a few tens of milliseconds apart.
>
> The routes are "push" queue routes (using qpid-route -s) and 
> delivering to headers exchanges. There's nothing fancy about the 
> route, so they are using normal unreliable links as per the default on 
> federated routes.
>
> can anyone shed any light on this sort of behaviour? Any thoughts on 
> what might be causing it and how to minimise the issue.
>
> BTW we've not proven that it is definitely any issue with qpid or the 
> routes so we're investigating message counts and sampling messages at 
> each hop, but we want to cover off all angles so any thoughts on 
> scenarios that may cause significant dupes would be really useful.
>
> MTIA,
> Frase


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org