Debugging delays/lost messages/poor throughput

[Toralf Lund <to...@pgs.com>]

I'm trying to get back to an old issue with one of our software 
installations based the C++ QPid Messaging API with AMPQ 0-10; It's been 
discussed here a couple of times before, but unfortunately not a lot has 
come out of it.

To make a long story short, I see the following issues in my software:

 1. qpid::messaging::Sender::send() on a relatively small message
    frequently takes several seconds (often 3 or 4, sometimes perhaps
    10) - even though there should be a reasonable sender capacity.
 2. Based on timestamps etc., it may look like there is sometimes a
    similar delay between the broker and receiver - from the the time
    given by "x-amqp-0-10.timestamp to the actual receive time in an
    application that *should* be ready for new data.
 3. Sometimes heartbeats are lost or take too long, so that the
    application reconnects. Heartbeat interval is currently.
 4. On reconnect, the broker often still has an active connection, which
    results in a session already in use message.

The broker and "client" application are connected to the same network 
switch, and I don't think the load should be *that* high, so I'd expect 
messages to be delivered in a few milliseconds rather than seconds. I 
also don't get the long delays all the time - some messages actually 
seem to arrive quite fast.

This all happens at a remove work site. I don't see the same problem on 
a local "test installation".

My question this time is, how can I check for connectivity issues, 
messaging throughput etc.? I'm thinking this should perhaps be done 
outside ouf or software, to test at a more fundamental level, so as to 
speak. Any suggestions?

- Toralf