You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2014/08/29 10:55:58 UTC
Another strange interoperability quirk between Messenger and the
Java Broker.
Hey all,
I've been messing around writing a soak test for the JavaScript port of
Messenger and came across another difference in behaviour between the
C++ and Java Brokers.
I've attached the code I've written, it's relatively simple in precis
its behaviour is
1. Subscribe to amqp://guest:guest@localhost:5673/#
2. When the subscription address is available send a Message to
amqp://guest:guest@localhost:5673/<subscriptionQueue> where
<subscriptionQueue> has been extracted from the subscription Address.
3. get and accept the Message.
4. send a new Message then repeat from 3.
With the C++ Broker I stand up a WebSocket->TCP Socket proxy that
listens on WS port 5673 and forwards to TCP port 5672, with the Java
Broker I'm using its WebSocket transport listening on WS port 5673.
With the C++ broker the code seems to behave as I'd expect, it basically
just runs "forever" and I've had it send and receive a couple of million
messages and everything is looking OK. On the C++ Broker I can see
msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
With the Java Broker it's weirder. I'm seeing 100 Messages received (my
console goes up to "Content: Message Number 99") but then it stops,
what's even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues
= 0 and msgDepth = 100. That seems really weird to me, my application
has clearly received the Messages, but yet the Broker is not registering
any Dequeues.
Does anyone have any thoughts on this? It's possible that I've done
something wrong, but the C++ Broker seems to be behaving exactly as I'd
intuitively expect and after a couple of million messages my application
was still going strong and didn't look like it was leaking anything.
Cheers,
Frase
Re: Another strange interoperability quirk between Messenger and
the Java Broker.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi again all - and in particular Rob & Gordon.
I posted the stuff below reporting on my observations of different
behaviour between the C++ and Java Broker when being prodded by a
Messenger based client that's writing then reading messages off a queue
with different settings of outgoing and ingoing window (the different
scenarios are described below along with my observations).
In addition another thing that I noticed was that when I ran a soak test
using scenario 3 against the Java Broker it initially looked fine and as
I say with this scenario the incomingWindow was letting the accepts work
and the Broker was reporting msgTotalEnqueues & msgTotalDequeues match
each other however what I noticed in my soak test was that after an
extended period of running (a million or so "Hello World!" type
messages) the performance started to drop off, when I checked top the
Java Broker CPU utilisation had risen to 100%.
I'd perhaps have expected this to happen with one of the scenarios where
msgTotalDequeues wasn't decreasing and msgDepth was growing 'cause then
I'd guess that messages were being sent to some store, but if the queue
wasn't growing then I don't see why the performance would drop off and
the CPU use increase in the way I observed.
I've attached the code I used.
Interested in any thoughts.
Frase
On 29/08/14 13:51, Fraser Adams wrote:
> Thanks Robbie, Rob has just pinged me too.
>
> I've just updated to r1621286 and the following is what I'm currently
> observing.
>
> 1. With no incoming or Outgoing Window set (my original had
> messenger.setOutgoingWindow(1024);) I see:
>
> C++ Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other.
>
> Java Broker - messages seem to be flowing (my consumer is definitely
> seeing the messages) and there doesn't seem to by a 100 message stall,
> but I'm seeing from the stats msgTotalDequeues of zero and
> msgTotalEnqueues & msgDepth growing. It seems very odd to me that I'm
> seeing msgTotalDequeues of zero, when I'm definitely receiving the
> messages!! I'm sending a message body of "Message Number " + count;
> and when I display the received message content I'm seeing the number
> incrementing, so it's not as if I'm continually receiving the oldest
> message on the queue, I definitely seem to be getting all the messages
> I expect, despite it reporting msgTotalDequeues of zero.
>
>
>
> 2. With messenger.setOutgoingWindow(1024); but no explicit settle on
> the sent message (though still doing messenger.accept(t); on the
> tracker I get from messenger.get(message);)
>
> C++ Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - messages stall at 100, msgTotalDequeues still zero,
> msgTotalEnqueues & msgDepth both 100
>
>
>
> 3. With messenger.setIncomingWindow(1024); and no Outgoing window.
>
> C++ Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other.
>
>
>
> 4. With messenger.setIncomingWindow(1024); and
> messenger.setOutgoingWindow(1024); but no explicit settle on the sent
> message (though still doing messenger.accept(t); on the tracker I get
> from messenger.get(message);)
>
> C++ Broker - my client goes *bang*, basically I've got a fixed size
> array buffer that represents the "virtual heap" and that's being
> exceeded so it looks like messages are backing up on the internal
> Messenger queue.
>
> Java Broker - messages stall at 100, msgTotalEnqueues &
> msgTotalDequeues both 100
>
>
>
> 5. With messenger.setIncomingWindow(1024); and
> messenger.setOutgoingWindow(1024); doing messenger.settle(); after I
> do messenger.put(message); message and doing messenger.accept(t); on
> the tracker I get from messenger.get(message);
>
> C++ Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each
> other.
>
>
>
>
> So I'm still observing definite differences in behaviour between the
> C++ and Java Brokers given the same Messenger client code.
>
> TBH I won't pretend that I really understand (or indeed remotely
> understand!!) what the Messenger incoming/outgoing windows are really
> doing, but it looks like a non-zero incoming window is required in
> order to be able to explicitly accept (or I guess reject) a message,
> that I guess makes sense, I'm not sure what message status gets
> returned if not explicitly accepted or rejected - the Java Broker
> appears to need an explicit accept response in order to dequeue the
> message (though I still don't understand why I'm receiving the most
> recent message without dequeues happening). The C++ broker appears to
> dequeue irrespective of whether an explicit accept has been sent. In
> "old speak" it *looks* like the C++ broker is behaving kind of like
> link: {reliability: unreliable} by default with Messenger (in other
> words it doesn't seem to need acknowledgements).
>
> In terms of the outgoing window it seems that if this is zero the Java
> Broker doesn't require an explicit settle, but if it's non-zero it
> does require a settle.
>
> I don't understand the interaction between incoming and outgoing
> windows, so as I say in point 4 above with the C++ broker my client
> goes bang (out of memory) if I have *both* incoming and outgoing
> windows non-zero and no explicit settle, but it's fine if I have a
> non-zero outgoing window and no explicit settle but with a zero
> incoming window, that seems weird to me, but like I say the whole
> window/disposition thing is a bit like voodoo magic to me.
>
>
> Hope this stuff gives some useful food for thought.
> Frase
>
>
> On 29/08/14 12:23, Robbie Gemmell wrote:
>> Hi Fraser,
>>
>> I happened to comment to Rob yesterday afternoon about a very similar
>> situation I was seeing. I worked it through this morning with him and
>> identified some probable bugs, one of which was in the Java broker
>> and Rob
>> has now made a change for:
>>
>> https://issues.apache.org/jira/browse/QPID-6054
>> https://svn.apache.org/r1621281
>>
>> That fixed the issue I noticed, suggest you give it a try.
>>
>> Robbie
>>
>> On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk>
>> wrote:
>>
>>> Hey all,
>>> I've been messing around writing a soak test for the JavaScript port of
>>> Messenger and came across another difference in behaviour between
>>> the C++
>>> and Java Brokers.
>>>
>>> I've attached the code I've written, it's relatively simple in
>>> precis its
>>> behaviour is
>>> 1. Subscribe to amqp://guest:guest@localhost:5673/#
>>> 2. When the subscription address is available send a Message to
>>> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
>>> <subscriptionQueue> has been extracted from the subscription Address.
>>> 3. get and accept the Message.
>>> 4. send a new Message then repeat from 3.
>>>
>>> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that
>>> listens
>>> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
>>> using its WebSocket transport listening on WS port 5673.
>>>
>>> With the C++ broker the code seems to behave as I'd expect, it
>>> basically
>>> just runs "forever" and I've had it send and receive a couple of
>>> million
>>> messages and everything is looking OK. On the C++ Broker I can see
>>> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>>>
>>>
>>> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
>>> console goes up to "Content: Message Number 99") but then it stops,
>>> what's
>>> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues =
>>> 0 and
>>> msgDepth = 100. That seems really weird to me, my application has
>>> clearly
>>> received the Messages, but yet the Broker is not registering any
>>> Dequeues.
>>>
>>> Does anyone have any thoughts on this? It's possible that I've done
>>> something wrong, but the C++ Broker seems to be behaving exactly as I'd
>>> intuitively expect and after a couple of million messages my
>>> application
>>> was still going strong and didn't look like it was leaking anything.
>>>
>>> Cheers,
>>> Frase
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>
Re: Another strange interoperability quirk between Messenger and
the Java Broker.
Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks Robbie, Rob has just pinged me too.
I've just updated to r1621286 and the following is what I'm currently
observing.
1. With no incoming or Outgoing Window set (my original had
messenger.setOutgoingWindow(1024);) I see:
C++ Broker - everything behaves as I'd intuitively expect, messages keep
flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
Java Broker - messages seem to be flowing (my consumer is definitely
seeing the messages) and there doesn't seem to by a 100 message stall,
but I'm seeing from the stats msgTotalDequeues of zero and
msgTotalEnqueues & msgDepth growing. It seems very odd to me that I'm
seeing msgTotalDequeues of zero, when I'm definitely receiving the
messages!! I'm sending a message body of "Message Number " + count; and
when I display the received message content I'm seeing the number
incrementing, so it's not as if I'm continually receiving the oldest
message on the queue, I definitely seem to be getting all the messages I
expect, despite it reporting msgTotalDequeues of zero.
2. With messenger.setOutgoingWindow(1024); but no explicit settle on the
sent message (though still doing messenger.accept(t); on the tracker I
get from messenger.get(message);)
C++ Broker - everything behaves as I'd intuitively expect, messages keep
flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
Looks like it's doing exactly the same as it was in scenario 1.
Java Broker - messages stall at 100, msgTotalDequeues still zero,
msgTotalEnqueues & msgDepth both 100
3. With messenger.setIncomingWindow(1024); and no Outgoing window.
C++ Broker - everything behaves as I'd intuitively expect, messages keep
flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
Looks like it's doing exactly the same as it was in scenario 1.
Java Broker - everything behaves as I'd intuitively expect, messages
keep flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
4. With messenger.setIncomingWindow(1024); and
messenger.setOutgoingWindow(1024); but no explicit settle on the sent
message (though still doing messenger.accept(t); on the tracker I get
from messenger.get(message);)
C++ Broker - my client goes *bang*, basically I've got a fixed size
array buffer that represents the "virtual heap" and that's being
exceeded so it looks like messages are backing up on the internal
Messenger queue.
Java Broker - messages stall at 100, msgTotalEnqueues & msgTotalDequeues
both 100
5. With messenger.setIncomingWindow(1024); and
messenger.setOutgoingWindow(1024); doing messenger.settle(); after I do
messenger.put(message); message and doing messenger.accept(t); on the
tracker I get from messenger.get(message);
C++ Broker - everything behaves as I'd intuitively expect, messages keep
flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
Looks like it's doing exactly the same as it was in scenario 1.
Java Broker - everything behaves as I'd intuitively expect, messages
keep flowing and the msgTotalEnqueues & msgTotalDequeues match each other.
So I'm still observing definite differences in behaviour between the C++
and Java Brokers given the same Messenger client code.
TBH I won't pretend that I really understand (or indeed remotely
understand!!) what the Messenger incoming/outgoing windows are really
doing, but it looks like a non-zero incoming window is required in order
to be able to explicitly accept (or I guess reject) a message, that I
guess makes sense, I'm not sure what message status gets returned if not
explicitly accepted or rejected - the Java Broker appears to need an
explicit accept response in order to dequeue the message (though I still
don't understand why I'm receiving the most recent message without
dequeues happening). The C++ broker appears to dequeue irrespective of
whether an explicit accept has been sent. In "old speak" it *looks* like
the C++ broker is behaving kind of like link: {reliability: unreliable}
by default with Messenger (in other words it doesn't seem to need
acknowledgements).
In terms of the outgoing window it seems that if this is zero the Java
Broker doesn't require an explicit settle, but if it's non-zero it does
require a settle.
I don't understand the interaction between incoming and outgoing
windows, so as I say in point 4 above with the C++ broker my client goes
bang (out of memory) if I have *both* incoming and outgoing windows
non-zero and no explicit settle, but it's fine if I have a non-zero
outgoing window and no explicit settle but with a zero incoming window,
that seems weird to me, but like I say the whole window/disposition
thing is a bit like voodoo magic to me.
Hope this stuff gives some useful food for thought.
Frase
On 29/08/14 12:23, Robbie Gemmell wrote:
> Hi Fraser,
>
> I happened to comment to Rob yesterday afternoon about a very similar
> situation I was seeing. I worked it through this morning with him and
> identified some probable bugs, one of which was in the Java broker and Rob
> has now made a change for:
>
> https://issues.apache.org/jira/browse/QPID-6054
> https://svn.apache.org/r1621281
>
> That fixed the issue I noticed, suggest you give it a try.
>
> Robbie
>
> On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>
>> Hey all,
>> I've been messing around writing a soak test for the JavaScript port of
>> Messenger and came across another difference in behaviour between the C++
>> and Java Brokers.
>>
>> I've attached the code I've written, it's relatively simple in precis its
>> behaviour is
>> 1. Subscribe to amqp://guest:guest@localhost:5673/#
>> 2. When the subscription address is available send a Message to
>> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
>> <subscriptionQueue> has been extracted from the subscription Address.
>> 3. get and accept the Message.
>> 4. send a new Message then repeat from 3.
>>
>> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
>> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
>> using its WebSocket transport listening on WS port 5673.
>>
>> With the C++ broker the code seems to behave as I'd expect, it basically
>> just runs "forever" and I've had it send and receive a couple of million
>> messages and everything is looking OK. On the C++ Broker I can see
>> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>>
>>
>> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
>> console goes up to "Content: Message Number 99") but then it stops, what's
>> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
>> msgDepth = 100. That seems really weird to me, my application has clearly
>> received the Messages, but yet the Broker is not registering any Dequeues.
>>
>> Does anyone have any thoughts on this? It's possible that I've done
>> something wrong, but the C++ Broker seems to be behaving exactly as I'd
>> intuitively expect and after a couple of million messages my application
>> was still going strong and didn't look like it was leaking anything.
>>
>> Cheers,
>> Frase
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Another strange interoperability quirk between Messenger and the
Java Broker.
Posted by Robbie Gemmell <ro...@gmail.com>.
Hi Fraser,
I happened to comment to Rob yesterday afternoon about a very similar
situation I was seeing. I worked it through this morning with him and
identified some probable bugs, one of which was in the Java broker and Rob
has now made a change for:
https://issues.apache.org/jira/browse/QPID-6054
https://svn.apache.org/r1621281
That fixed the issue I noticed, suggest you give it a try.
Robbie
On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> Hey all,
> I've been messing around writing a soak test for the JavaScript port of
> Messenger and came across another difference in behaviour between the C++
> and Java Brokers.
>
> I've attached the code I've written, it's relatively simple in precis its
> behaviour is
> 1. Subscribe to amqp://guest:guest@localhost:5673/#
> 2. When the subscription address is available send a Message to
> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
> <subscriptionQueue> has been extracted from the subscription Address.
> 3. get and accept the Message.
> 4. send a new Message then repeat from 3.
>
> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
> using its WebSocket transport listening on WS port 5673.
>
> With the C++ broker the code seems to behave as I'd expect, it basically
> just runs "forever" and I've had it send and receive a couple of million
> messages and everything is looking OK. On the C++ Broker I can see
> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>
>
> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
> console goes up to "Content: Message Number 99") but then it stops, what's
> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
> msgDepth = 100. That seems really weird to me, my application has clearly
> received the Messages, but yet the Broker is not registering any Dequeues.
>
> Does anyone have any thoughts on this? It's possible that I've done
> something wrong, but the C++ Broker seems to be behaving exactly as I'd
> intuitively expect and after a couple of million messages my application
> was still going strong and didn't look like it was leaking anything.
>
> Cheers,
> Frase
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
Re: Another strange interoperability quirk between Messenger and the
Java Broker.
Posted by Rob Godfrey <ro...@gmail.com>.
Hi Fraser,
so the Java Broker imposes a limit of 100 unsettled messages sent to a
Queue... i.e. it won't give you any more credit if you haven't settled (or
asked the Broker to settle) the messages you have sent. What settlement
mode are you using? Are you settling your messages? It'd probably be good
to see the protocol logs for this interaction.
Similarly on the dequeue side - it'd be good to see the logs. Obviously
the fact that the broker has sent the message does not mean that they will
get dequeued. Again I think the dequeue won't happen until settlement has
occurred.
-- Rob
On 29 August 2014 10:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:
> Hey all,
> I've been messing around writing a soak test for the JavaScript port of
> Messenger and came across another difference in behaviour between the C++
> and Java Brokers.
>
> I've attached the code I've written, it's relatively simple in precis its
> behaviour is
> 1. Subscribe to amqp://guest:guest@localhost:5673/#
> 2. When the subscription address is available send a Message to
> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
> <subscriptionQueue> has been extracted from the subscription Address.
> 3. get and accept the Message.
> 4. send a new Message then repeat from 3.
>
> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
> using its WebSocket transport listening on WS port 5673.
>
> With the C++ broker the code seems to behave as I'd expect, it basically
> just runs "forever" and I've had it send and receive a couple of million
> messages and everything is looking OK. On the C++ Broker I can see
> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>
>
> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
> console goes up to "Content: Message Number 99") but then it stops, what's
> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
> msgDepth = 100. That seems really weird to me, my application has clearly
> received the Messages, but yet the Broker is not registering any Dequeues.
>
> Does anyone have any thoughts on this? It's possible that I've done
> something wrong, but the C++ Broker seems to be behaving exactly as I'd
> intuitively expect and after a couple of million messages my application
> was still going strong and didn't look like it was leaking anything.
>
> Cheers,
> Frase
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>