You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Fraser Adams <fr...@blueyonder.co.uk> on 2014/08/29 10:55:58 UTC

Another strange interoperability quirk between Messenger and the Java Broker.

Hey all,
I've been messing around writing a soak test for the JavaScript port of 
Messenger and came across another difference in behaviour between the 
C++ and Java Brokers.

I've attached the code I've written, it's relatively simple in precis 
its behaviour is
1. Subscribe to amqp://guest:guest@localhost:5673/#
2. When the subscription address is available send a Message to 
amqp://guest:guest@localhost:5673/<subscriptionQueue> where 
<subscriptionQueue> has been extracted from the subscription Address.
3. get and accept the Message.
4. send a new Message then repeat from 3.

With the C++ Broker I stand up a WebSocket->TCP Socket proxy that 
listens on WS port 5673 and forwards to TCP port 5672, with the Java 
Broker I'm using its WebSocket transport listening on WS port 5673.

With the C++ broker the code seems to behave as I'd expect, it basically 
just runs "forever" and I've had it send and receive a couple of million 
messages and everything is looking OK. On the C++ Broker I can see 
msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.


With the Java Broker it's weirder. I'm seeing 100 Messages received (my 
console goes up to "Content: Message Number 99") but then it stops, 
what's even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues 
= 0 and msgDepth = 100. That seems really weird to me, my application 
has clearly received the Messages, but yet the Broker is not registering 
any Dequeues.

Does anyone have any thoughts on this? It's possible that I've done 
something wrong, but the C++ Broker seems to be behaving exactly as I'd 
intuitively expect and after a couple of million messages my application 
was still going strong and didn't look like it was leaking anything.

Cheers,
Frase


Re: Another strange interoperability quirk between Messenger and the Java Broker.

Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Hi again all - and in particular Rob & Gordon.

I posted the stuff below reporting on my observations of different 
behaviour between the C++ and Java Broker when being prodded by a 
Messenger based client that's writing then reading messages off a queue 
with different settings of outgoing and ingoing window (the different 
scenarios are described below along with my observations).

In addition another thing that I noticed was that when I ran a soak test 
using scenario 3 against the Java Broker it initially looked fine and as 
I say with this scenario the incomingWindow was letting the accepts work 
and the Broker was reporting msgTotalEnqueues & msgTotalDequeues match 
each other however what I noticed in my soak test was that after an 
extended period of running (a million or so "Hello World!" type 
messages) the performance started to drop off, when I checked top the 
Java Broker CPU utilisation had risen to 100%.

I'd perhaps have expected this to happen with one of the scenarios where 
msgTotalDequeues wasn't decreasing and msgDepth was growing 'cause then 
I'd guess that messages were being sent to some store, but if the queue 
wasn't growing then I don't see why the performance would drop off and 
the CPU use increase in the way I observed.

I've attached the code I used.

Interested in any thoughts.
Frase



On 29/08/14 13:51, Fraser Adams wrote:
> Thanks Robbie, Rob has just pinged me too.
>
> I've just updated to r1621286 and the following is what I'm currently 
> observing.
>
> 1. With no incoming or Outgoing Window set (my original had 
> messenger.setOutgoingWindow(1024);) I see:
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
> Java Broker - messages seem to be flowing (my consumer is definitely 
> seeing the messages) and there doesn't seem to by a 100 message stall, 
> but I'm seeing from the stats msgTotalDequeues of zero and 
> msgTotalEnqueues & msgDepth growing. It seems very odd to me that I'm 
> seeing msgTotalDequeues of zero, when I'm definitely receiving the 
> messages!! I'm sending a message body of "Message Number " + count; 
> and when I display the received message content I'm seeing the number 
> incrementing, so it's not as if I'm continually receiving the oldest 
> message on the queue, I definitely seem to be getting all the messages 
> I expect, despite it reporting msgTotalDequeues of zero.
>
>
>
> 2. With messenger.setOutgoingWindow(1024); but no explicit settle on 
> the sent message (though still doing messenger.accept(t); on the 
> tracker I get from messenger.get(message);)
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - messages stall at 100, msgTotalDequeues still zero, 
> msgTotalEnqueues & msgDepth both 100
>
>
>
> 3. With messenger.setIncomingWindow(1024); and no Outgoing window.
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
>
>
> 4. With messenger.setIncomingWindow(1024); and 
> messenger.setOutgoingWindow(1024); but no explicit settle on the sent 
> message (though still doing messenger.accept(t); on the tracker I get 
> from messenger.get(message);)
>
> C++ Broker - my client goes *bang*, basically I've got a fixed size 
> array buffer that represents the "virtual heap" and that's being 
> exceeded so it looks like messages are backing up on the internal 
> Messenger queue.
>
> Java Broker - messages stall at 100, msgTotalEnqueues & 
> msgTotalDequeues both 100
>
>
>
> 5. With messenger.setIncomingWindow(1024); and 
> messenger.setOutgoingWindow(1024); doing messenger.settle(); after I 
> do messenger.put(message); message and doing messenger.accept(t); on 
> the tracker I get from messenger.get(message);
>
> C++ Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other. Looks like it's doing exactly the same as it was in scenario 1.
>
> Java Broker - everything behaves as I'd intuitively expect, messages 
> keep flowing and the msgTotalEnqueues & msgTotalDequeues match each 
> other.
>
>
>
>
> So I'm still observing definite differences in behaviour between the 
> C++ and Java Brokers given the same Messenger client code.
>
> TBH I won't pretend that I really understand (or indeed remotely 
> understand!!) what the Messenger incoming/outgoing windows are really 
> doing, but it looks like a non-zero incoming window is required in 
> order to be able to explicitly accept (or I guess reject) a message, 
> that I guess makes sense, I'm not sure what message status gets 
> returned if not explicitly accepted or rejected - the Java Broker 
> appears to need an explicit accept response in order to dequeue the 
> message (though I still don't understand why I'm receiving the most 
> recent message without dequeues happening). The C++ broker appears to 
> dequeue irrespective of whether an explicit accept has been sent. In 
> "old speak" it *looks* like the C++ broker is behaving kind of like 
> link: {reliability: unreliable} by default with Messenger (in other 
> words it doesn't seem to need acknowledgements).
>
> In terms of the outgoing window it seems that if this is zero the Java 
> Broker doesn't require an explicit settle, but if it's non-zero it 
> does require a settle.
>
> I don't understand the interaction between incoming and outgoing 
> windows, so as I say in point 4 above with the C++ broker my client 
> goes bang (out of memory) if I have *both* incoming and outgoing 
> windows non-zero and no explicit settle, but it's fine if I have a 
> non-zero outgoing window and no explicit settle but with a zero 
> incoming window, that seems weird to me, but like I say the whole 
> window/disposition thing is a bit like voodoo magic to me.
>
>
> Hope this stuff gives some useful food for thought.
> Frase
>
>
> On 29/08/14 12:23, Robbie Gemmell wrote:
>> Hi Fraser,
>>
>> I happened to comment to Rob yesterday afternoon about a very similar
>> situation I was seeing. I worked it through this morning with him and
>> identified some probable bugs, one of which was in the Java broker 
>> and Rob
>> has now made a change for:
>>
>> https://issues.apache.org/jira/browse/QPID-6054
>> https://svn.apache.org/r1621281
>>
>> That fixed the issue I noticed, suggest you give it a try.
>>
>> Robbie
>>
>> On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk> 
>> wrote:
>>
>>> Hey all,
>>> I've been messing around writing a soak test for the JavaScript port of
>>> Messenger and came across another difference in behaviour between 
>>> the C++
>>> and Java Brokers.
>>>
>>> I've attached the code I've written, it's relatively simple in 
>>> precis its
>>> behaviour is
>>> 1. Subscribe to amqp://guest:guest@localhost:5673/#
>>> 2. When the subscription address is available send a Message to
>>> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
>>> <subscriptionQueue> has been extracted from the subscription Address.
>>> 3. get and accept the Message.
>>> 4. send a new Message then repeat from 3.
>>>
>>> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that 
>>> listens
>>> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
>>> using its WebSocket transport listening on WS port 5673.
>>>
>>> With the C++ broker the code seems to behave as I'd expect, it 
>>> basically
>>> just runs "forever" and I've had it send and receive a couple of 
>>> million
>>> messages and everything is looking OK. On the C++ Broker I can see
>>> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>>>
>>>
>>> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
>>> console goes up to "Content: Message Number 99") but then it stops, 
>>> what's
>>> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 
>>> 0 and
>>> msgDepth = 100. That seems really weird to me, my application has 
>>> clearly
>>> received the Messages, but yet the Broker is not registering any 
>>> Dequeues.
>>>
>>> Does anyone have any thoughts on this? It's possible that I've done
>>> something wrong, but the C++ Broker seems to be behaving exactly as I'd
>>> intuitively expect and after a couple of million messages my 
>>> application
>>> was still going strong and didn't look like it was leaking anything.
>>>
>>> Cheers,
>>> Frase
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>


Re: Another strange interoperability quirk between Messenger and the Java Broker.

Posted by Fraser Adams <fr...@blueyonder.co.uk>.
Thanks Robbie, Rob has just pinged me too.

I've just updated to r1621286 and the following is what I'm currently 
observing.

1. With no incoming or Outgoing Window set (my original had 
messenger.setOutgoingWindow(1024);) I see:

C++ Broker - everything behaves as I'd intuitively expect, messages keep 
flowing and the msgTotalEnqueues & msgTotalDequeues match each other.

Java Broker - messages seem to be flowing (my consumer is definitely 
seeing the messages) and there doesn't seem to by a 100 message stall, 
but I'm seeing from the stats msgTotalDequeues of zero and 
msgTotalEnqueues & msgDepth growing. It seems very odd to me that I'm 
seeing msgTotalDequeues of zero, when I'm definitely receiving the 
messages!! I'm sending a message body of "Message Number " + count; and 
when I display the received message content I'm seeing the number 
incrementing, so it's not as if I'm continually receiving the oldest 
message on the queue, I definitely seem to be getting all the messages I 
expect, despite it reporting msgTotalDequeues of zero.



2. With messenger.setOutgoingWindow(1024); but no explicit settle on the 
sent message (though still doing messenger.accept(t); on the tracker I 
get from messenger.get(message);)

C++ Broker - everything behaves as I'd intuitively expect, messages keep 
flowing and the msgTotalEnqueues & msgTotalDequeues match each other. 
Looks like it's doing exactly the same as it was in scenario 1.

Java Broker - messages stall at 100, msgTotalDequeues still zero, 
msgTotalEnqueues & msgDepth both 100



3. With messenger.setIncomingWindow(1024); and no Outgoing window.

C++ Broker - everything behaves as I'd intuitively expect, messages keep 
flowing and the msgTotalEnqueues & msgTotalDequeues match each other. 
Looks like it's doing exactly the same as it was in scenario 1.

Java Broker - everything behaves as I'd intuitively expect, messages 
keep flowing and the msgTotalEnqueues & msgTotalDequeues match each other.



4. With messenger.setIncomingWindow(1024); and 
messenger.setOutgoingWindow(1024); but no explicit settle on the sent 
message (though still doing messenger.accept(t); on the tracker I get 
from messenger.get(message);)

C++ Broker - my client goes *bang*, basically I've got a fixed size 
array buffer that represents the "virtual heap" and that's being 
exceeded so it looks like messages are backing up on the internal 
Messenger queue.

Java Broker - messages stall at 100, msgTotalEnqueues & msgTotalDequeues 
both 100



5. With messenger.setIncomingWindow(1024); and 
messenger.setOutgoingWindow(1024); doing messenger.settle(); after I do 
messenger.put(message); message and doing messenger.accept(t); on the 
tracker I get from messenger.get(message);

C++ Broker - everything behaves as I'd intuitively expect, messages keep 
flowing and the msgTotalEnqueues & msgTotalDequeues match each other. 
Looks like it's doing exactly the same as it was in scenario 1.

Java Broker - everything behaves as I'd intuitively expect, messages 
keep flowing and the msgTotalEnqueues & msgTotalDequeues match each other.




So I'm still observing definite differences in behaviour between the C++ 
and Java Brokers given the same Messenger client code.

TBH I won't pretend that I really understand (or indeed remotely 
understand!!) what the Messenger incoming/outgoing windows are really 
doing, but it looks like a non-zero incoming window is required in order 
to be able to explicitly accept (or I guess reject) a message, that I 
guess makes sense, I'm not sure what message status gets returned if not 
explicitly accepted or rejected - the Java Broker appears to need an 
explicit accept response in order to dequeue the message (though I still 
don't understand why I'm receiving the most recent message without 
dequeues happening). The C++ broker appears to dequeue irrespective of 
whether an explicit accept has been sent. In "old speak" it *looks* like 
the C++ broker is behaving kind of like link: {reliability: unreliable} 
by default with Messenger (in other words it doesn't seem to need 
acknowledgements).

In terms of the outgoing window it seems that if this is zero the Java 
Broker doesn't require an explicit settle, but if it's non-zero it does 
require a settle.

I don't understand the interaction between incoming and outgoing 
windows, so as I say in point 4 above with the C++ broker my client goes 
bang (out of memory) if I have *both* incoming and outgoing windows 
non-zero and no explicit settle, but it's fine if I have a non-zero 
outgoing window and no explicit settle but with a zero incoming window, 
that seems weird to me, but like I say the whole window/disposition 
thing is a bit like voodoo magic to me.


Hope this stuff gives some useful food for thought.
Frase


On 29/08/14 12:23, Robbie Gemmell wrote:
> Hi Fraser,
>
> I happened to comment to Rob yesterday afternoon about a very similar
> situation I was seeing. I worked it through this morning with him and
> identified some probable bugs, one of which was in the Java broker and Rob
> has now made a change for:
>
> https://issues.apache.org/jira/browse/QPID-6054
> https://svn.apache.org/r1621281
>
> That fixed the issue I noticed, suggest you give it a try.
>
> Robbie
>
> On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:
>
>> Hey all,
>> I've been messing around writing a soak test for the JavaScript port of
>> Messenger and came across another difference in behaviour between the C++
>> and Java Brokers.
>>
>> I've attached the code I've written, it's relatively simple in precis its
>> behaviour is
>> 1. Subscribe to amqp://guest:guest@localhost:5673/#
>> 2. When the subscription address is available send a Message to
>> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
>> <subscriptionQueue> has been extracted from the subscription Address.
>> 3. get and accept the Message.
>> 4. send a new Message then repeat from 3.
>>
>> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
>> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
>> using its WebSocket transport listening on WS port 5673.
>>
>> With the C++ broker the code seems to behave as I'd expect, it basically
>> just runs "forever" and I've had it send and receive a couple of million
>> messages and everything is looking OK. On the C++ Broker I can see
>> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>>
>>
>> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
>> console goes up to "Content: Message Number 99") but then it stops, what's
>> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
>> msgDepth = 100. That seems really weird to me, my application has clearly
>> received the Messages, but yet the Broker is not registering any Dequeues.
>>
>> Does anyone have any thoughts on this? It's possible that I've done
>> something wrong, but the C++ Broker seems to be behaving exactly as I'd
>> intuitively expect and after a couple of million messages my application
>> was still going strong and didn't look like it was leaking anything.
>>
>> Cheers,
>> Frase
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: Another strange interoperability quirk between Messenger and the Java Broker.

Posted by Robbie Gemmell <ro...@gmail.com>.
Hi Fraser,

I happened to comment to Rob yesterday afternoon about a very similar
situation I was seeing. I worked it through this morning with him and
identified some probable bugs, one of which was in the Java broker and Rob
has now made a change for:

https://issues.apache.org/jira/browse/QPID-6054
https://svn.apache.org/r1621281

That fixed the issue I noticed, suggest you give it a try.

Robbie

On 29 August 2014 09:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:

> Hey all,
> I've been messing around writing a soak test for the JavaScript port of
> Messenger and came across another difference in behaviour between the C++
> and Java Brokers.
>
> I've attached the code I've written, it's relatively simple in precis its
> behaviour is
> 1. Subscribe to amqp://guest:guest@localhost:5673/#
> 2. When the subscription address is available send a Message to
> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
> <subscriptionQueue> has been extracted from the subscription Address.
> 3. get and accept the Message.
> 4. send a new Message then repeat from 3.
>
> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
> using its WebSocket transport listening on WS port 5673.
>
> With the C++ broker the code seems to behave as I'd expect, it basically
> just runs "forever" and I've had it send and receive a couple of million
> messages and everything is looking OK. On the C++ Broker I can see
> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>
>
> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
> console goes up to "Content: Message Number 99") but then it stops, what's
> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
> msgDepth = 100. That seems really weird to me, my application has clearly
> received the Messages, but yet the Broker is not registering any Dequeues.
>
> Does anyone have any thoughts on this? It's possible that I've done
> something wrong, but the C++ Broker seems to be behaving exactly as I'd
> intuitively expect and after a couple of million messages my application
> was still going strong and didn't look like it was leaking anything.
>
> Cheers,
> Frase
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

Re: Another strange interoperability quirk between Messenger and the Java Broker.

Posted by Rob Godfrey <ro...@gmail.com>.
Hi Fraser,

so the Java Broker imposes a limit of 100 unsettled messages sent to a
Queue... i.e. it won't give you any more credit if you haven't settled (or
asked the Broker to settle) the messages you have sent.  What settlement
mode are you using?  Are you settling your messages?  It'd probably be good
to see the protocol logs for this interaction.

Similarly on the dequeue side - it'd be good to see the logs.  Obviously
the fact that the broker has sent the message does not mean that they will
get dequeued.  Again I think the dequeue won't happen until settlement has
occurred.

-- Rob


On 29 August 2014 10:55, Fraser Adams <fr...@blueyonder.co.uk> wrote:

> Hey all,
> I've been messing around writing a soak test for the JavaScript port of
> Messenger and came across another difference in behaviour between the C++
> and Java Brokers.
>
> I've attached the code I've written, it's relatively simple in precis its
> behaviour is
> 1. Subscribe to amqp://guest:guest@localhost:5673/#
> 2. When the subscription address is available send a Message to
> amqp://guest:guest@localhost:5673/<subscriptionQueue> where
> <subscriptionQueue> has been extracted from the subscription Address.
> 3. get and accept the Message.
> 4. send a new Message then repeat from 3.
>
> With the C++ Broker I stand up a WebSocket->TCP Socket proxy that listens
> on WS port 5673 and forwards to TCP port 5672, with the Java Broker I'm
> using its WebSocket transport listening on WS port 5673.
>
> With the C++ broker the code seems to behave as I'd expect, it basically
> just runs "forever" and I've had it send and receive a couple of million
> messages and everything is looking OK. On the C++ Broker I can see
> msgTotalEnqueues and msgTotalDequeues both increasing as I'd expect.
>
>
> With the Java Broker it's weirder. I'm seeing 100 Messages received (my
> console goes up to "Content: Message Number 99") but then it stops, what's
> even odder is that I see msgTotalEnqueues = 100, msgTotalDequeues = 0 and
> msgDepth = 100. That seems really weird to me, my application has clearly
> received the Messages, but yet the Broker is not registering any Dequeues.
>
> Does anyone have any thoughts on this? It's possible that I've done
> something wrong, but the C++ Broker seems to be behaving exactly as I'd
> intuitively expect and after a couple of million messages my application
> was still going strong and didn't look like it was leaking anything.
>
> Cheers,
> Frase
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>