You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by Garlapati Sreeram Kumar <sr...@live.com> on 2016/04/06 02:23:22 UTC
RE: Proton-j Reactor - Receiver
Hello Robbie,
We are using proton-j client with SSL and many of our customers are hitting this issue.
Here are my findings after debugging through this issue:
- When incoming bytes arrive on the SocketChannel – proton-j client gets signaled by nio & as a result it unwinds the transport stack – as a result all the TransportInput implementations performs its task on the Read Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to frameparser etc).
- While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput reads(16k bytes) from _inputBuffer and the result - decoded bytes are written to _decodedInputBuffer – as an intermediate buffer.
- It then flushes bytes from intermediate buffer to the next layer & invokes an _underlyingInput.Process() – to signal it that it has bytes in its input buffer.
- If the underlyingInput (lets say FrameParser) buffer size is small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over time this accrues.
The fix here is to flush decodedInputBuffer to the Next transport in the Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73
Pl. let me know if we need to do more to fix this issue comprehensively.
Thx!
Sree
From: Robbie Gemmell<ma...@gmail.com>
Sent: Thursday, March 31, 2016 9:19 AM
To: proton@qpid.apache.org<ma...@qpid.apache.org>
Subject: Re: Proton-j Reactor - Receiver
On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sr...@live.com> wrote:
> Hello All!
>
> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages (from Microsoft Azure Event Hubs): https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>
> Am using the onDelivery(Event) callback to receive messages. I really appreciate your help with this issue/behavior:
>
> ISSUE: I noticed that the last few messages on the Queue are not being issued to onDelivery(Event) callback by the Reactor
> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and discovered that the Transfer frames corresponding to those messages were not even delivered to Client. Then, I looked at our Service Proton Frames and can clearly see that they are being delivered by the Service. And other AMQP clients (for ex: .net client can see the Transfer frames)
> - Is this a known behavior?
> Does Reactor code path disable Nagle on underlying socket – could this be related? or is there any other Configuration that we should be setting to see all Transfer frames received on the Socket?
>
> Please advice.
>
> Thanks a lot in Advance!
> Sree
>
> Sent from Mail for Windows 10
>
I'm not aware of anyone else reporting anything like that. I don't see
anything in the code suggesting the reactor sets TCP_NODELAY trueon
the socket, but I wouldn't think that should matter here.
The frame trace logging is done after the bytes are given to the
Transport and are processed into frames, so a lack of logging could
suggest various things such as they didnt actually get there, they
werent processed, something went wrong before they did/were, something
went wrong decoding them, etc. Its hard to say much more without more
info.
Robbie
Re: Proton-j Reactor - Receiver
Posted by Robbie Gemmell <ro...@gmail.com>.
Hi Sree,
Thanks for the analysis and PR, I'll try to take a proper look soon.
It's not an area of the code I'm familiar with so I'll need to have a
bit of a dig myself to see if the change seems ok. I'd note that any
not-insignificant bug fix such as this should probably have a test
with it (and a JIRA, though I see you have since created one of those)
:)
Robbie
On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sr...@live.com> wrote:
> Hello Robbie,
>
> We are using proton-j client with SSL and many of our customers are hitting this issue.
> Here are my findings after debugging through this issue:
>
> - When incoming bytes arrive on the SocketChannel – proton-j client gets signaled by nio & as a result it unwinds the transport stack – as a result all the TransportInput implementations performs its task on the Read Bytes and hands off to the Next Layer in the stack (transport to ssl, ssl to frameparser etc).
>
> - While unwinding that stack, SimpleSSLTransportWrapper.unwrapInput reads(16k bytes) from _inputBuffer and the result - decoded bytes are written to _decodedInputBuffer – as an intermediate buffer.
>
> - It then flushes bytes from intermediate buffer to the next layer & invokes an _underlyingInput.Process() – to signal it that it has bytes in its input buffer.
>
> - If the underlyingInput (lets say FrameParser) buffer size is small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & Over time this accrues.
>
> The fix here is to flush decodedInputBuffer to the Next transport in the Network Stack & call _underlyingInput.Process() - until decodedInputBuffer is empty. Here’s the pull request - https://github.com/apache/qpid-proton/pull/73
>
> Pl. let me know if we need to do more to fix this issue comprehensively.
>
> Thx!
> Sree
>
> From: Robbie Gemmell<ma...@gmail.com>
> Sent: Thursday, March 31, 2016 9:19 AM
> To: proton@qpid.apache.org<ma...@qpid.apache.org>
> Subject: Re: Proton-j Reactor - Receiver
>
> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sr...@live.com> wrote:
>> Hello All!
>>
>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP Messages (from Microsoft Azure Event Hubs): https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124
>>
>> Am using the onDelivery(Event) callback to receive messages. I really appreciate your help with this issue/behavior:
>>
>> ISSUE: I noticed that the last few messages on the Queue are not being issued to onDelivery(Event) callback by the Reactor
>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) and discovered that the Transfer frames corresponding to those messages were not even delivered to Client. Then, I looked at our Service Proton Frames and can clearly see that they are being delivered by the Service. And other AMQP clients (for ex: .net client can see the Transfer frames)
>> - Is this a known behavior?
>> Does Reactor code path disable Nagle on underlying socket – could this be related? or is there any other Configuration that we should be setting to see all Transfer frames received on the Socket?
>>
>> Please advice.
>>
>> Thanks a lot in Advance!
>> Sree
>>
>> Sent from Mail for Windows 10
>>
>
> I'm not aware of anyone else reporting anything like that. I don't see
> anything in the code suggesting the reactor sets TCP_NODELAY trueon
> the socket, but I wouldn't think that should matter here.
>
> The frame trace logging is done after the bytes are given to the
> Transport and are processed into frames, so a lack of logging could
> suggest various things such as they didnt actually get there, they
> werent processed, something went wrong before they did/were, something
> went wrong decoding them, etc. Its hard to say much more without more
> info.
>
> Robbie