You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Robbie Gemmell <ro...@gmail.com> on 2021/03/01 15:09:55 UTC

Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

On Fri, 26 Feb 2021 at 11:46, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We are trying to find out root cause for this issue. Can you please provide
> details on below query which can help us to investigate for future issue
>
> 1) What is the idle way to check link status between JMS consumer and broker
> at runtime?
>   - We can get to know via TRACE level logs, but we can't turn on in PROD
> server.
>   - Is there anything can we do at runtime to check link status between
> consumer and broker?
>

This isnt simple since the direct link state is well inside the inner
portion of the client, which you dont have direct access to from the
JMS objects in the outer portion. In both the JMS and AMQP sense its
also complicated by parent child relationships, i.e parents
session/connections closing implicitly shuts down their child
resources, even if they aren't explicitly individually closed protocol
wise. Essentially, if the consumer itself thinks it is closed, it will
throw on various methods as you already know.

> 2) To check consumer status - I disabled endpoint and waited for 6-7m, but
> we didn't receive any error for 6-7minutes on "onException" method neither
> any System.Error, so we concluded that consumer was already detached from
> long time, that's why it was not receiving message from more than 12hrs.  Is
> my understanding correct? Or I should do something else to verify?
>


So again, there could be many reasons. I cant reason much from the
information available or say much that I didnt in prior mails.

> 3) We have added JmsConnectionListener and added logger on all operations
> except "onInboundMessage" operation, because "onInboundMessage" operation
> are getting called for receive message.
> I have not enabled logger on session class, because JmsConnectionListener
> will give us enough information.
> Do you think, I need to enabled any other logger which will give us detached
> link information or this should be enough?  (I think, this should be enough,
> just cross verifying for safer side)
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
No, I dont know any ways of 'hacking' ServiceBus to do all things the
test does with its peer. We use a test peer specifically because such
things are often difficult or at least slow. You'd have to ask
Microsoft for questions you have about ServiceBus in any case.

See my mail from last time, there are various options for establishing
your own test specific server to validate particular behaviours:
https://lists.apache.org/thread.html/r6e016adafddf563445d9754cdf284f53f5d45a528adc7c07ee370134%40%3Cusers.qpid.apache.org%3E

Though, essentially copying the test as you did doesn't seem likely to
show anything the test didnt.

You can use protocol trace logs to determine and then reproduce
specific behaviours of interest yourself. From the stracktrace only
though, I could say the 'unknown error from peer' description earlier
in this thread was the result of the server remotely closing the
consumer without specifying the error field to indicate why. That all
seems unrelated to what I was testing, though that would similarly say
'unknown error from peer' if I had not included an error in the remote
session closure to validate it got reported.

On Fri, 12 Mar 2021 at 12:48, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for your quick reply and confirmation on this issue.
> I referred your test case "testRemotelyEndSessionWithMessageListener"  and
> trying to replicate same test case with Azure ServiceBus, So I can replicate
> this issue and provide confirmation for fixes.
>
> Main challenge for me to perform action based on "TestAmqpPeer" class such
> as "remotelyEndLastOpenedSession()", "expectBegin()",
> "waitForAllHandlersToComplete()" operations.
> I think all these operations are performing under Proton library and don't
> have direct access to us.
>
> Please find sample test class -
> https://github.com/abhikt48/servicebus/blob/master/Qpid-57/TestSessionClosedException.java
> It is not working fine due to missing equivalent operation of "TestAmqpPeer"
> Class.
>
> Can you please suggest any hacking through which I can perform same
> functionality of "TestAmqpPeer" with ServiceBus and qpid SDK.
> It will be really helpful if you can provide us any clue to perform
> equivalent operation of "TestAmqpPeer", So I can execute test case and
> confidently share test result with you.
>
> Regarding reproducing of 'unknown error from peer' and complete logger - We
> don't have any standard way to reproduce this issue, infact we are dependent
> on unknown ServiceBus Broker activity and  we don't have any control on
> this.
>
> Going forward, I will enable all logs including "JmsConnectionListener" and
> Traces logs, and deploy below two applications with 56 and 57-SNAPSHOT. I
> will keep these applications running for next 7-10days, and will update you
> if we got same issue again with all traces
> 1. Deploy application with 56.0 version
> 2. Deploy application with 57.0-SNAPSHOT version
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

Thanks a lot for your quick reply and confirmation on this issue. 
I referred your test case "testRemotelyEndSessionWithMessageListener"  and
trying to replicate same test case with Azure ServiceBus, So I can replicate
this issue and provide confirmation for fixes.

Main challenge for me to perform action based on "TestAmqpPeer" class such
as "remotelyEndLastOpenedSession()", "expectBegin()",
"waitForAllHandlersToComplete()" operations.
I think all these operations are performing under Proton library and don't
have direct access to us. 

Please find sample test class -
https://github.com/abhikt48/servicebus/blob/master/Qpid-57/TestSessionClosedException.java
It is not working fine due to missing equivalent operation of "TestAmqpPeer"
Class.

Can you please suggest any hacking through which I can perform same
functionality of "TestAmqpPeer" with ServiceBus and qpid SDK. 
It will be really helpful if you can provide us any clue to perform
equivalent operation of "TestAmqpPeer", So I can execute test case and
confidently share test result with you.

Regarding reproducing of 'unknown error from peer' and complete logger - We
don't have any standard way to reproduce this issue, infact we are dependent
on unknown ServiceBus Broker activity and  we don't have any control on
this. 

Going forward, I will enable all logs including "JmsConnectionListener" and
Traces logs, and deploy below two applications with 56 and 57-SNAPSHOT. I
will keep these applications running for next 7-10days, and will update you
if we got same issue again with all traces
1. Deploy application with 56.0 version 
2. Deploy application with 57.0-SNAPSHOT version

Regards,
Abhishek Kumar




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
On Mon, 8 Mar 2021 at 14:09, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We got this issue again in multiple instances. We got exception stack trace
> this time on "onException" block on one of instances.
>
> Overall Issue - Consumer is not consuming message from endpoint without
> throwing any exception
>
> ##JMS Infrastructure##
> qpid-jms-client - 56.0
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
>     (a)topic:A/subscriptions/a1
>         (b)topic:B/subscriptions/b1
> 1 publisher -
>
> Note - every consumer/publisher have separate JMS session, MessageConsumer
> and
> producer
>
>
> Impact/Action/Observations -
>
> 1. Depolyed Application on -    3/5/2021, 1:14:00.550 PM  -- Application was
> idle not receiving single messages at all
> 2. Got only one "Unknown error from remote peer" error on "onException"
> block, Please refer error stacktrace for more details -
> https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/Exception_Stack_Trace.txt
> 3. Checked connection status with "isConnectionActive()" method - Connection
> was active, so didn't reconnected.
> 4. Checked "isConsumerActive" for first receiver(A/a1) - It was broken, so
> reconnected on this receiver
> 5. Checked "isConsumerActive" for first receiver(B/b1) - it was active, so
> didn't reconnected
> 6. Application was idle over weekend and did tested today. I published
> message on both receiver and found that
>      (a) receiver(A/a1) - consuming msg perfectly, this was broken and
> reconnected after "Unknown error from remote peer"
>          (b) receiver(B/b1) - not consuming messages, this was active at time of
> "Unknown error from remote peer"
>
> Please refer first post for implementation of "isConsumerActive"  and
> "isConnectionActive()"
>
> Application is very light and simple, which i created for testing of
> ServiceBus only. Please refer below thread dump if this can help -
> https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/threadDump.tdump
>
>
> Note - We have more than 100 instances running on Qpid-45, we haven't
> received this type of issue in all 100+ instances. But we are facing
> continuously this issue on some applications which are using Qpid-56.
>
> Main Issues - Whenever we got "Unknown error from remote peer" then only one
> consumer got broken and other consumer status shows active for this error.
> We should receive -
> 1. Multiple connection exception for every receiver
> 2. Or all consumer should be inactive for "Unknown error from remote peer"
>
> Could you please take a look on this issue and let us know for any
> information.
>
> Regards,
> Abhishek Kumar
>
>

The 'unknown error from peer' is identifiable from the stack as a
single remote consumer closure, consistent with the resulting
behaviour described. Other than that, we dont know what else actually
occurred from the information. I suspect its possible the consumer
sessions were each closed though.

Since you have a reproducer, reproducing this with the protocol
logging enabled as well would have given a much better idea, and
another run with the all the regular client loggers at trace as well
as the protocol logging might also have been useful.

You noted having 8 consumer sessions but the thread dump only shows
thread related evidence of 4, with 4 of the original sessions on the
connection (i.e 1-4), presumably for the 'first listener', and another
session (the 786th) evident being used for an async completion
producer. That is consistent with 4 sessions being closed, or not
existing, for the 'second listener'. This all appears to be on the
second connection created.

Having considered this and having another dig at the client, I found
an issue that could be related. Though I could have sworn it did so,
the client did not fire the ExceptionListener if a Session is
remotely-closed while having a consumer with a MessageListener, it
only did so if the consumer itself was remotely-closed. That could be
a potential explanation for the behaviour you see, I have just pushed
a fix for via https://issues.apache.org/jira/browse/QPIDJMS-526 that
you can try out and see if it helps. Although, I would note that
0.45.0 also had the same behaviour, so it shouldn't really make a
difference.

>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

We got this issue again in multiple instances. We got exception stack trace
this time on "onException" block on one of instances.

Overall Issue - Consumer is not consuming message from endpoint without
throwing any exception

##JMS Infrastructure##
qpid-jms-client - 56.0
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
    (a)topic:A/subscriptions/a1
	(b)topic:B/subscriptions/b1
1 publisher -
	
Note - every consumer/publisher have separate JMS session, MessageConsumer
and
producer


Impact/Action/Observations -

1. Depolyed Application on - 	3/5/2021, 1:14:00.550 PM  -- Application was
idle not receiving single messages at all
2. Got only one "Unknown error from remote peer" error on "onException" 
block, Please refer error stacktrace for more details -
https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/Exception_Stack_Trace.txt 
3. Checked connection status with "isConnectionActive()" method - Connection
was active, so didn't reconnected. 
4. Checked "isConsumerActive" for first receiver(A/a1) - It was broken, so
reconnected on this receiver
5. Checked "isConsumerActive" for first receiver(B/b1) - it was active, so
didn't reconnected
6. Application was idle over weekend and did tested today. I published
message on both receiver and found that 
     (a) receiver(A/a1) - consuming msg perfectly, this was broken and
reconnected after "Unknown error from remote peer"
	 (b) receiver(B/b1) - not consuming messages, this was active at time of
"Unknown error from remote peer"
	 
Please refer first post for implementation of "isConsumerActive"  and
"isConnectionActive()"

Application is very light and simple, which i created for testing of
ServiceBus only. Please refer below thread dump if this can help -
https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/threadDump.tdump 


Note - We have more than 100 instances running on Qpid-45, we haven't
received this type of issue in all 100+ instances. But we are facing
continuously this issue on some applications which are using Qpid-56.

Main Issues - Whenever we got "Unknown error from remote peer" then only one
consumer got broken and other consumer status shows active for this error.
We should receive - 
1. Multiple connection exception for every receiver
2. Or all consumer should be inactive for "Unknown error from remote peer"

Could you please take a look on this issue and let us know for any
information.

Regards,
Abhishek Kumar



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
On Thu, 18 Mar 2021 at 12:58, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> One more details on my last post. We have added JmsConnectionListener in
> Qpid-56 application. We got three log entries from JmsConnectionListener.
> Please find below details.
>
> 1. 3/18/2021, 3:48:18.176 AM - "onConsumerClosed" method -  Consumer closed
> of '**-test-**/subscriptions/56-sub1'
>     ExceptionStackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/onConsumerClosed_03_48_18_AM.txt
> 2. 3/18/2021, 3:48:18.245 AM - "onConnectionEstablished" method - Connection
> established URI
> 'amqps://****.servicebus.windows.net?transport.tcpKeepAlive=true&amqp.traceFrames=true'
> 3. 3/18/2021, 3:48:18.381 AM - "onConnectionFailure" method - Connection
> failure
>         ExceptionStackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/onConnectionFailure_03_48_18_AM.txt
>
>
> Surprisingly -
>    1. 3:48:18 AM - when my method checks for "isConnectionActive" then
> connection was active(true) which should be false
>    2. 3:48:18 AM - when my method "isConsumerActive" checks for
> '**-test-**/subs..s/56-sub1' then consumer was active
>    3. Looks like connection got established automatically without reconnect
> attempt and we are not using failover protocol as well.
>
> Questions -
> 1) As we are not using failover protocol, so connection should not be
> recover automatically. Can you please suggest why connection got reconnect
> automatically?

The obvious suggestion is that it didnt happen that way. I've outlined
my thinking of some potential actual behaviour in my earlier reply to
the previous message.

The client literally doesn't load its reconnection logic if you aren't
using failover. It can then only establish a TCP connection during a
factory.createConnection(..) call. The logging shows evidence that
such a call was actually made, as the connection details changed in a
way that happens during those calls.

> 2) Looks like consumer also recover automatically without reconnect attempt.
> We don't have any “consumerEstablished” notification in
> JmsConnectionListener, but based on testing I can confirm that consumer
> still receiving messages.
>    Can you please suggest why consumer got reconnected automatically without
> trying to reconnect consumer?

Similarly, the obvious answer is that it didnt happen that way. The
only place I'm aware the client can reestablish consumers by itself is
in the failover provider logic, which as above simply wasnt loaded if
you aren't using failover. Plus it was a new connection so the client
would have no knowledge of consumers anyway.

>
> Please take a look on this issue and let me know for any information.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

One more details on my last post. We have added JmsConnectionListener in
Qpid-56 application. We got three log entries from JmsConnectionListener.
Please find below details.

1. 3/18/2021, 3:48:18.176 AM - "onConsumerClosed" method -  Consumer closed
of '**-test-**/subscriptions/56-sub1'	
    ExceptionStackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/onConsumerClosed_03_48_18_AM.txt 
2. 3/18/2021, 3:48:18.245 AM - "onConnectionEstablished" method - Connection
established URI
'amqps://****.servicebus.windows.net?transport.tcpKeepAlive=true&amqp.traceFrames=true'
3. 3/18/2021, 3:48:18.381 AM - "onConnectionFailure" method - Connection
failure
	ExceptionStackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/onConnectionFailure_03_48_18_AM.txt 
	

Surprisingly - 
   1. 3:48:18 AM - when my method checks for "isConnectionActive" then
connection was active(true) which should be false
   2. 3:48:18 AM - when my method "isConsumerActive" checks for
'**-test-**/subs..s/56-sub1' then consumer was active
   3. Looks like connection got established automatically without reconnect
attempt and we are not using failover protocol as well.

Questions -  
1) As we are not using failover protocol, so connection should not be
recover automatically. Can you please suggest why connection got reconnect
automatically?
2) Looks like consumer also recover automatically without reconnect attempt.
We don't have any “consumerEstablished” notification in
JmsConnectionListener, but based on testing I can confirm that consumer
still receiving messages.
   Can you please suggest why consumer got reconnected automatically without
trying to reconnect consumer?

Please take a look on this issue and let me know for any information.

Regards,
Abhishek Kumar




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
The logs after 3:45:34.157 show that your application creates a new
session then creates a new sender, then closes a previous sender and
then its session, then sits idle a bit until after a few minutes
Service Bus remotely closes all the links and sessions and then the
connection itself, without any errors specified which will provoke
various 'unknown error from remote peer' exceptions, with those for
consumers with listeners and the connection closure being delivered to
the exception listener. A second connection is then established, and 8
consumers established on it, then eventually a sender.

I would theorise what is happening is that because there are likely to
be multiple exceptions raised here, first for the message listener
remote closures and then one for the connection remote closure, and
the exception listener is necessarily fired async from their actual
arrival as they are processed, and you are doing your own 'detect
closure' in the ExceptionListener, you are quite possibly noticing the
connection has become failed during an ExceptionListener run prompted
by an earlier (consumer related?) exception delivery, either in your
detection check or because attempting to create a consumer/session
throws if the connection is now failed, before you then receive the
exception that explicitly resulted from the connections actual remote
closure. You might then be repeating your checks during subsequent
exception delivery for either a consumer or the connection closure and
indicating the connection isnt closed (which again per previous mails,
it must be for the connection closure exception as the exception
delivered was itself only created after that point) quite possibly
because what you checked isnt the same connection, if your exception
handling already replaced it and the listener can access and operate
on that connection object (this would go back to my earlier notes
around whether your exception listener can only interact with the
connection object it was set on).

That would also fit in with your followup mail. The 'connection
listener' events are also fired async, in the same executor thread as
the ExceptionListener, so you could well see one ExceptionListener
handling after e.g a consumer closure do work that creates a new
connection (as thats also become failed) and cause things leading to
its logging, before the 'connection listener' report is actually made
aware of the prior connections failure. Note there is no 'connection
listener' event for opening consumers/producers/sessions and thus no
related logging.


On Thu, 18 Mar 2021 at 11:06, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We got 'Unknown error from remote peer' today in both applications QPid-56
> and Qpid-57, surprisingly both applications are working fine and consuming
> messages as expected.
>
> However I would like to share interesting details of "QPid-56" application.
> Note - In this application, I am not considering connection is closed for
> JmsConnectionRemotelyClosedException/JmsConnectionFailedException. I am
> relying on “isConnectionActive” method to check connection status– details
> are in the first post of this issue.
>
> ## JMS Infra ##
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
>              (a)topic:A/subscriptions/a1
>              (b)topic:B/subscriptions/b1
> 1 publisher – (session + producer)
> qpid-jms-client - 56.0
>
> Steps -
> 1)
> 3/18/2021, 3:48:18.380 AM - Received error on "onException" - Connection
> exception from 'servicebus_2', isConnectionActive 'true', Reason 'Unknown
> error from remote peer' - 'class
> org.apache.qpid.jms.JmsConnectionRemotelyClosedException'
> StackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/ConnectionActive_true_JmsConnectionRemotelyClosedException.txt
> 2)Checked "isConsumerActive" for first receiver(A/a1) - It was active, so no
> need to reconnect
> 3)Checked "isConsumerActive" for first receiver(B/b1) - it was active, so no
> need to reconnect
>
> 4) Published message at 3/18/2021, 10 AM - Both receiver consumed message
> successfully
>
> We have enable proton TRACES logs. I attach TRACES of last 12 hour, so we
> can't miss anything, you can check after "3/18/2021, 3:45:34.157 AM".
> Please refer -
> https://github.com/abhikt48/servicebus/blob/master/18-March/Proton_Traces.txt
>
> As we have received "JmsConnectionRemotelyClosedException", so we are
> assuming that connection should be broken, but connection and receiver
> wasn't broken. Both receivers are consuming message perfectly.
>
> I am not sure this is expected behaviour or not. Could you please take a
> look on this and suggest your suggestion.
>
> P.S. - I haven't shared TRACES and log details on Qpid-57 application for
> comparison, as expected it is working fine. But if you need TRACES and log
> details of Qpid-57 application as well, then please let me know. I can
> extract and share with you.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

We got 'Unknown error from remote peer' today in both applications QPid-56
and Qpid-57, surprisingly both applications are working fine and consuming
messages as expected.

However I would like to share interesting details of "QPid-56" application.
Note - In this application, I am not considering connection is closed for
JmsConnectionRemotelyClosedException/JmsConnectionFailedException. I am
relying on “isConnectionActive” method to check connection status– details
are in the first post of this issue.

## JMS Infra ##
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
             (a)topic:A/subscriptions/a1
             (b)topic:B/subscriptions/b1
1 publisher – (session + producer)
qpid-jms-client - 56.0

Steps - 
1)
3/18/2021, 3:48:18.380 AM - Received error on "onException" - Connection
exception from 'servicebus_2', isConnectionActive 'true', Reason 'Unknown
error from remote peer' - 'class
org.apache.qpid.jms.JmsConnectionRemotelyClosedException'	
StackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/ConnectionActive_true_JmsConnectionRemotelyClosedException.txt 
2)Checked "isConsumerActive" for first receiver(A/a1) - It was active, so no
need to reconnect
3)Checked "isConsumerActive" for first receiver(B/b1) - it was active, so no
need to reconnect

4) Published message at 3/18/2021, 10 AM - Both receiver consumed message
successfully

We have enable proton TRACES logs. I attach TRACES of last 12 hour, so we
can't miss anything, you can check after "3/18/2021, 3:45:34.157 AM".
Please refer -
https://github.com/abhikt48/servicebus/blob/master/18-March/Proton_Traces.txt 

As we have received "JmsConnectionRemotelyClosedException", so we are
assuming that connection should be broken, but connection and receiver
wasn't broken. Both receivers are consuming message perfectly.

I am not sure this is expected behaviour or not. Could you please take a
look on this and suggest your suggestion.

P.S. - I haven't shared TRACES and log details on Qpid-57 application for
comparison, as expected it is working fine. But if you need TRACES and log
details of Qpid-57 application as well, then please let me know. I can
extract and share with you.

Regards,
Abhishek Kumar




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
Yes, a JmsConnectionRemotelyClosedException, or something with a
ProviderConnectionRemotelyClosedException based cause, is indicates
the connection is done for, that the client received a connection
close frame for some reason and that it wasnt expected (i.e it wasnt
expecting it as response to a requested closure). Similarly a
JmsConnectionFailedException, or something with
ProviderFailedException based cause, can indicate other
connection-breaking issues such as the connection dropping or some
failure in its processing where a close frame wasnt necessarily
received.

On Fri, 12 Mar 2021 at 16:57, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for quick reply.
>
> Can i assume that JMS Connection will be closed for
> "JmsConnectionRemotelyClosedException" or
> "ProviderConnectionRemotelyClosedException" exception.
>
> So whenever I got "JmsConnectionRemotelyClosedException" or
> "ProviderConnectionRemotelyClosedException"  on "onException()" method then
> I will recreate JMS connection.
>
> Please let me know if there are any other JMS connection closed Exception
> classes, so i will handle them as well.
>
> Could you please confirm on this or correct me if I am wrong?
>
>
> Regards,
> Abhishek Kumar
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

Thanks a lot for quick reply. 

Can i assume that JMS Connection will be closed for
"JmsConnectionRemotelyClosedException" or
"ProviderConnectionRemotelyClosedException" exception.

So whenever I got "JmsConnectionRemotelyClosedException" or
"ProviderConnectionRemotelyClosedException"  on "onException()" method then
I will recreate JMS connection.

Please let me know if there are any other JMS connection closed Exception
classes, so i will handle them as well.

Could you please confirm on this or correct me if I am wrong?


Regards,
Abhishek Kumar



--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
I have looked at the reported issue multiple times for a significant
period. I tried reproducing it and was unable to. I got a second
opinion, they independently came to the same conclusion as I had.

I've thus described why I dont believe the behaviour is occurring as
you specified (you may need to report a JVM defect if it were), and so
it would seem something else must be happening. I dont know what that
is, but suggested a possibility. If you investigate and can point to
something specific in the client I can look again, but for now I've
exhausted the available info and not found anything suggesting a
client issue, only the reverse.

The stacktrace did only show Qpid classes, but that is no surprise and
doesn't suggest anything about where an issue might be. The exception
reported was created inside the client, by the client, upon network
activity, in order to report the resulting situation. It isnt going to
reference things outside the client in that case.

On Fri, 12 Mar 2021 at 14:04, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for quick update. I looked on our codebase, logs and
> stacktrace, and I can confirm that
>
> 1. "javax.jms.ExceptionListener#onException()" is not getting called from
> any of our defined classes/methods.
> 2. ExceptionListener is used only by connection object
>
> We got this error in simple application which have -
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
> 1 publisher – (session + producer)
> qpid-jms-client - 56.0
>
> So Application has only one JMS connection and one exception listener which
> is only operated by JMS connection object.
>
> I shared exact stacktrace with you without any customization and we can see
> all call hierarchy from Qpid classes.
>
> We are logging from “JmsConnectionListener” as well on
> "onConnectionFailure()" method. Please find below stacktrace details, but
> there is no difference between them
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_isConnectionActive_False.txt
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_onConnectionFailure_True.txt
>
> With above details, I must request you to please look on this issue again. I
> got this issue in other application as well, where we have multiple JMS
> connections and listeners, but I shared details of simple application with
> only one JMS connection to avoid confusion.
>
> Please feel free to let me know for any information.
>
> Regards,
> Abhishek Kumar
>
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie, 

Thanks a lot for quick update. I looked on our codebase, logs and
stacktrace, and I can confirm that 

1. "javax.jms.ExceptionListener#onException()" is not getting called from
any of our defined classes/methods.
2. ExceptionListener is used only by connection object

We got this error in simple application which have - 
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
1 publisher – (session + producer)
qpid-jms-client - 56.0

So Application has only one JMS connection and one exception listener which
is only operated by JMS connection object.

I shared exact stacktrace with you without any customization and we can see
all call hierarchy from Qpid classes.

We are logging from “JmsConnectionListener” as well on
"onConnectionFailure()" method. Please find below stacktrace details, but
there is no difference between them
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_isConnectionActive_False.txt 
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_onConnectionFailure_True.txt 

With above details, I must request you to please look on this issue again. I
got this issue in other application as well, where we have multiple JMS
connections and listeners, but I shared details of simple application with
only one JMS connection to avoid confusion.

Please feel free to let me know for any information. 

Regards,
Abhishek Kumar





--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by Robbie Gemmell <ro...@gmail.com>.
Looking into your stack traces, I dont see how it is possible for the
situation to occur as described.

The final exception with stacktrace as shown, was only created after a
point at which the handling leading to its creation has rendered it
impossible for the getClientID() method to be called without throwing.
The failure cause was atomically set before the exception indicated
was created, and this would ensure that calling getClientID
subsequently throws. The stack of the exception itself reinforces this
as it spans the point where it occurs.

I think you must be seeing cross-connection object interaction
somehow. Ensure that your ExceptionListener can only operate on the
connection object it is being set on.

On Tue, 9 Mar 2021 at 14:04, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> I would like to provide you one more input. We got 'Unknown error from
> remote peer' error today morning due to
> JmsConnectionRemotelyClosedException.
> In first scenario connection was active and in second scenario connection
> was active. I compared both stacktrace and both are similar.
>
> Expected behaviour - As we got “JmsConnectionRemotelyClosedException”, then
> in both scenario connection should be inactive. Please correct me if I am
> wrong.
>
> Please find below exception details -
>
> First Exception -
> Connection exception from 'servicebusConnector_2', isConnectionActive
> 'false', Reason 'Unknown error from remote peer'
> Refer here for Stacktrace -
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_False.txt
>
> Second Exception -
> Connection exception from 'servicebusConnector_2', isConnectionActive
> 'true', Reason 'Unknown error from remote peer'
> Refer here for Stacktrace -
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_True.txt
>
> Please suggest your opinion.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving message without any connection failure and detached receiver error

Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,

I would like to provide you one more input. We got 'Unknown error from
remote peer' error today morning due to
JmsConnectionRemotelyClosedException. 
In first scenario connection was active and in second scenario connection
was active. I compared both stacktrace and both are similar.

Expected behaviour - As we got “JmsConnectionRemotelyClosedException”, then
in both scenario connection should be inactive. Please correct me if I am
wrong.

Please find below exception details - 

First Exception - 
Connection exception from 'servicebusConnector_2', isConnectionActive
'false', Reason 'Unknown error from remote peer'	
Refer here for Stacktrace -
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_False.txt 

Second Exception - 
Connection exception from 'servicebusConnector_2', isConnectionActive
'true', Reason 'Unknown error from remote peer'	
Refer here for Stacktrace -
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_True.txt 

Please suggest your opinion.

Regards,
Abhishek Kumar




--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org