You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by akabhishek1 <ma...@gmail.com> on 2021/02/17 08:52:21 UTC
qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Hi Team,
We recently upgraded qpid-jms-client to 56.0. We are using qpid-jms-client
ti receive message from ServiceBus. We need your urgent support on this
issue, because we are facing this issue in PROD environment.
Issue - Some consumers are not receiving message from queue without any
connection failure and detached receiver error
JMS Infrastructure -
JMS connection - 1
Exception listener - 1
7 destination (Queue/topic) - every destination has 4 consumers
28 consumers - every consumer have separate JMS session and MessageConsumerr
Scenario -
1. Received only one below error on "onException" block -
Connection exception from 'ServiceBus_Connector', isConnectionActive 'true',
Reason 'Unknown error from remote peer' - 'class javax.jms.JMSException',
cause 'org.apache.qpid.jms.provider.ProviderException: Unknown error from
remote peer'
2. As connection is active, so we didn't establish re-connection. We checked
connection status with below method
3. Checked receiver status with below isConsumerActive method and found that
only reciever is closed, so re-established receiver and this consumer is
consuming messages perfectly.
Impact - 1. One consumer receiving message perfectly
2. Six consumer does not throw any connection error, or detached
error
3. These six consumers stopped receiving message from queue
Could you please urgently suggest your opinion on this issue ?
public boolean isConnectionActive(Connection connection) {
boolean connectionStatus = false;
try {
String clientID = connection.getClientID();
connectionStatus = true;
} catch (Exception e) {
connectionStatus = false;
}
return connectionStatus;
}
public boolean isConsumerActive(MessageConsumer consumer) {
boolean isConsumerActive = false;
try {
consumer.getMessageListener();
isConsumerActive = true;
} catch (Exception e) {
isConsumerActive = false;
}
return isConsumerActive;
}
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
No, I dont know any ways of 'hacking' ServiceBus to do all things the
test does with its peer. We use a test peer specifically because such
things are often difficult or at least slow. You'd have to ask
Microsoft for questions you have about ServiceBus in any case.
See my mail from last time, there are various options for establishing
your own test specific server to validate particular behaviours:
https://lists.apache.org/thread.html/r6e016adafddf563445d9754cdf284f53f5d45a528adc7c07ee370134%40%3Cusers.qpid.apache.org%3E
Though, essentially copying the test as you did doesn't seem likely to
show anything the test didnt.
You can use protocol trace logs to determine and then reproduce
specific behaviours of interest yourself. From the stracktrace only
though, I could say the 'unknown error from peer' description earlier
in this thread was the result of the server remotely closing the
consumer without specifying the error field to indicate why. That all
seems unrelated to what I was testing, though that would similarly say
'unknown error from peer' if I had not included an error in the remote
session closure to validate it got reported.
On Fri, 12 Mar 2021 at 12:48, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for your quick reply and confirmation on this issue.
> I referred your test case "testRemotelyEndSessionWithMessageListener" and
> trying to replicate same test case with Azure ServiceBus, So I can replicate
> this issue and provide confirmation for fixes.
>
> Main challenge for me to perform action based on "TestAmqpPeer" class such
> as "remotelyEndLastOpenedSession()", "expectBegin()",
> "waitForAllHandlersToComplete()" operations.
> I think all these operations are performing under Proton library and don't
> have direct access to us.
>
> Please find sample test class -
> https://github.com/abhikt48/servicebus/blob/master/Qpid-57/TestSessionClosedException.java
> It is not working fine due to missing equivalent operation of "TestAmqpPeer"
> Class.
>
> Can you please suggest any hacking through which I can perform same
> functionality of "TestAmqpPeer" with ServiceBus and qpid SDK.
> It will be really helpful if you can provide us any clue to perform
> equivalent operation of "TestAmqpPeer", So I can execute test case and
> confidently share test result with you.
>
> Regarding reproducing of 'unknown error from peer' and complete logger - We
> don't have any standard way to reproduce this issue, infact we are dependent
> on unknown ServiceBus Broker activity and we don't have any control on
> this.
>
> Going forward, I will enable all logs including "JmsConnectionListener" and
> Traces logs, and deploy below two applications with 56 and 57-SNAPSHOT. I
> will keep these applications running for next 7-10days, and will update you
> if we got same issue again with all traces
> 1. Deploy application with 56.0 version
> 2. Deploy application with 57.0-SNAPSHOT version
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
Thanks a lot for your quick reply and confirmation on this issue.
I referred your test case "testRemotelyEndSessionWithMessageListener" and
trying to replicate same test case with Azure ServiceBus, So I can replicate
this issue and provide confirmation for fixes.
Main challenge for me to perform action based on "TestAmqpPeer" class such
as "remotelyEndLastOpenedSession()", "expectBegin()",
"waitForAllHandlersToComplete()" operations.
I think all these operations are performing under Proton library and don't
have direct access to us.
Please find sample test class -
https://github.com/abhikt48/servicebus/blob/master/Qpid-57/TestSessionClosedException.java
It is not working fine due to missing equivalent operation of "TestAmqpPeer"
Class.
Can you please suggest any hacking through which I can perform same
functionality of "TestAmqpPeer" with ServiceBus and qpid SDK.
It will be really helpful if you can provide us any clue to perform
equivalent operation of "TestAmqpPeer", So I can execute test case and
confidently share test result with you.
Regarding reproducing of 'unknown error from peer' and complete logger - We
don't have any standard way to reproduce this issue, infact we are dependent
on unknown ServiceBus Broker activity and we don't have any control on
this.
Going forward, I will enable all logs including "JmsConnectionListener" and
Traces logs, and deploy below two applications with 56 and 57-SNAPSHOT. I
will keep these applications running for next 7-10days, and will update you
if we got same issue again with all traces
1. Deploy application with 56.0 version
2. Deploy application with 57.0-SNAPSHOT version
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
On Mon, 8 Mar 2021 at 14:09, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We got this issue again in multiple instances. We got exception stack trace
> this time on "onException" block on one of instances.
>
> Overall Issue - Consumer is not consuming message from endpoint without
> throwing any exception
>
> ##JMS Infrastructure##
> qpid-jms-client - 56.0
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
> (a)topic:A/subscriptions/a1
> (b)topic:B/subscriptions/b1
> 1 publisher -
>
> Note - every consumer/publisher have separate JMS session, MessageConsumer
> and
> producer
>
>
> Impact/Action/Observations -
>
> 1. Depolyed Application on - 3/5/2021, 1:14:00.550 PM -- Application was
> idle not receiving single messages at all
> 2. Got only one "Unknown error from remote peer" error on "onException"
> block, Please refer error stacktrace for more details -
> https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/Exception_Stack_Trace.txt
> 3. Checked connection status with "isConnectionActive()" method - Connection
> was active, so didn't reconnected.
> 4. Checked "isConsumerActive" for first receiver(A/a1) - It was broken, so
> reconnected on this receiver
> 5. Checked "isConsumerActive" for first receiver(B/b1) - it was active, so
> didn't reconnected
> 6. Application was idle over weekend and did tested today. I published
> message on both receiver and found that
> (a) receiver(A/a1) - consuming msg perfectly, this was broken and
> reconnected after "Unknown error from remote peer"
> (b) receiver(B/b1) - not consuming messages, this was active at time of
> "Unknown error from remote peer"
>
> Please refer first post for implementation of "isConsumerActive" and
> "isConnectionActive()"
>
> Application is very light and simple, which i created for testing of
> ServiceBus only. Please refer below thread dump if this can help -
> https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/threadDump.tdump
>
>
> Note - We have more than 100 instances running on Qpid-45, we haven't
> received this type of issue in all 100+ instances. But we are facing
> continuously this issue on some applications which are using Qpid-56.
>
> Main Issues - Whenever we got "Unknown error from remote peer" then only one
> consumer got broken and other consumer status shows active for this error.
> We should receive -
> 1. Multiple connection exception for every receiver
> 2. Or all consumer should be inactive for "Unknown error from remote peer"
>
> Could you please take a look on this issue and let us know for any
> information.
>
> Regards,
> Abhishek Kumar
>
>
The 'unknown error from peer' is identifiable from the stack as a
single remote consumer closure, consistent with the resulting
behaviour described. Other than that, we dont know what else actually
occurred from the information. I suspect its possible the consumer
sessions were each closed though.
Since you have a reproducer, reproducing this with the protocol
logging enabled as well would have given a much better idea, and
another run with the all the regular client loggers at trace as well
as the protocol logging might also have been useful.
You noted having 8 consumer sessions but the thread dump only shows
thread related evidence of 4, with 4 of the original sessions on the
connection (i.e 1-4), presumably for the 'first listener', and another
session (the 786th) evident being used for an async completion
producer. That is consistent with 4 sessions being closed, or not
existing, for the 'second listener'. This all appears to be on the
second connection created.
Having considered this and having another dig at the client, I found
an issue that could be related. Though I could have sworn it did so,
the client did not fire the ExceptionListener if a Session is
remotely-closed while having a consumer with a MessageListener, it
only did so if the consumer itself was remotely-closed. That could be
a potential explanation for the behaviour you see, I have just pushed
a fix for via https://issues.apache.org/jira/browse/QPIDJMS-526 that
you can try out and see if it helps. Although, I would note that
0.45.0 also had the same behaviour, so it shouldn't really make a
difference.
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
We got this issue again in multiple instances. We got exception stack trace
this time on "onException" block on one of instances.
Overall Issue - Consumer is not consuming message from endpoint without
throwing any exception
##JMS Infrastructure##
qpid-jms-client - 56.0
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
(a)topic:A/subscriptions/a1
(b)topic:B/subscriptions/b1
1 publisher -
Note - every consumer/publisher have separate JMS session, MessageConsumer
and
producer
Impact/Action/Observations -
1. Depolyed Application on - 3/5/2021, 1:14:00.550 PM -- Application was
idle not receiving single messages at all
2. Got only one "Unknown error from remote peer" error on "onException"
block, Please refer error stacktrace for more details -
https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/Exception_Stack_Trace.txt
3. Checked connection status with "isConnectionActive()" method - Connection
was active, so didn't reconnected.
4. Checked "isConsumerActive" for first receiver(A/a1) - It was broken, so
reconnected on this receiver
5. Checked "isConsumerActive" for first receiver(B/b1) - it was active, so
didn't reconnected
6. Application was idle over weekend and did tested today. I published
message on both receiver and found that
(a) receiver(A/a1) - consuming msg perfectly, this was broken and
reconnected after "Unknown error from remote peer"
(b) receiver(B/b1) - not consuming messages, this was active at time of
"Unknown error from remote peer"
Please refer first post for implementation of "isConsumerActive" and
"isConnectionActive()"
Application is very light and simple, which i created for testing of
ServiceBus only. Please refer below thread dump if this can help -
https://github.com/abhikt48/servicebus/blob/master/5_march_error_for_qpid/threadDump.tdump
Note - We have more than 100 instances running on Qpid-45, we haven't
received this type of issue in all 100+ instances. But we are facing
continuously this issue on some applications which are using Qpid-56.
Main Issues - Whenever we got "Unknown error from remote peer" then only one
consumer got broken and other consumer status shows active for this error.
We should receive -
1. Multiple connection exception for every receiver
2. Or all consumer should be inactive for "Unknown error from remote peer"
Could you please take a look on this issue and let us know for any
information.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
On Thu, 18 Mar 2021 at 12:58, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> One more details on my last post. We have added JmsConnectionListener in
> Qpid-56 application. We got three log entries from JmsConnectionListener.
> Please find below details.
>
> 1. 3/18/2021, 3:48:18.176 AM - "onConsumerClosed" method - Consumer closed
> of '**-test-**/subscriptions/56-sub1'
> ExceptionStackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/onConsumerClosed_03_48_18_AM.txt
> 2. 3/18/2021, 3:48:18.245 AM - "onConnectionEstablished" method - Connection
> established URI
> 'amqps://****.servicebus.windows.net?transport.tcpKeepAlive=true&amqp.traceFrames=true'
> 3. 3/18/2021, 3:48:18.381 AM - "onConnectionFailure" method - Connection
> failure
> ExceptionStackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/onConnectionFailure_03_48_18_AM.txt
>
>
> Surprisingly -
> 1. 3:48:18 AM - when my method checks for "isConnectionActive" then
> connection was active(true) which should be false
> 2. 3:48:18 AM - when my method "isConsumerActive" checks for
> '**-test-**/subs..s/56-sub1' then consumer was active
> 3. Looks like connection got established automatically without reconnect
> attempt and we are not using failover protocol as well.
>
> Questions -
> 1) As we are not using failover protocol, so connection should not be
> recover automatically. Can you please suggest why connection got reconnect
> automatically?
The obvious suggestion is that it didnt happen that way. I've outlined
my thinking of some potential actual behaviour in my earlier reply to
the previous message.
The client literally doesn't load its reconnection logic if you aren't
using failover. It can then only establish a TCP connection during a
factory.createConnection(..) call. The logging shows evidence that
such a call was actually made, as the connection details changed in a
way that happens during those calls.
> 2) Looks like consumer also recover automatically without reconnect attempt.
> We don't have any “consumerEstablished” notification in
> JmsConnectionListener, but based on testing I can confirm that consumer
> still receiving messages.
> Can you please suggest why consumer got reconnected automatically without
> trying to reconnect consumer?
Similarly, the obvious answer is that it didnt happen that way. The
only place I'm aware the client can reestablish consumers by itself is
in the failover provider logic, which as above simply wasnt loaded if
you aren't using failover. Plus it was a new connection so the client
would have no knowledge of consumers anyway.
>
> Please take a look on this issue and let me know for any information.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
One more details on my last post. We have added JmsConnectionListener in
Qpid-56 application. We got three log entries from JmsConnectionListener.
Please find below details.
1. 3/18/2021, 3:48:18.176 AM - "onConsumerClosed" method - Consumer closed
of '**-test-**/subscriptions/56-sub1'
ExceptionStackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/onConsumerClosed_03_48_18_AM.txt
2. 3/18/2021, 3:48:18.245 AM - "onConnectionEstablished" method - Connection
established URI
'amqps://****.servicebus.windows.net?transport.tcpKeepAlive=true&amqp.traceFrames=true'
3. 3/18/2021, 3:48:18.381 AM - "onConnectionFailure" method - Connection
failure
ExceptionStackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/onConnectionFailure_03_48_18_AM.txt
Surprisingly -
1. 3:48:18 AM - when my method checks for "isConnectionActive" then
connection was active(true) which should be false
2. 3:48:18 AM - when my method "isConsumerActive" checks for
'**-test-**/subs..s/56-sub1' then consumer was active
3. Looks like connection got established automatically without reconnect
attempt and we are not using failover protocol as well.
Questions -
1) As we are not using failover protocol, so connection should not be
recover automatically. Can you please suggest why connection got reconnect
automatically?
2) Looks like consumer also recover automatically without reconnect attempt.
We don't have any “consumerEstablished” notification in
JmsConnectionListener, but based on testing I can confirm that consumer
still receiving messages.
Can you please suggest why consumer got reconnected automatically without
trying to reconnect consumer?
Please take a look on this issue and let me know for any information.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
The logs after 3:45:34.157 show that your application creates a new
session then creates a new sender, then closes a previous sender and
then its session, then sits idle a bit until after a few minutes
Service Bus remotely closes all the links and sessions and then the
connection itself, without any errors specified which will provoke
various 'unknown error from remote peer' exceptions, with those for
consumers with listeners and the connection closure being delivered to
the exception listener. A second connection is then established, and 8
consumers established on it, then eventually a sender.
I would theorise what is happening is that because there are likely to
be multiple exceptions raised here, first for the message listener
remote closures and then one for the connection remote closure, and
the exception listener is necessarily fired async from their actual
arrival as they are processed, and you are doing your own 'detect
closure' in the ExceptionListener, you are quite possibly noticing the
connection has become failed during an ExceptionListener run prompted
by an earlier (consumer related?) exception delivery, either in your
detection check or because attempting to create a consumer/session
throws if the connection is now failed, before you then receive the
exception that explicitly resulted from the connections actual remote
closure. You might then be repeating your checks during subsequent
exception delivery for either a consumer or the connection closure and
indicating the connection isnt closed (which again per previous mails,
it must be for the connection closure exception as the exception
delivered was itself only created after that point) quite possibly
because what you checked isnt the same connection, if your exception
handling already replaced it and the listener can access and operate
on that connection object (this would go back to my earlier notes
around whether your exception listener can only interact with the
connection object it was set on).
That would also fit in with your followup mail. The 'connection
listener' events are also fired async, in the same executor thread as
the ExceptionListener, so you could well see one ExceptionListener
handling after e.g a consumer closure do work that creates a new
connection (as thats also become failed) and cause things leading to
its logging, before the 'connection listener' report is actually made
aware of the prior connections failure. Note there is no 'connection
listener' event for opening consumers/producers/sessions and thus no
related logging.
On Thu, 18 Mar 2021 at 11:06, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We got 'Unknown error from remote peer' today in both applications QPid-56
> and Qpid-57, surprisingly both applications are working fine and consuming
> messages as expected.
>
> However I would like to share interesting details of "QPid-56" application.
> Note - In this application, I am not considering connection is closed for
> JmsConnectionRemotelyClosedException/JmsConnectionFailedException. I am
> relying on “isConnectionActive” method to check connection status– details
> are in the first post of this issue.
>
> ## JMS Infra ##
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
> (a)topic:A/subscriptions/a1
> (b)topic:B/subscriptions/b1
> 1 publisher – (session + producer)
> qpid-jms-client - 56.0
>
> Steps -
> 1)
> 3/18/2021, 3:48:18.380 AM - Received error on "onException" - Connection
> exception from 'servicebus_2', isConnectionActive 'true', Reason 'Unknown
> error from remote peer' - 'class
> org.apache.qpid.jms.JmsConnectionRemotelyClosedException'
> StackTrace -
> https://github.com/abhikt48/servicebus/blob/master/18-March/ConnectionActive_true_JmsConnectionRemotelyClosedException.txt
> 2)Checked "isConsumerActive" for first receiver(A/a1) - It was active, so no
> need to reconnect
> 3)Checked "isConsumerActive" for first receiver(B/b1) - it was active, so no
> need to reconnect
>
> 4) Published message at 3/18/2021, 10 AM - Both receiver consumed message
> successfully
>
> We have enable proton TRACES logs. I attach TRACES of last 12 hour, so we
> can't miss anything, you can check after "3/18/2021, 3:45:34.157 AM".
> Please refer -
> https://github.com/abhikt48/servicebus/blob/master/18-March/Proton_Traces.txt
>
> As we have received "JmsConnectionRemotelyClosedException", so we are
> assuming that connection should be broken, but connection and receiver
> wasn't broken. Both receivers are consuming message perfectly.
>
> I am not sure this is expected behaviour or not. Could you please take a
> look on this and suggest your suggestion.
>
> P.S. - I haven't shared TRACES and log details on Qpid-57 application for
> comparison, as expected it is working fine. But if you need TRACES and log
> details of Qpid-57 application as well, then please let me know. I can
> extract and share with you.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
We got 'Unknown error from remote peer' today in both applications QPid-56
and Qpid-57, surprisingly both applications are working fine and consuming
messages as expected.
However I would like to share interesting details of "QPid-56" application.
Note - In this application, I am not considering connection is closed for
JmsConnectionRemotelyClosedException/JmsConnectionFailedException. I am
relying on “isConnectionActive” method to check connection status– details
are in the first post of this issue.
## JMS Infra ##
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
(a)topic:A/subscriptions/a1
(b)topic:B/subscriptions/b1
1 publisher – (session + producer)
qpid-jms-client - 56.0
Steps -
1)
3/18/2021, 3:48:18.380 AM - Received error on "onException" - Connection
exception from 'servicebus_2', isConnectionActive 'true', Reason 'Unknown
error from remote peer' - 'class
org.apache.qpid.jms.JmsConnectionRemotelyClosedException'
StackTrace -
https://github.com/abhikt48/servicebus/blob/master/18-March/ConnectionActive_true_JmsConnectionRemotelyClosedException.txt
2)Checked "isConsumerActive" for first receiver(A/a1) - It was active, so no
need to reconnect
3)Checked "isConsumerActive" for first receiver(B/b1) - it was active, so no
need to reconnect
4) Published message at 3/18/2021, 10 AM - Both receiver consumed message
successfully
We have enable proton TRACES logs. I attach TRACES of last 12 hour, so we
can't miss anything, you can check after "3/18/2021, 3:45:34.157 AM".
Please refer -
https://github.com/abhikt48/servicebus/blob/master/18-March/Proton_Traces.txt
As we have received "JmsConnectionRemotelyClosedException", so we are
assuming that connection should be broken, but connection and receiver
wasn't broken. Both receivers are consuming message perfectly.
I am not sure this is expected behaviour or not. Could you please take a
look on this and suggest your suggestion.
P.S. - I haven't shared TRACES and log details on Qpid-57 application for
comparison, as expected it is working fine. But if you need TRACES and log
details of Qpid-57 application as well, then please let me know. I can
extract and share with you.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
Yes, a JmsConnectionRemotelyClosedException, or something with a
ProviderConnectionRemotelyClosedException based cause, is indicates
the connection is done for, that the client received a connection
close frame for some reason and that it wasnt expected (i.e it wasnt
expecting it as response to a requested closure). Similarly a
JmsConnectionFailedException, or something with
ProviderFailedException based cause, can indicate other
connection-breaking issues such as the connection dropping or some
failure in its processing where a close frame wasnt necessarily
received.
On Fri, 12 Mar 2021 at 16:57, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for quick reply.
>
> Can i assume that JMS Connection will be closed for
> "JmsConnectionRemotelyClosedException" or
> "ProviderConnectionRemotelyClosedException" exception.
>
> So whenever I got "JmsConnectionRemotelyClosedException" or
> "ProviderConnectionRemotelyClosedException" on "onException()" method then
> I will recreate JMS connection.
>
> Please let me know if there are any other JMS connection closed Exception
> classes, so i will handle them as well.
>
> Could you please confirm on this or correct me if I am wrong?
>
>
> Regards,
> Abhishek Kumar
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
Thanks a lot for quick reply.
Can i assume that JMS Connection will be closed for
"JmsConnectionRemotelyClosedException" or
"ProviderConnectionRemotelyClosedException" exception.
So whenever I got "JmsConnectionRemotelyClosedException" or
"ProviderConnectionRemotelyClosedException" on "onException()" method then
I will recreate JMS connection.
Please let me know if there are any other JMS connection closed Exception
classes, so i will handle them as well.
Could you please confirm on this or correct me if I am wrong?
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
I have looked at the reported issue multiple times for a significant
period. I tried reproducing it and was unable to. I got a second
opinion, they independently came to the same conclusion as I had.
I've thus described why I dont believe the behaviour is occurring as
you specified (you may need to report a JVM defect if it were), and so
it would seem something else must be happening. I dont know what that
is, but suggested a possibility. If you investigate and can point to
something specific in the client I can look again, but for now I've
exhausted the available info and not found anything suggesting a
client issue, only the reverse.
The stacktrace did only show Qpid classes, but that is no surprise and
doesn't suggest anything about where an issue might be. The exception
reported was created inside the client, by the client, upon network
activity, in order to report the resulting situation. It isnt going to
reference things outside the client in that case.
On Fri, 12 Mar 2021 at 14:04, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for quick update. I looked on our codebase, logs and
> stacktrace, and I can confirm that
>
> 1. "javax.jms.ExceptionListener#onException()" is not getting called from
> any of our defined classes/methods.
> 2. ExceptionListener is used only by connection object
>
> We got this error in simple application which have -
> JMS connection - 1
> Exception listener - 1
> 2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
> sessions
> 1 publisher – (session + producer)
> qpid-jms-client - 56.0
>
> So Application has only one JMS connection and one exception listener which
> is only operated by JMS connection object.
>
> I shared exact stacktrace with you without any customization and we can see
> all call hierarchy from Qpid classes.
>
> We are logging from “JmsConnectionListener” as well on
> "onConnectionFailure()" method. Please find below stacktrace details, but
> there is no difference between them
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_isConnectionActive_False.txt
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_onConnectionFailure_True.txt
>
> With above details, I must request you to please look on this issue again. I
> got this issue in other application as well, where we have multiple JMS
> connections and listeners, but I shared details of simple application with
> only one JMS connection to avoid confusion.
>
> Please feel free to let me know for any information.
>
> Regards,
> Abhishek Kumar
>
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
Thanks a lot for quick update. I looked on our codebase, logs and
stacktrace, and I can confirm that
1. "javax.jms.ExceptionListener#onException()" is not getting called from
any of our defined classes/methods.
2. ExceptionListener is used only by connection object
We got this error in simple application which have -
JMS connection - 1
Exception listener - 1
2 listener - (8(2*4) consumers) - Every listener have 4 consumers and
sessions
1 publisher – (session + producer)
qpid-jms-client - 56.0
So Application has only one JMS connection and one exception listener which
is only operated by JMS connection object.
I shared exact stacktrace with you without any customization and we can see
all call hierarchy from Qpid classes.
We are logging from “JmsConnectionListener” as well on
"onConnectionFailure()" method. Please find below stacktrace details, but
there is no difference between them
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_isConnectionActive_False.txt
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionListener_onConnectionFailure_True.txt
With above details, I must request you to please look on this issue again. I
got this issue in other application as well, where we have multiple JMS
connections and listeners, but I shared details of simple application with
only one JMS connection to avoid confusion.
Please feel free to let me know for any information.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
Looking into your stack traces, I dont see how it is possible for the
situation to occur as described.
The final exception with stacktrace as shown, was only created after a
point at which the handling leading to its creation has rendered it
impossible for the getClientID() method to be called without throwing.
The failure cause was atomically set before the exception indicated
was created, and this would ensure that calling getClientID
subsequently throws. The stack of the exception itself reinforces this
as it spans the point where it occurs.
I think you must be seeing cross-connection object interaction
somehow. Ensure that your ExceptionListener can only operate on the
connection object it is being set on.
On Tue, 9 Mar 2021 at 14:04, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> I would like to provide you one more input. We got 'Unknown error from
> remote peer' error today morning due to
> JmsConnectionRemotelyClosedException.
> In first scenario connection was active and in second scenario connection
> was active. I compared both stacktrace and both are similar.
>
> Expected behaviour - As we got “JmsConnectionRemotelyClosedException”, then
> in both scenario connection should be inactive. Please correct me if I am
> wrong.
>
> Please find below exception details -
>
> First Exception -
> Connection exception from 'servicebusConnector_2', isConnectionActive
> 'false', Reason 'Unknown error from remote peer'
> Refer here for Stacktrace -
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_False.txt
>
> Second Exception -
> Connection exception from 'servicebusConnector_2', isConnectionActive
> 'true', Reason 'Unknown error from remote peer'
> Refer here for Stacktrace -
> https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_True.txt
>
> Please suggest your opinion.
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
I would like to provide you one more input. We got 'Unknown error from
remote peer' error today morning due to
JmsConnectionRemotelyClosedException.
In first scenario connection was active and in second scenario connection
was active. I compared both stacktrace and both are similar.
Expected behaviour - As we got “JmsConnectionRemotelyClosedException”, then
in both scenario connection should be inactive. Please correct me if I am
wrong.
Please find below exception details -
First Exception -
Connection exception from 'servicebusConnector_2', isConnectionActive
'false', Reason 'Unknown error from remote peer'
Refer here for Stacktrace -
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_False.txt
Second Exception -
Connection exception from 'servicebusConnector_2', isConnectionActive
'true', Reason 'Unknown error from remote peer'
Refer here for Stacktrace -
https://github.com/abhikt48/servicebus/blob/master/9_march_2021/ConnectionRemoteClose_isConnectionActive_True.txt
Please suggest your opinion.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
On Fri, 26 Feb 2021 at 11:46, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> We are trying to find out root cause for this issue. Can you please provide
> details on below query which can help us to investigate for future issue
>
> 1) What is the idle way to check link status between JMS consumer and broker
> at runtime?
> - We can get to know via TRACE level logs, but we can't turn on in PROD
> server.
> - Is there anything can we do at runtime to check link status between
> consumer and broker?
>
This isnt simple since the direct link state is well inside the inner
portion of the client, which you dont have direct access to from the
JMS objects in the outer portion. In both the JMS and AMQP sense its
also complicated by parent child relationships, i.e parents
session/connections closing implicitly shuts down their child
resources, even if they aren't explicitly individually closed protocol
wise. Essentially, if the consumer itself thinks it is closed, it will
throw on various methods as you already know.
> 2) To check consumer status - I disabled endpoint and waited for 6-7m, but
> we didn't receive any error for 6-7minutes on "onException" method neither
> any System.Error, so we concluded that consumer was already detached from
> long time, that's why it was not receiving message from more than 12hrs. Is
> my understanding correct? Or I should do something else to verify?
>
So again, there could be many reasons. I cant reason much from the
information available or say much that I didnt in prior mails.
> 3) We have added JmsConnectionListener and added logger on all operations
> except "onInboundMessage" operation, because "onInboundMessage" operation
> are getting called for receive message.
> I have not enabled logger on session class, because JmsConnectionListener
> will give us enough information.
> Do you think, I need to enabled any other logger which will give us detached
> link information or this should be enough? (I think, this should be enough,
> just cross verifying for safer side)
>
> Regards,
> Abhishek Kumar
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
We are trying to find out root cause for this issue. Can you please provide
details on below query which can help us to investigate for future issue
1) What is the idle way to check link status between JMS consumer and broker
at runtime?
- We can get to know via TRACE level logs, but we can't turn on in PROD
server.
- Is there anything can we do at runtime to check link status between
consumer and broker?
2) To check consumer status - I disabled endpoint and waited for 6-7m, but
we didn't receive any error for 6-7minutes on "onException" method neither
any System.Error, so we concluded that consumer was already detached from
long time, that's why it was not receiving message from more than 12hrs. Is
my understanding correct? Or I should do something else to verify?
3) We have added JmsConnectionListener and added logger on all operations
except "onInboundMessage" operation, because "onInboundMessage" operation
are getting called for receive message.
I have not enabled logger on session class, because JmsConnectionListener
will give us enough information.
Do you think, I need to enabled any other logger which will give us detached
link information or this should be enough? (I think, this should be enough,
just cross verifying for safer side)
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
Thanks a lot for quick suggestion. We have implemented JmsConnectionListener
and planned to roll out in lower environmnet. This will give us more
granular details for any connection failure, session/producer closer.
We will let you know if we found any root cause.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
On Thu, 18 Feb 2021 at 17:00, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Robbie,
>
> Thanks a lot for quick reply. We received only one below exception. There
> are 7 destinations connected with single serviceBus connection. If links are
> detached on “unknown peer error”, then we should receive at least 7
> exceptions at "onException" block, but received only one.
>
> I checked the log and find out that Application has not requested to close
> JMS connection and does not initiate to close session/consumer of 6
> destinations, but initiate to close/create session/consumer of only one
> destination.
>
> As Application received one exception, and found issue in only one
> destination, So application re-established communication with only one
> receiver (sub1). I am hoping closer of one destination (session/consumer)
> shouldn't impact other session/consumer of 6 destinations.
>
> Big trouble points for us that we haven't received any exception for 6
> destinations and links were detached as well (tested with disabling the
> queue which didn’t bubbled up error on onException block).
>
> Just FYI - All Seven destinations(sub1, sub2, sub3..) are subscriptions of
> single topic ('TestTopic')
>
> Unfortunately I don't have any stacktrace, so i can't provide much details.
> Please take a look below thread dumps, if it helps - First_App_ThreadDump
> <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/Qpid-ServiceBus-threadDump.tdump>
>
> Going forward, we will log stacktrace in onException block.
Its hard to tell much from singular thread dumps, though there are
some interesting things:
The above file shows evidence of [at least remnants of] 18
connections, and lots and lots of sessions (say 150+) across them.
This doesnt seem to match expectations from your overview, suggesting
perhaps at least there is an issue with the cleanup and
reestablishment bits.
>
> -------------------Details of another Application which have same issue like
> above---------------
>
> We are using latest jar 3 applications and we received this type of error in
> two applications. I looked error on other application and that application
> is very light, not heavier like above one. Please find below findings
>
> ##JMS Infrastructure##
> JMS connection - 1
> Exception listener - 1
> 3 listener - (12 (3*4) consumers)
> 7 publisher -
> every consumer/publisher have separate JMS session, MessageConsumer and
> producer
>
> Listener Details -
> 1. QueueA
> 2. QueueA/$DeadLetterQueue
> 3. QueueB
>
> ####Actions after exception ###
> 1. Exception on "onException" block -
> Connection exception from 'SBus_Connector_1', isConnectionActive 'true',
> Reason 'Unknown error from remote peer' - 'class javax.jms.JMSException',
> cause 'org.apache.qpid.jms.provider.ProviderException: Unknown error from
> remote peer'
>
> 2. Checked isConsumerActive for 'QueueA' - result “true” - (We have problem
> here, not consuming messages)
> 3. Checked isConsumerActive for 'QueueA/$DeadLetterQueue' - result “false”,
> Re-established consumer
> - I have doubt here, As 'QueueA/$DeadLetterQueue' consumer is deactive, so
> 'QueueA' consumer should be deactive as well. Please suggest your opinion.
If you have 2 seperate consumers, if one gets closed it doesnt
necessarily mean they are both. There is also nothing that means even
if they both are, that it happens in a particular order at a
particular time.
> 4. Checked isConsumerActive for 'QueueB' - result “true” - (Did nothing)
>
> FYI - We re-establish consumer only if consumer is deactive.
>
> Same Application have another JMS connection(Separate ServiceBus broker
> endpoint), on which we got same exception,
> Error - Connection exception from 'SBus_Connector_2', isConnectionActive
> 'true', Reason 'Unknown error from remote peer' - 'class
> javax.jms.JMSException', cause
> 'org.apache.qpid.jms.provider.ProviderException: Unknown error from remote
> peer'
>
> This JMS connection has one publisher which is trying to publish message,
> while publishing it also throws "ProviderException - Unknown error from
> remote peer".
>
> Please find exception StackTrace - StackTrace_Unknown_Remote_Error.json
> <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/StackTrace_Unknown_Remote_Error.json>
>
This is almost unreadable, and seems to omit line numbers for the
stack, removing a good amount of useful info.
>
> ThreadDump - Application_Thread_Dump
> <https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/qpid-AppB-threadDump.tdump>
>
> ##Overall Obesrvation ##
> 1. Exception listener does not receiver multiple notification for receiver
> detachment. At least one notification should trigger for one destination.
If there is actually a reason to, i.e they actually were closed, and
e.g. if none of the reasons they wouldnt be occured already, e.g
parent session/connection bits have already been closed for example.
Cant really tell what happened from the limited detail available.
> 2. JmsMessageConsumer.getMessageListener()->checkClosed() --> does not
> provide correct closed status. It could be sync up issue.
>
> Suggestion/Question - We can't enable TRACE in PROD environment because it
> emits lot of logs. Can you please tell us any qpid class which logs TRACE
> event only on failure like "Unknown error from remote peer". So we can log
> enable Qpid log atleast for failure.
>
The session has some info logging of e.g producer/consumer closures. I
dont think there is one place to get minimal logging of those
exceptions, the exceptions themselves are typically that indication.
There is an 'connection listener' (using the JmsConnectionListener
interface) that we use within tests at times which is called at the
end of the overall process, though note asynchronously from the
exception listener being called, you could perhaps use it to add your
own additional extra logging.
>
> Please suggest your opinion on above query, So we can proceed further. I
> will keep on working parallelly to replicate this issue if possible and
> update you if found any.
>
> Regards,
> Abhishek Kumar
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Robbie,
Thanks a lot for quick reply. We received only one below exception. There
are 7 destinations connected with single serviceBus connection. If links are
detached on “unknown peer error”, then we should receive at least 7
exceptions at "onException" block, but received only one.
I checked the log and find out that Application has not requested to close
JMS connection and does not initiate to close session/consumer of 6
destinations, but initiate to close/create session/consumer of only one
destination.
As Application received one exception, and found issue in only one
destination, So application re-established communication with only one
receiver (sub1). I am hoping closer of one destination (session/consumer)
shouldn't impact other session/consumer of 6 destinations.
Big trouble points for us that we haven't received any exception for 6
destinations and links were detached as well (tested with disabling the
queue which didn’t bubbled up error on onException block).
Just FYI - All Seven destinations(sub1, sub2, sub3..) are subscriptions of
single topic ('TestTopic')
Unfortunately I don't have any stacktrace, so i can't provide much details.
Please take a look below thread dumps, if it helps - First_App_ThreadDump
<https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/Qpid-ServiceBus-threadDump.tdump>
Going forward, we will log stacktrace in onException block.
-------------------Details of another Application which have same issue like
above---------------
We are using latest jar 3 applications and we received this type of error in
two applications. I looked error on other application and that application
is very light, not heavier like above one. Please find below findings
##JMS Infrastructure##
JMS connection - 1
Exception listener - 1
3 listener - (12 (3*4) consumers)
7 publisher -
every consumer/publisher have separate JMS session, MessageConsumer and
producer
Listener Details -
1. QueueA
2. QueueA/$DeadLetterQueue
3. QueueB
####Actions after exception ###
1. Exception on "onException" block -
Connection exception from 'SBus_Connector_1', isConnectionActive 'true',
Reason 'Unknown error from remote peer' - 'class javax.jms.JMSException',
cause 'org.apache.qpid.jms.provider.ProviderException: Unknown error from
remote peer'
2. Checked isConsumerActive for 'QueueA' - result “true” - (We have problem
here, not consuming messages)
3. Checked isConsumerActive for 'QueueA/$DeadLetterQueue' - result “false”,
Re-established consumer
- I have doubt here, As 'QueueA/$DeadLetterQueue' consumer is deactive, so
'QueueA' consumer should be deactive as well. Please suggest your opinion.
4. Checked isConsumerActive for 'QueueB' - result “true” - (Did nothing)
FYI - We re-establish consumer only if consumer is deactive.
Same Application have another JMS connection(Separate ServiceBus broker
endpoint), on which we got same exception,
Error - Connection exception from 'SBus_Connector_2', isConnectionActive
'true', Reason 'Unknown error from remote peer' - 'class
javax.jms.JMSException', cause
'org.apache.qpid.jms.provider.ProviderException: Unknown error from remote
peer'
This JMS connection has one publisher which is trying to publish message,
while publishing it also throws "ProviderException - Unknown error from
remote peer".
Please find exception StackTrace - StackTrace_Unknown_Remote_Error.json
<https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/StackTrace_Unknown_Remote_Error.json>
ThreadDump - Application_Thread_Dump
<https://github.com/abhikt48/servicebus/blob/master/For%20Qpid/qpid-AppB-threadDump.tdump>
##Overall Obesrvation ##
1. Exception listener does not receiver multiple notification for receiver
detachment. At least one notification should trigger for one destination.
2. JmsMessageConsumer.getMessageListener()->checkClosed() --> does not
provide correct closed status. It could be sync up issue.
Suggestion/Question - We can't enable TRACE in PROD environment because it
emits lot of logs. Can you please tell us any qpid class which logs TRACE
event only on failure like "Unknown error from remote peer". So we can log
enable Qpid log atleast for failure.
Please suggest your opinion on above query, So we can proceed further. I
will keep on working parallelly to replicate this issue if possible and
update you if found any.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by Robbie Gemmell <ro...@gmail.com>.
If you have exceptions then please provide the stacktraces, it may
contain context helpful in itself which the messages doesnt
(particularly when the exception message appears to be, the peer did
not specify an error message).
There haven't been many significant changes since 0.45.0 I would
expect could alter behaviour in this space. Most or all of those were
made to address bugs or change behaviours identified from your reports
both with 0.45.0 and subsequent releases.
Its difficult to reason much at all from the information available, however...
There are a variety of ways I might not expect to receive an exception
in the listener upon detaching, such as if the error was already
reported (something did, unclear what from the minimal
detail...posting the rest more may or may not make it clearer), or the
resource was previously / is currently being attempted to be locally
closed, or the parent session / connection was closed (which naturally
implicitly closes their children, but doesnt provoke a storm of
related exceptions), and i'm sure there are others.
You may also have blocked the thread again somehow, as has happened
before in your use case, perhaps stopping further issues being
communicated.
You also previously reported unexpected 'disabling subscription'
behaviour as a client bug, whereby 0.56.0 reported the very exceptions
you are now indicating it didnt, when really it was the server that
just didnt do what you expected, perhaps that is true here as well?
We do have a test that shows the client fires the exception listener
when a consumer with listener is remotely closed, though it's
certainly possible your somewhat more elaborate use case is getting
into a corner case, but I dont immediately see one, and again not much
has changed in those areas.
It's of course worth adding the obvious, that while a consumer not
receiving messages could indeed indicate an issue with it, it doesn't
necessarily. It may just not have been given any.
Robbie
On Wed, 17 Feb 2021 at 08:52, akabhishek1
<ma...@gmail.com> wrote:
>
> Hi Team,
>
> We recently upgraded qpid-jms-client to 56.0. We are using qpid-jms-client
> ti receive message from ServiceBus. We need your urgent support on this
> issue, because we are facing this issue in PROD environment.
>
> Issue - Some consumers are not receiving message from queue without any
> connection failure and detached receiver error
>
> JMS Infrastructure -
> JMS connection - 1
> Exception listener - 1
> 7 destination (Queue/topic) - every destination has 4 consumers
> 28 consumers - every consumer have separate JMS session and MessageConsumerr
>
> Scenario -
> 1. Received only one below error on "onException" block -
>
> Connection exception from 'ServiceBus_Connector', isConnectionActive 'true',
> Reason 'Unknown error from remote peer' - 'class javax.jms.JMSException',
> cause 'org.apache.qpid.jms.provider.ProviderException: Unknown error from
> remote peer'
>
> 2. As connection is active, so we didn't establish re-connection. We checked
> connection status with below method
> 3. Checked receiver status with below isConsumerActive method and found that
> only reciever is closed, so re-established receiver and this consumer is
> consuming messages perfectly.
>
> Impact - 1. One consumer receiving message perfectly
> 2. Six consumer does not throw any connection error, or detached
> error
> 3. These six consumers stopped receiving message from queue
>
> Could you please urgently suggest your opinion on this issue ?
>
>
> public boolean isConnectionActive(Connection connection) {
> boolean connectionStatus = false;
> try {
> String clientID = connection.getClientID();
> connectionStatus = true;
> } catch (Exception e) {
> connectionStatus = false;
> }
> return connectionStatus;
> }
>
> public boolean isConsumerActive(MessageConsumer consumer) {
> boolean isConsumerActive = false;
> try {
> consumer.getMessageListener();
> isConsumerActive = true;
> } catch (Exception e) {
> isConsumerActive = false;
> }
> return isConsumerActive;
> }
>
>
> Regards,
> Abhishek Kumar
>
>
>
> --
> Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: qpid-jms-client-56.0 - Prod issue - consumer stopped receiving
message without any connection failure and detached receiver error
Posted by akabhishek1 <ma...@gmail.com>.
Hi Team,
After observing this issue for 12 hours , we performed one activity
- We disabled the one subscription which should throw detached error on
"onException" block, But we haven't received any error.
It means link got detached from broker without informing to connection
exception listener.
We haven't observed this type of issue in 45 from more than years.
Could you please take a look on this issue. This looks like a defect.
Regards,
Abhishek Kumar
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org