Warnings - Client connection failed / Cleared up resources for session

["Wilcox, Jim" <Ji...@maxar.com.INVALID>]

We see several Artemis production level defects still open involving these warnings showing up and then consequences with losing messages:

2021-03-18 19:31:37,045 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 7bf29b0d-8820-11eb-b383-90b11c6cd262
2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 7bf29b0d-8820-11eb-b383-90b11c6cd262
2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server] AMQ222061: Client connection failed, clearing up resources for session 7c007dbe-8820-11eb-b383-90b11c6cd262
2021-03-18 19:31:37,048 WARN [org.apache.activemq.artemis.core.server] AMQ222107: Cleared up resources for session 7c007dbe-8820-11eb-b383-90b11c6cd262

Can I get some more detail on what causes the above to show up?

And we are also producing the above at some point in our jms client tests against artemis 2.20 and they previously showed up with 2.17. Our tests do "pass" but we are seeing issues with unused queues not getting deleted as expected... and we do not know at this point if they are related.

It would seem we should not see those warnings in our logs during normal operations correct? Unless perhaps the client is truly having a connection/internet issue? Which in this case I am not observing that.

Thanks, Jim

This electronic communication and any attachments may contain confidential and proprietary information of Maxar, Inc. If you are not the intended recipient, or an agent or employee responsible for delivering this communication to the intended recipient, or if you have received this communication in error, please do not print, copy, retransmit, disseminate or otherwise use the information. Please indicate to the sender that you have received this communication in error, and delete the copy you received.

Maxar reserves the right to monitor any electronic communication sent or received by its employees, agents or representatives.

RE: Warnings - Client connection failed / Cleared up resources for session

["Wilcox, Jim" <Ji...@maxar.com.INVALID>]

Thanks much Justin for taking the time to respond with that insight and yes that is very helpful.

Fyi, it looks like we may keep supporting classic for a while longer, but this context is great to have as we dive in more on what additionally we may need to convert etc.

Jim

-----Original Message-----
From: Justin Bertram <jb...@apache.org> 
Sent: Thursday, February 3, 2022 10:15 AM
To: users@activemq.apache.org
Subject: Re: Warnings - Client connection failed / Cleared up resources for session

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


> We see several Artemis production level defects still open involving
these warnings showing up and then consequences with losing messages...Can I get some more detail on what causes the above to show up?

For what it's worth, these WARN messages are not necessarily unexpected.
They are emitted when a client fails to send data within the allotted time.
The most common cause of this is a client opening a connection and then not closing that connection before it exits.

The exact behavior varies from protocol to protocol, but any protocol that supports any kind of connection keep-alive mechanism is subject to this check. It's important for the broker to keep track of connections like this to prevent unnecessary resource consumption. I don't think they represent "production level defects" in the broker as you indicated, and they certainly wouldn't represent the loss of messages - at least not from the broker's perspective.

> And we are also producing the above at some point in our jms client 
> tests
against artemis 2.20 and they previously showed up with 2.17.

I'm not aware of any existing or suspected bugs in this area so my first guess is that you have a client somewhere in your tests that isn't closing its connection properly.

> Our tests do "pass" but we are seeing issues with unused queues not
getting deleted as expected... and we do not know at this point if they are related.

Can you be more specific about this? In what way exactly is queue deletion not meeting your expectation? There's been a few changes recently around queue deletion. Most notably, the default configuration no longer enables automatic queue deletion.

In any event, by default the broker should only delete a queue when it is empty soas to avoid message loss.

> It would seem we should not see those warnings in our logs during 
> normal
operations correct? Unless perhaps the client is truly having a connection/internet issue? Which in this case I am not observing that.

I suppose it depends on what "normal operations" means for you. I've seen user environments with lots of these kinds of warnings and they accepted them as normal because they knew the network was flaky or that the client was buggy and leaked connections occasionally. A client leaking a connection is likely not something you would "see" - at least not from the client's perspective. However, you would definitely see this WARN on the broker if that was the case.

Hope that helps!


Justin

On Wed, Feb 2, 2022 at 12:13 PM Wilcox, Jim <Ji...@maxar.com.invalid>
wrote:

> We see several Artemis production level defects still open involving 
> these warnings showing up and then consequences with losing messages:
>
> 2021-03-18 19:31:37,045 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 7bf29b0d-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 7bf29b0d-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 7c007dbe-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,048 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 7c007dbe-8820-11eb-b383-90b11c6cd262
>
> Can I get some more detail on what causes the above to show up?
>
> And we are also producing the above at some point in our jms client 
> tests against artemis 2.20 and they previously showed up with 2.17. 
> Our tests do "pass" but we are seeing issues with unused queues not 
> getting deleted as expected... and we do not know at this point if they are related.
>
> It would seem we should not see those warnings in our logs during 
> normal operations correct? Unless perhaps the client is truly having a 
> connection/internet issue? Which in this case I am not observing that.
>
> Thanks, Jim
>
> This electronic communication and any attachments may contain 
> confidential and proprietary information of Maxar, Inc. If you are not 
> the intended recipient, or an agent or employee responsible for 
> delivering this communication to the intended recipient, or if you 
> have received this communication in error, please do not print, copy, 
> retransmit, disseminate or otherwise use the information. Please 
> indicate to the sender that you have received this communication in error, and delete the copy you received.
>
> Maxar reserves the right to monitor any electronic communication sent 
> or received by its employees, agents or representatives.
>

Re: Warnings - Client connection failed / Cleared up resources for session

[Justin Bertram <jb...@apache.org>]

> We see several Artemis production level defects still open involving
these warnings showing up and then consequences with losing messages...Can
I get some more detail on what causes the above to show up?

For what it's worth, these WARN messages are not necessarily unexpected.
They are emitted when a client fails to send data within the allotted time.
The most common cause of this is a client opening a connection and then not
closing that connection before it exits.

The exact behavior varies from protocol to protocol, but any protocol that
supports any kind of connection keep-alive mechanism is subject to this
check. It's important for the broker to keep track of connections like this
to prevent unnecessary resource consumption. I don't think they represent
"production level defects" in the broker as you indicated, and they
certainly wouldn't represent the loss of messages - at least not from the
broker's perspective.

> And we are also producing the above at some point in our jms client tests
against artemis 2.20 and they previously showed up with 2.17.

I'm not aware of any existing or suspected bugs in this area so my first
guess is that you have a client somewhere in your tests that isn't closing
its connection properly.

> Our tests do "pass" but we are seeing issues with unused queues not
getting deleted as expected... and we do not know at this point if they are
related.

Can you be more specific about this? In what way exactly is queue deletion
not meeting your expectation? There's been a few changes recently around
queue deletion. Most notably, the default configuration no longer enables
automatic queue deletion.

In any event, by default the broker should only delete a queue when it is
empty soas to avoid message loss.

> It would seem we should not see those warnings in our logs during normal
operations correct? Unless perhaps the client is truly having a
connection/internet issue? Which in this case I am not observing that.

I suppose it depends on what "normal operations" means for you. I've seen
user environments with lots of these kinds of warnings and they accepted
them as normal because they knew the network was flaky or that the client
was buggy and leaked connections occasionally. A client leaking a
connection is likely not something you would "see" - at least not from the
client's perspective. However, you would definitely see this WARN on the
broker if that was the case.

Hope that helps!


Justin

On Wed, Feb 2, 2022 at 12:13 PM Wilcox, Jim <Ji...@maxar.com.invalid>
wrote:

> We see several Artemis production level defects still open involving these
> warnings showing up and then consequences with losing messages:
>
> 2021-03-18 19:31:37,045 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 7bf29b0d-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 7bf29b0d-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,047 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 7c007dbe-8820-11eb-b383-90b11c6cd262
> 2021-03-18 19:31:37,048 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 7c007dbe-8820-11eb-b383-90b11c6cd262
>
> Can I get some more detail on what causes the above to show up?
>
> And we are also producing the above at some point in our jms client tests
> against artemis 2.20 and they previously showed up with 2.17. Our tests do
> "pass" but we are seeing issues with unused queues not getting deleted as
> expected... and we do not know at this point if they are related.
>
> It would seem we should not see those warnings in our logs during normal
> operations correct? Unless perhaps the client is truly having a
> connection/internet issue? Which in this case I am not observing that.
>
> Thanks, Jim
>
> This electronic communication and any attachments may contain confidential
> and proprietary information of Maxar, Inc. If you are not the intended
> recipient, or an agent or employee responsible for delivering this
> communication to the intended recipient, or if you have received this
> communication in error, please do not print, copy, retransmit, disseminate
> or otherwise use the information. Please indicate to the sender that you
> have received this communication in error, and delete the copy you received.
>
> Maxar reserves the right to monitor any electronic communication sent or
> received by its employees, agents or representatives.
>