Artemis MQTT broker limits the number of clients and keeps old bindings forever

[ac...@tengo.one]

Hi there,

I’m using Artemis 2.6.3  as an MQTT broker embedded in a Spring 5 application with persistence enabled. I have not configured a limit for the number of connections or anything.

One of our customers reported an issue when they subscribe to MQTT topics. To find the root cause, I create a a large number of clients in Java using Eclipse Paho, each with its own client ID. The clients connect with cleanSession=false to the broker and then subscribe to some topics. I found out that after about 140 client connections (actual number differs from test run to test run), new clients receive "java.io.IOException: Connection is lost.“ with reason 32109. After a few seconds, a few new clients can connect (around 10). When I terminate all connected clients without unsubscribing from the topics (basically I’m simulating a crash of the clients here), again only a few new clients (with client IDs not used so far) can connect before receiving the IOException. Reconnecting the 140 clients with their original client ID is no problem.

Looking into the logs, I see the following message for each client and each topic the client subscribed to: 
2020-06-17 12:39:15,446 [Thread-27 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@6ef80f24)] DEBUG o.a.a.a.core.server.impl.QueueImpl - Scanning for expires on client-96.base-topic.*.topic
The "Scanning for expires“ is triggered every 30 seconds, so I see hundreds of these messages in the log every 30 seconds. These messages keep coming for hours (maybe forever), even for the „creashed“ clients. This seems to me that our customers will sooner or later run into problems when they have a large number of clients that crash from time to time. In particular, if they do not reuse the client IDs.

From reading the documentation (https://activemq.apache.org/components/artemis/documentation/1.0.0/resource-limits.html), I would expect that the number of connections is not limited.

So I have two issues where I would appreciate any help:
1. Why is the number of connections limited in my scenario?
2. How can my broker get rid of the „dead" bindings of the „crashed“ clients?

Thanks a lot
Tobias

Re: Artemis MQTT broker limits the number of clients and keeps old bindings forever

[Clebert Suconic <cl...@gmail.com>]

if you're facing some bug more specifically, i would ask you to
produce a testcase and try it with a later version.

Perhaps what you need is a non durable queue, that would be removed
when the connection is gone?


On Tue, Jun 23, 2020 at 4:40 PM Clebert Suconic
<cl...@gmail.com> wrote:
>
> Artemis doesn't impose a limit on the number of connections, but that
> doesn't mean you shouldn't cleanup after your queues.
>
> You should probably manage the queue creation. if you are allowing
> user's to create durable subscription, their subscriptions will be
> available beyond your client life cycle. You should manage these
> topics and queues.
>
>
> On Wed, Jun 17, 2020 at 9:06 AM <ac...@tengo.one> wrote:
> >
> > Hi there,
> >
> > I’m using Artemis 2.6.3  as an MQTT broker embedded in a Spring 5 application with persistence enabled. I have not configured a limit for the number of connections or anything.
> >
> > One of our customers reported an issue when they subscribe to MQTT topics. To find the root cause, I create a a large number of clients in Java using Eclipse Paho, each with its own client ID. The clients connect with cleanSession=false to the broker and then subscribe to some topics. I found out that after about 140 client connections (actual number differs from test run to test run), new clients receive "java.io.IOException: Connection is lost.“ with reason 32109. After a few seconds, a few new clients can connect (around 10). When I terminate all connected clients without unsubscribing from the topics (basically I’m simulating a crash of the clients here), again only a few new clients (with client IDs not used so far) can connect before receiving the IOException. Reconnecting the 140 clients with their original client ID is no problem.
> >
> > Looking into the logs, I see the following message for each client and each topic the client subscribed to:
> > 2020-06-17 12:39:15,446 [Thread-27 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@6ef80f24)] DEBUG o.a.a.a.core.server.impl.QueueImpl - Scanning for expires on client-96.base-topic.*.topic
> > The "Scanning for expires“ is triggered every 30 seconds, so I see hundreds of these messages in the log every 30 seconds. These messages keep coming for hours (maybe forever), even for the „creashed“ clients. This seems to me that our customers will sooner or later run into problems when they have a large number of clients that crash from time to time. In particular, if they do not reuse the client IDs.
> >
> > From reading the documentation (https://activemq.apache.org/components/artemis/documentation/1.0.0/resource-limits.html), I would expect that the number of connections is not limited.
> >
> > So I have two issues where I would appreciate any help:
> > 1. Why is the number of connections limited in my scenario?
> > 2. How can my broker get rid of the „dead" bindings of the „crashed“ clients?
> >
> > Thanks a lot
> > Tobias
>
>
>
> --
> Clebert Suconic



-- 
Clebert Suconic

Re: Artemis MQTT broker limits the number of clients and keeps old bindings forever

[Clebert Suconic <cl...@gmail.com>]

Artemis doesn't impose a limit on the number of connections, but that
doesn't mean you shouldn't cleanup after your queues.

You should probably manage the queue creation. if you are allowing
user's to create durable subscription, their subscriptions will be
available beyond your client life cycle. You should manage these
topics and queues.


On Wed, Jun 17, 2020 at 9:06 AM <ac...@tengo.one> wrote:
>
> Hi there,
>
> I’m using Artemis 2.6.3  as an MQTT broker embedded in a Spring 5 application with persistence enabled. I have not configured a limit for the number of connections or anything.
>
> One of our customers reported an issue when they subscribe to MQTT topics. To find the root cause, I create a a large number of clients in Java using Eclipse Paho, each with its own client ID. The clients connect with cleanSession=false to the broker and then subscribe to some topics. I found out that after about 140 client connections (actual number differs from test run to test run), new clients receive "java.io.IOException: Connection is lost.“ with reason 32109. After a few seconds, a few new clients can connect (around 10). When I terminate all connected clients without unsubscribing from the topics (basically I’m simulating a crash of the clients here), again only a few new clients (with client IDs not used so far) can connect before receiving the IOException. Reconnecting the 140 clients with their original client ID is no problem.
>
> Looking into the logs, I see the following message for each client and each topic the client subscribed to:
> 2020-06-17 12:39:15,446 [Thread-27 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@6ef80f24)] DEBUG o.a.a.a.core.server.impl.QueueImpl - Scanning for expires on client-96.base-topic.*.topic
> The "Scanning for expires“ is triggered every 30 seconds, so I see hundreds of these messages in the log every 30 seconds. These messages keep coming for hours (maybe forever), even for the „creashed“ clients. This seems to me that our customers will sooner or later run into problems when they have a large number of clients that crash from time to time. In particular, if they do not reuse the client IDs.
>
> From reading the documentation (https://activemq.apache.org/components/artemis/documentation/1.0.0/resource-limits.html), I would expect that the number of connections is not limited.
>
> So I have two issues where I would appreciate any help:
> 1. Why is the number of connections limited in my scenario?
> 2. How can my broker get rid of the „dead" bindings of the „crashed“ clients?
>
> Thanks a lot
> Tobias



-- 
Clebert Suconic