MQTT Subscriber gets disconnected frequently when there are large number of MQTT clients connected

[Shobhana <sh...@quickride.in>]

Background : We use AMQ to exchange MQTT non-persistent messages between our
server components and clients (Android/iOS devices). About 100-200
messages/second get exchanged during peak hours.

This is our MQTT transport connector configuration :
<transportConnector name="mqtt+nio"
uri="mqtt+nio://0.0.0.0:1883?maximumConnections=200000&amp;wireFormat.maxInactivityDuration=180000&amp;wireFormat.maxFrameSize=104857600&amp;jms.useAsyncSend=true&amp;jms.alwaysSessionAsync=false"/>

When there are more than 10000 MQTT connections (and roughly about 30000
topics), the MQTT subscribers get disconnected. Even publish operations time
out (timeout is set to 60 seconds on the publisher and subscriber end).
After a few attempts, connection gets re-established. Once broker enters
this state, disconnects become more frequent and after sometime, the
reconnect also fails forever until the broker is restarted!

Any suggestions on how to investigate why this happens? Since this is a
production environment, I can't debug/profile the broker. I have checked
memory usage and CPU usage .. both are well under control. Thread dumps show
that there are no blocked threads. Log level is set to WARN only. Sometimes
I see following error in log files when this happens :



Any clue about what is happening or how to investigate further?

TIA,
Regards,
Shobhana



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: MQTT Subscriber gets disconnected frequently when there are large number of MQTT clients connected

[Tim Bain <tb...@alumni.duke.edu>]

OK. Those log lines indicate that the far side of the connection (the
client) closed the connection when the near end (the broker) wasn't
expecting it. The root cause may be something on the broker's side (for
example, the broker got so busy that it stopped responding to the client,
who reacted by closing the connection), but for now we need to follow the
trail over to the client side and see what its logs say. Can you please
post whatever content is in the client logs shortly before you see the
content you posted from the broker's logs?

Generic reasons why the broker could become unresponsive include not having
enough heap (so the JVM spends most of its time GCing), having so many
connections that it can't service them all quickly enough, disk I/O that's
too slow, or of course maybe there's a bug. Some of those (e.g. spending
too much time GCing) manifest as high CPU usage, and you said CPU usage is
low/normal, but you may still want to double check GC and CPU usage from
JConsole or something similar.

You could also take a thread dump (via jstack or another method, there are
several ways to do this) of your production broker and see if maybe there
are lots of threads blocked waiting for a lock while trying to do real work
(i.e. not the threads in a thread pool that are blocked trying to get a
task from their queue, those are expected and don't indicate a problem).

Tim

On Oct 29, 2017 11:21 PM, "Shobhana" <sh...@quickride.in> wrote:

> Hi Tim,
>
> The log content quoted in my post was from broker log.
>
> I'll try to reproduce this in a test environment. Meanwhile if you could
> think of any common (generic) reasons why broker may becomes unresponsive,
> please share.
>
> Thanks,
> Shobhana
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>

Re: MQTT Subscriber gets disconnected frequently when there are large number of MQTT clients connected

[Shobhana <sh...@quickride.in>]

Hi Tim,

The log content quoted in my post was from broker log.

I'll try to reproduce this in a test environment. Meanwhile if you could
think of any common (generic) reasons why broker may becomes unresponsive,
please share.

Thanks,
Shobhana



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: MQTT Subscriber gets disconnected frequently when there are large number of MQTT clients connected

[Tim Bain <tb...@alumni.duke.edu>]

Is there anything in the broker's logs when this happens? I believe the log
content you quoted was from the client, not the broker, and I wasn't sure
if what you wrote meant that there was nothing in the broker's logs, so I
want to make sure.

Can you reproduce the problem in a non-production environment where the use
of a debugger or a profiler or increasing the logging level of the broker
would be an option? Those techniques are the ones that jump to mind for how
to investigate further, starting with increasing the logging level.

Tim

On Oct 27, 2017 2:19 AM, "Shobhana" <sh...@quickride.in> wrote:

> Background : We use AMQ to exchange MQTT non-persistent messages between
> our
> server components and clients (Android/iOS devices). About 100-200
> messages/second get exchanged during peak hours.
>
> This is our MQTT transport connector configuration :
> <transportConnector name="mqtt+nio"
> uri="mqtt+nio://0.0.0.0:1883?maximumConnections=200000&amp;wireFormat.
> maxInactivityDuration=180000&amp;wireFormat.maxFrameSize=
> 104857600&amp;jms.useAsyncSend=true&amp;jms.alwaysSessionAsync=false"/>
>
> When there are more than 10000 MQTT connections (and roughly about 30000
> topics), the MQTT subscribers get disconnected. Even publish operations
> time
> out (timeout is set to 60 seconds on the publisher and subscriber end).
> After a few attempts, connection gets re-established. Once broker enters
> this state, disconnects become more frequent and after sometime, the
> reconnect also fails forever until the broker is restarted!
>
> Any suggestions on how to investigate why this happens? Since this is a
> production environment, I can't debug/profile the broker. I have checked
> memory usage and CPU usage .. both are well under control. Thread dumps
> show
> that there are no blocked threads. Log level is set to WARN only. Sometimes
> I see following error in log files when this happens :
>
>
>
> Any clue about what is happening or how to investigate further?
>
> TIA,
> Regards,
> Shobhana
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>