Re: local-idle-timeout expired

[Chenxiong Qi <qc...@gmail.com>]

On Wed, Nov 29, 2017 at 5:43 PM Gordon Sim <gs...@redhat.com> wrote:
>
> On 29/11/17 06:57, Michael Ivanov wrote:
> > Greetings,
> >
> > I observe following messages from qpidd in /var/log/messages:
> >
> >     qpidd: 2017-11-29 13:11:47 [Network] error qpid.10.1.1.11:5672-10.1.1.1:65355 connection failed: transport error:
> > amqp:resource-limit-exceeded, local-idle-timeout expired
> >
> > I assume these messages are generated when open connection from
> > client to broker stays idle too long (I was not able to find any
> > description for this in qpidd manual). In our system this can be
> > exactly the case: the client subscribes to the queue and then the
> > link can be idle for several days. How do I suppress these messages?
> > Do I need to send some kind of heartbeat over the idle links?
>
> Yes, it means a client did not send a heartbeat as expected. You
> shouldn't need to do anything tbh though, clients should automatically
> be handling this themselves. What client(s) are you using? Is it
> possible that there were network failures or that the client
> applications hung in some way?
>
Hi,

I met this problem as well. I'm using python-qpid-proton 0.24.0 and
there is no network issue in my case.

My case is the message handler usually runs to handle task for a long
time and messages are coming from producer continuously in the same
time during the task is being handled.

I wrote a simple script to simulate this case by sleeping more than 10
seconds in a message handler on_message, for example,

class ReceiverHandler(MessagingHandler):
    ...

    def on_message(self, event):
        print(event)
        sleep(15)

It can receive a few messages, but will disconnect at some time and
this error is logged:

[0x5606a5060f30]:0 <- @close(24) [error=@error(29)
[condition=:"amqp:resource-limit-exceeded",
description="local-idle-timeout expired"]]

I'm not sure what I can do to fix this. Any suggestion? Also some
questions here.

1. Does it mean by "amqp:resource-limit-exceeded" the broker needs to
be configured to enlarge resource limitation in order to hold more
messages?

2. Does blocking MessagingHandler.on_message also mean the core proton
is blocked as a result proton cannot send heartbeat and even try to
reconnect during the on_message is being blocked? So, is on_message
called synchronously or asynchronously?

3. Is there any connection between max retry delay (10s) and the
heartbeat? If yes, how do they work together?

Thanks.

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


-- 
Regards,
Chenxiong Qi

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: local-idle-timeout expired

[Gordon Sim <gs...@redhat.com>]

On 10/08/18 10:18, Chenxiong Qi wrote:
> I met this problem as well. I'm using python-qpid-proton 0.24.0 and
> there is no network issue in my case.
> 
> My case is the message handler usually runs to handle task for a long
> time and messages are coming from producer continuously in the same
> time during the task is being handled.
> 
> I wrote a simple script to simulate this case by sleeping more than 10
> seconds in a message handler on_message, for example,
> 
> class ReceiverHandler(MessagingHandler):
>      ...
> 
>      def on_message(self, event):
>          print(event)
>          sleep(15)

The proton python client is single threaded. If you block the thread by 
sleeping you will prevent anything happening, including heartbeats.

> It can receive a few messages, but will disconnect at some time and
> this error is logged:
> 
> [0x5606a5060f30]:0 <- @close(24) [error=@error(29)
> [condition=:"amqp:resource-limit-exceeded",
> description="local-idle-timeout expired"]]
> 
> I'm not sure what I can do to fix this. Any suggestion?

Any blocking work needs to be done on a separate thread. You can use 
ApplicationEvent and EventInjector to communicate safely from other 
threads to the event thread. (See the db_send.py/db_recv.py for one 
example).

> Also some
> questions here.
> 
> 1. Does it mean by "amqp:resource-limit-exceeded" the broker needs to
> be configured to enlarge resource limitation in order to hold more
> messages?

No, it just means the time between heartbeats was too long. Now you 
mention it, I agree that it does seem a bit of an odd condition. Not 
sure where that originated from.

> 2. Does blocking MessagingHandler.on_message also mean the core proton
> is blocked as a result proton cannot send heartbeat and even try to
> reconnect during the on_message is being blocked? So, is on_message
> called synchronously or asynchronously?

It is called synchronously by the event thread (the thread that calls 
Container.run().

> 3. Is there any connection between max retry delay (10s) and the
> heartbeat? If yes, how do they work together?

The heartbeat is used to determine if a connection is still alive. Once 
a connection fails, the library will attempt to reconnect (assuming 
configuration allows that). If it continues to fail, it will increase 
the interval between retry attempts. The max retry delay limits the 
amount by which it will increase that interval between reconnect attempts.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org