You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Michael Ivanov <iv...@logit-ag.de> on 2017/11/29 06:57:05 UTC
local-idle-timeout expired
Greetings,
I observe following messages from qpidd in /var/log/messages:
qpidd: 2017-11-29 13:11:47 [Network] error qpid.10.1.1.11:5672-10.1.1.1:65355 connection failed: transport error:
amqp:resource-limit-exceeded, local-idle-timeout expired
I assume these messages are generated when open connection from
client to broker stays idle too long (I was not able to find any
description for this in qpidd manual). In our system this can be
exactly the case: the client subscribes to the queue and then the
link can be idle for several days. How do I suppress these messages?
Do I need to send some kind of heartbeat over the idle links?
Best regards,
--
\ / | |
(OvO) | Mikhail Iwanow |
(^^^) | Voice: +7 (911) 223-1300 |
\^/ | E-mail: ivans@logit-ag.de |
^ ^ | |
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Gordon Sim <gs...@redhat.com>.
On 10/08/18 10:18, Chenxiong Qi wrote:
> I met this problem as well. I'm using python-qpid-proton 0.24.0 and
> there is no network issue in my case.
>
> My case is the message handler usually runs to handle task for a long
> time and messages are coming from producer continuously in the same
> time during the task is being handled.
>
> I wrote a simple script to simulate this case by sleeping more than 10
> seconds in a message handler on_message, for example,
>
> class ReceiverHandler(MessagingHandler):
> ...
>
> def on_message(self, event):
> print(event)
> sleep(15)
The proton python client is single threaded. If you block the thread by
sleeping you will prevent anything happening, including heartbeats.
> It can receive a few messages, but will disconnect at some time and
> this error is logged:
>
> [0x5606a5060f30]:0 <- @close(24) [error=@error(29)
> [condition=:"amqp:resource-limit-exceeded",
> description="local-idle-timeout expired"]]
>
> I'm not sure what I can do to fix this. Any suggestion?
Any blocking work needs to be done on a separate thread. You can use
ApplicationEvent and EventInjector to communicate safely from other
threads to the event thread. (See the db_send.py/db_recv.py for one
example).
> Also some
> questions here.
>
> 1. Does it mean by "amqp:resource-limit-exceeded" the broker needs to
> be configured to enlarge resource limitation in order to hold more
> messages?
No, it just means the time between heartbeats was too long. Now you
mention it, I agree that it does seem a bit of an odd condition. Not
sure where that originated from.
> 2. Does blocking MessagingHandler.on_message also mean the core proton
> is blocked as a result proton cannot send heartbeat and even try to
> reconnect during the on_message is being blocked? So, is on_message
> called synchronously or asynchronously?
It is called synchronously by the event thread (the thread that calls
Container.run().
> 3. Is there any connection between max retry delay (10s) and the
> heartbeat? If yes, how do they work together?
The heartbeat is used to determine if a connection is still alive. Once
a connection fails, the library will attempt to reconnect (assuming
configuration allows that). If it continues to fail, it will increase
the interval between retry attempts. The max retry delay limits the
amount by which it will increase that interval between reconnect attempts.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Chenxiong Qi <qc...@gmail.com>.
On Wed, Nov 29, 2017 at 5:43 PM Gordon Sim <gs...@redhat.com> wrote:
>
> On 29/11/17 06:57, Michael Ivanov wrote:
> > Greetings,
> >
> > I observe following messages from qpidd in /var/log/messages:
> >
> > qpidd: 2017-11-29 13:11:47 [Network] error qpid.10.1.1.11:5672-10.1.1.1:65355 connection failed: transport error:
> > amqp:resource-limit-exceeded, local-idle-timeout expired
> >
> > I assume these messages are generated when open connection from
> > client to broker stays idle too long (I was not able to find any
> > description for this in qpidd manual). In our system this can be
> > exactly the case: the client subscribes to the queue and then the
> > link can be idle for several days. How do I suppress these messages?
> > Do I need to send some kind of heartbeat over the idle links?
>
> Yes, it means a client did not send a heartbeat as expected. You
> shouldn't need to do anything tbh though, clients should automatically
> be handling this themselves. What client(s) are you using? Is it
> possible that there were network failures or that the client
> applications hung in some way?
>
Hi,
I met this problem as well. I'm using python-qpid-proton 0.24.0 and
there is no network issue in my case.
My case is the message handler usually runs to handle task for a long
time and messages are coming from producer continuously in the same
time during the task is being handled.
I wrote a simple script to simulate this case by sleeping more than 10
seconds in a message handler on_message, for example,
class ReceiverHandler(MessagingHandler):
...
def on_message(self, event):
print(event)
sleep(15)
It can receive a few messages, but will disconnect at some time and
this error is logged:
[0x5606a5060f30]:0 <- @close(24) [error=@error(29)
[condition=:"amqp:resource-limit-exceeded",
description="local-idle-timeout expired"]]
I'm not sure what I can do to fix this. Any suggestion? Also some
questions here.
1. Does it mean by "amqp:resource-limit-exceeded" the broker needs to
be configured to enlarge resource limitation in order to hold more
messages?
2. Does blocking MessagingHandler.on_message also mean the core proton
is blocked as a result proton cannot send heartbeat and even try to
reconnect during the on_message is being blocked? So, is on_message
called synchronously or asynchronously?
3. Is there any connection between max retry delay (10s) and the
heartbeat? If yes, how do they work together?
Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
Regards,
Chenxiong Qi
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Michael Ivanov <iv...@logit-ag.de>.
All clients are running in a single tomcat instance which runs on
the same VM as broker (yes the system runs in VM).
Most of the clients (probably even all I'm not 100% sure) use JMS.
29.11.2017 20:32, Gordon Sim пишет:
> On 29/11/17 12:39, Michael Ivanov wrote:
>> Thanks for reply!
>>
>> The clients are mostly run in tomcat environment using proton-j-0.11.0.jar
>> and qpid-jms-client-0.6.0.jar.
>
> Are they all using the JMS api?
>
>> I did not observe network failures when this
>> happens, but yes sometimes tomcat loops and then the number of such messages
>> increases noticeably. But they are also present (though not that frequent)
>> when tomcat is restarted and system runs normal.
>
> Usually I would expect connections to be closed when the process is killed (it is not running in a VM by any chance is it?), but
> if a connection if the broker thinks the connection is still open but it gets no heartbeats, it will close the connection.
> Unfortunately there is no good way to prevent the log message as doing so would prevent other errors from being reported.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
\ / | |
(OvO) | Mikhail Iwanow |
(^^^) | Voice: +7 (911) 223-1300 |
\^/ | E-mail: ivans@logit-ag.de |
^ ^ | |
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Gordon Sim <gs...@redhat.com>.
On 29/11/17 12:39, Michael Ivanov wrote:
> Thanks for reply!
>
> The clients are mostly run in tomcat environment using proton-j-0.11.0.jar
> and qpid-jms-client-0.6.0.jar.
Are they all using the JMS api?
> I did not observe network failures when this
> happens, but yes sometimes tomcat loops and then the number of such messages
> increases noticeably. But they are also present (though not that frequent)
> when tomcat is restarted and system runs normal.
Usually I would expect connections to be closed when the process is
killed (it is not running in a VM by any chance is it?), but if a
connection if the broker thinks the connection is still open but it gets
no heartbeats, it will close the connection. Unfortunately there is no
good way to prevent the log message as doing so would prevent other
errors from being reported.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Michael Ivanov <iv...@logit-ag.de>.
Thanks for reply!
The clients are mostly run in tomcat environment using proton-j-0.11.0.jar
and qpid-jms-client-0.6.0.jar. I did not observe network failures when this
happens, but yes sometimes tomcat loops and then the number of such messages
increases noticeably. But they are also present (though not that frequent)
when tomcat is restarted and system runs normal.
Best regards,
29.11.2017 12:43, Gordon Sim пишет:
> On 29/11/17 06:57, Michael Ivanov wrote:
>> Greetings,
>>
>> I observe following messages from qpidd in /var/log/messages:
>>
>> qpidd: 2017-11-29 13:11:47 [Network] error qpid.10.1.1.11:5672-10.1.1.1:65355 connection failed: transport error:
>> amqp:resource-limit-exceeded, local-idle-timeout expired
>>
>> I assume these messages are generated when open connection from
>> client to broker stays idle too long (I was not able to find any
>> description for this in qpidd manual). In our system this can be
>> exactly the case: the client subscribes to the queue and then the
>> link can be idle for several days. How do I suppress these messages?
>> Do I need to send some kind of heartbeat over the idle links?
>
> Yes, it means a client did not send a heartbeat as expected. You shouldn't need to do anything tbh though, clients should
> automatically be handling this themselves. What client(s) are you using? Is it possible that there were network failures or that
> the client applications hung in some way?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
\ / | |
(OvO) | Mikhail Iwanow |
(^^^) | Voice: +7 (911) 223-1300 |
\^/ | E-mail: ivans@logit-ag.de |
^ ^ | |
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: local-idle-timeout expired
Posted by Gordon Sim <gs...@redhat.com>.
On 29/11/17 06:57, Michael Ivanov wrote:
> Greetings,
>
> I observe following messages from qpidd in /var/log/messages:
>
> qpidd: 2017-11-29 13:11:47 [Network] error qpid.10.1.1.11:5672-10.1.1.1:65355 connection failed: transport error:
> amqp:resource-limit-exceeded, local-idle-timeout expired
>
> I assume these messages are generated when open connection from
> client to broker stays idle too long (I was not able to find any
> description for this in qpidd manual). In our system this can be
> exactly the case: the client subscribes to the queue and then the
> link can be idle for several days. How do I suppress these messages?
> Do I need to send some kind of heartbeat over the idle links?
Yes, it means a client did not send a heartbeat as expected. You
shouldn't need to do anything tbh though, clients should automatically
be handling this themselves. What client(s) are you using? Is it
possible that there were network failures or that the client
applications hung in some way?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org