You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Dmitry Potapov <po...@gmail.com> on 2013/11/27 19:09:29 UTC
AbstractNIOConnPool.processPendingRequests is never called?
Hello everyone,
I've got OOM today and trying to understand if I'm using HttpAsyncClient wrong.
So. I have HttpAsyncClient sending 100-150 requests per second to single remote
host. Connection request timeout is 50 ms, connect timeout is 25 ms, socket
timeout is 100 ms.
In heap dump I've found about 200,000 LeaseRequest object. Since
completedRequests is cleaned in the end of each lease(...) function call, I'm
pretty sure that this is a leasingRequests.
Looks like that remote host went down for a while and stopped requests
processing. So, CPool started to accumulate lease requests and after several
hours tons of accumulated LeaseRequest objects caused OOM.
I've tried to find where leasingRequests must be cleaned and found that only
function which performs cleanup by request deadline is
AbstractNIOConnPool.processPendingRequests() which is called only by
PoolingNHttpClientConnectionManager.closeIdleConnections() or
.closeExpiredConnections().
So, my question is, should I call this functions periodically on my side, or
there is another leasingRequests clean up mechanism which I didn't take into
account?
--
Thanks in advance,
Dmitry Potapov
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: AbstractNIOConnPool.processPendingRequests is never called?
Posted by Dmitry Potapov <po...@gmail.com>.
On Thu, Nov 28, 2013 at 10:18:06AM +0100, Oleg Kalnichevski wrote:
> On Wed, 2013-11-27 at 22:09 +0400, Dmitry Potapov wrote:
> > Hello everyone,
> >
> > I've got OOM today and trying to understand if I'm using HttpAsyncClient wrong.
> >
> > So. I have HttpAsyncClient sending 100-150 requests per second to single remote
> > host. Connection request timeout is 50 ms, connect timeout is 25 ms, socket
> > timeout is 100 ms.
> > In heap dump I've found about 200,000 LeaseRequest object. Since
> > completedRequests is cleaned in the end of each lease(...) function call, I'm
> > pretty sure that this is a leasingRequests.
> >
> > Looks like that remote host went down for a while and stopped requests
> > processing. So, CPool started to accumulate lease requests and after several
> > hours tons of accumulated LeaseRequest objects caused OOM.
> > I've tried to find where leasingRequests must be cleaned and found that only
> > function which performs cleanup by request deadline is
> > AbstractNIOConnPool.processPendingRequests() which is called only by
> > PoolingNHttpClientConnectionManager.closeIdleConnections() or
> > .closeExpiredConnections().
> >
> > So, my question is, should I call this functions periodically on my side, or
> > there is another leasingRequests clean up mechanism which I didn't take into
> > account?
> >
>
> Dmitry,
>
> The only way to force the pooling connection manager to do a full linear
> scan on the connection request queue is by calling
> #closeExpiredConnections every once in a while (usually from a monitor
> thread).
>
> However, the queue _should_ not get clogged with requests as long as
> both connect and connection request timeouts are set to a positive
> value. A connect timeout always causes eviction of expired requests from
> the queue until a non-expired request is found. This strategy generally
> should be good enough.
>
> Would it be possible for you to reproduce the issue with the context
> logging for connection management / request execution turned on?
>
> Oleg
>
Oleg,
Thank you for your reply. Unfortunately I was unable to reproduce this issue on
synthetic test. If I would be so (un)lucky to encounter this issue once again,
I'll try to debug this on live system.
--
Thanks,
Dmitry Potapov
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: AbstractNIOConnPool.processPendingRequests is never called?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-11-27 at 22:09 +0400, Dmitry Potapov wrote:
> Hello everyone,
>
> I've got OOM today and trying to understand if I'm using HttpAsyncClient wrong.
>
> So. I have HttpAsyncClient sending 100-150 requests per second to single remote
> host. Connection request timeout is 50 ms, connect timeout is 25 ms, socket
> timeout is 100 ms.
> In heap dump I've found about 200,000 LeaseRequest object. Since
> completedRequests is cleaned in the end of each lease(...) function call, I'm
> pretty sure that this is a leasingRequests.
>
> Looks like that remote host went down for a while and stopped requests
> processing. So, CPool started to accumulate lease requests and after several
> hours tons of accumulated LeaseRequest objects caused OOM.
> I've tried to find where leasingRequests must be cleaned and found that only
> function which performs cleanup by request deadline is
> AbstractNIOConnPool.processPendingRequests() which is called only by
> PoolingNHttpClientConnectionManager.closeIdleConnections() or
> .closeExpiredConnections().
>
> So, my question is, should I call this functions periodically on my side, or
> there is another leasingRequests clean up mechanism which I didn't take into
> account?
>
Dmitry,
The only way to force the pooling connection manager to do a full linear
scan on the connection request queue is by calling
#closeExpiredConnections every once in a while (usually from a monitor
thread).
However, the queue _should_ not get clogged with requests as long as
both connect and connection request timeouts are set to a positive
value. A connect timeout always causes eviction of expired requests from
the queue until a non-expired request is found. This strategy generally
should be good enough.
Would it be possible for you to reproduce the issue with the context
logging for connection management / request execution turned on?
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org