When less is more; Re: AbstractNIOConnPool memory leak?

[Oleg Kalnichevski <ol...@apache.org>]

On Sat, 2013-01-05 at 15:58 -0800, vigna wrote:
> Oh, well, I'm sorry, I'm not really a network person :). I meant that we want
> to keep 20K connections busy and transferring data while respecting
> politeness, not to keep them open in the TCP sense. My fault.
> 
> 

Try reducing the number of concurrent connections from 20k to, say, 2k
and you may be surprised to find out that a smaller number of
connections can actually chew through the same workload faster. If the
JVM spends less time switching between contexts (be it thread context
switching or switching channels in a i/o selector) it is more likely to
spend more time actually doing something useful like reading and
processing data. So, it is _really_ that necessary to keep 20k
connections open at the same time?  

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

Re: When less is more; Re: AbstractNIOConnPool memory leak?

[Oleg Kalnichevski <ol...@apache.org>]

On Sun, 2013-01-06 at 23:14 -0800, vigna wrote:
> > Try reducing the number of concurrent connections from 20k to, say, 2k 
> > and you may be surprised to find out that a smaller number of 
> > connections can actually chew through the same workload faster. If the 
> 
> Well... no. :) We have an experimental setup with a local proxy generating a
> "fake web" that we use to check the speed of the pipeline independently of
> the network conditions.
> 
> With 1000 parallel DefaultHttpClient instances (different instances, not one
> instance with pooling) we download >10000 pages/s.
> 
> With 1000 parallel requests on a DefaultHttpAsyncClient we download >500
> pages/s, but as soon as we try to increase the number of parallel requests
> the speed drops to 100 pages/s, which makes the client useless for us at the
> moment.
> 
> Of course this is somewhat artificial—you don't actually download at
> 100MB/s. But the fact that actually with 2000 parallel requests you go
> *slower* is a problem.
> 

I am sorry but I fail to see how that all proves your point (or
disproves mine). It even sounds completely unrelated to what I was
trying to tell you. Well, then, let us just agree to disagree.

Oleg



---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org

Re: When less is more; Re: AbstractNIOConnPool memory leak?

[vigna <vi...@di.unimi.it>]

> Try reducing the number of concurrent connections from 20k to, say, 2k 
> and you may be surprised to find out that a smaller number of 
> connections can actually chew through the same workload faster. If the 

Well... no. :) We have an experimental setup with a local proxy generating a
"fake web" that we use to check the speed of the pipeline independently of
the network conditions.

With 1000 parallel DefaultHttpClient instances (different instances, not one
instance with pooling) we download >10000 pages/s.

With 1000 parallel requests on a DefaultHttpAsyncClient we download >500
pages/s, but as soon as we try to increase the number of parallel requests
the speed drops to 100 pages/s, which makes the client useless for us at the
moment.

Of course this is somewhat artificial—you don't actually download at
100MB/s. But the fact that actually with 2000 parallel requests you go
*slower* is a problem.



--
View this message in context: http://httpcomponents.10934.n7.nabble.com/AbstractNIOConnPool-memory-leak-tp18554p18667.html
Sent from the HttpClient-User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org