You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Michael <mi...@gmail.com> on 2009/07/09 17:05:14 UTC
performance tuning possibilities?
I completed my migration to the latest version of the HttpClient
successfully. It is just a bit slower now than I would have ecpected...
Maybe somebody has some tuning experiences to share?
* Reliability is not that important for me
* It should do around 20 accesses per minute using slow proxies, connection
timeout is set to 6 seconds for them. Currently it does those 20 accesses in
100 seconds :-(
* I have 3 threads executing http gets concurrently.
* I am going with ThreadSafeClientConnManager and 30 connections at max. Is
that enough? Or too much?
* I use a retry hanlder with up to 6 retries. Not sure if that is eating up
much time?
* TCPNoDelay is false. Would turning that on improve performance to some
extent? How much?
* Stale Checking is enabled. Maybe I dont need this since I use the
IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5
seconds?
* Are there more tuning opportunities available?
Thx for helping!
Michael
Re: performance tuning possibilities?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, Jul 09, 2009 at 08:27:33PM +0200, Michael Gnatz wrote:
> OK, Oleg's suggestions improved the whole thing around 30%, great stuff!
>
> Where would the performance boost with using the Core directly come from?
>
HttpCore implements just a very basic HTTP transport. It can transfer HTTP
messages across the wire very efficiently, but it cannot (and is not supposed
to) do anything beyond that. HttpClient adds connection management, state
management, authentication and rediect handling logic on top of HttpCore. All
that extra functionality results in a ~30% performance penalty. If you do not
mind managing connections manually, do not need cookie support, authentication
and automatic redirects, you may well consider using HttpCore directly.
Oleg
>
> 2009/7/9 Oleg Kalnichevski <ol...@apache.org>
>
> > On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote:
> > > I completed my migration to the latest version of the HttpClient
> > > successfully. It is just a bit slower now than I would have ecpected...
> > >
> > > Maybe somebody has some tuning experiences to share?
> > >
> >
> > Michael,
> >
> > HttpClient should be _at least_ as fast as HttpClient 3.x. If performance
> > is an
> > overriding factor for you, you should consider using HttpCore directly and
> > get
> > 30 to 40% performance boost.
> >
> > http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore
> >
> >
> >
> > > * Reliability is not that important for me
> >
> > Make sure to turn off stale connection check and expect-continue handshake.
> >
> > > * It should do around 20 accesses per minute using slow proxies,
> > connection
> > > timeout is set to 6 seconds for them. Currently it does those 20 accesses
> > in
> > > 100 seconds :-(
> > >
> >
> > To find out where time is being spent, turn on context logging. Make sure
> > to
> > disable the wire log, though, otherwise you will be measuring performance
> > of
> > console output.
> >
> > http://hc.apache.org/httpcomponents-client/logging.html
> >
> >
> > >
> > > * I have 3 threads executing http gets concurrently.
> > > * I am going with ThreadSafeClientConnManager and 30 connections at max.
> > Is
> > > that enough? Or too much?
> >
> > You also have to increase the default limit of 2 concurrent connections per
> > host. It should be at least the same as the number of worker thread to
> > ensure
> > minimal resource contension.
> >
> >
> > > * I use a retry hanlder with up to 6 retries. Not sure if that is eating
> > up
> > > much time?
> > > * TCPNoDelay is false. Would turning that on improve performance to some
> > > extent? How much?
> >
> > Whooooaaaaa! Nagle's algorithm just kills HTTP performance. Make sure the
> > TCPNoDelay is set to true.
> >
> > > * Stale Checking is enabled. Maybe I dont need this since I use the
> > > IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every
> > 5
> > > seconds?
> > >
> >
> > The stale connection check is more evil than the Church of Scientology.
> > Just do
> > not use it.
> >
> > Hope this helps
> >
> > Oleg
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: performance tuning possibilities?
Posted by Michael Gnatz <mi...@balniki.de>.
OK, Oleg's suggestions improved the whole thing around 30%, great stuff!
Where would the performance boost with using the Core directly come from?
2009/7/9 Oleg Kalnichevski <ol...@apache.org>
> On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote:
> > I completed my migration to the latest version of the HttpClient
> > successfully. It is just a bit slower now than I would have ecpected...
> >
> > Maybe somebody has some tuning experiences to share?
> >
>
> Michael,
>
> HttpClient should be _at least_ as fast as HttpClient 3.x. If performance
> is an
> overriding factor for you, you should consider using HttpCore directly and
> get
> 30 to 40% performance boost.
>
> http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore
>
>
>
> > * Reliability is not that important for me
>
> Make sure to turn off stale connection check and expect-continue handshake.
>
> > * It should do around 20 accesses per minute using slow proxies,
> connection
> > timeout is set to 6 seconds for them. Currently it does those 20 accesses
> in
> > 100 seconds :-(
> >
>
> To find out where time is being spent, turn on context logging. Make sure
> to
> disable the wire log, though, otherwise you will be measuring performance
> of
> console output.
>
> http://hc.apache.org/httpcomponents-client/logging.html
>
>
> >
> > * I have 3 threads executing http gets concurrently.
> > * I am going with ThreadSafeClientConnManager and 30 connections at max.
> Is
> > that enough? Or too much?
>
> You also have to increase the default limit of 2 concurrent connections per
> host. It should be at least the same as the number of worker thread to
> ensure
> minimal resource contension.
>
>
> > * I use a retry hanlder with up to 6 retries. Not sure if that is eating
> up
> > much time?
> > * TCPNoDelay is false. Would turning that on improve performance to some
> > extent? How much?
>
> Whooooaaaaa! Nagle's algorithm just kills HTTP performance. Make sure the
> TCPNoDelay is set to true.
>
> > * Stale Checking is enabled. Maybe I dont need this since I use the
> > IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every
> 5
> > seconds?
> >
>
> The stale connection check is more evil than the Church of Scientology.
> Just do
> not use it.
>
> Hope this helps
>
> Oleg
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: EXPECT-CONTINUE HANDSHAKE
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, Jul 09, 2009 at 06:32:29PM +0200, Joan Balaguer? Valls wrote:
> Hello Oleg,
>
> Can I set the expect-continue handshake at httpclient level?
>
> HttpParams objHttpParams = new BasicHttpParams();
> HttpProtocolParams.setVersion(objHttpParams, HttpVersion.HTTP_1_1);
> ClientConnectionManager cm = new
> ThreadSafeClientConnManager(this.objHttpParams,
> HttpUtils.createDefaultSchemeRegistry());
>
> this.objHttp = new DefaultHttpClient(cm, this.objHttpParams);
>
> --> this.objHttpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE,
>
> Boolean.FALSE);
>
> Or should I set at request level?
>
You can do both.
Oleg
> Thanks,
>
> Joan.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
EXPECT-CONTINUE HANDSHAKE
Posted by Joan Balagueró Valls <jo...@grupoventus.com>.
Hello Oleg,
Can I set the expect-continue handshake at httpclient level?
HttpParams objHttpParams = new BasicHttpParams();
HttpProtocolParams.setVersion(objHttpParams, HttpVersion.HTTP_1_1);
ClientConnectionManager cm = new
ThreadSafeClientConnManager(this.objHttpParams,
HttpUtils.createDefaultSchemeRegistry());
this.objHttp = new DefaultHttpClient(cm, this.objHttpParams);
--> this.objHttpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE,
Boolean.FALSE);
Or should I set at request level?
Thanks,
Joan.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: performance tuning possibilities?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, Jul 09, 2009 at 05:05:14PM +0200, Michael wrote:
> I completed my migration to the latest version of the HttpClient
> successfully. It is just a bit slower now than I would have ecpected...
>
> Maybe somebody has some tuning experiences to share?
>
Michael,
HttpClient should be _at least_ as fast as HttpClient 3.x. If performance is an
overriding factor for you, you should consider using HttpCore directly and get
30 to 40% performance boost.
http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore
> * Reliability is not that important for me
Make sure to turn off stale connection check and expect-continue handshake.
> * It should do around 20 accesses per minute using slow proxies, connection
> timeout is set to 6 seconds for them. Currently it does those 20 accesses in
> 100 seconds :-(
>
To find out where time is being spent, turn on context logging. Make sure to
disable the wire log, though, otherwise you will be measuring performance of
console output.
http://hc.apache.org/httpcomponents-client/logging.html
>
> * I have 3 threads executing http gets concurrently.
> * I am going with ThreadSafeClientConnManager and 30 connections at max. Is
> that enough? Or too much?
You also have to increase the default limit of 2 concurrent connections per
host. It should be at least the same as the number of worker thread to ensure
minimal resource contension.
> * I use a retry hanlder with up to 6 retries. Not sure if that is eating up
> much time?
> * TCPNoDelay is false. Would turning that on improve performance to some
> extent? How much?
Whooooaaaaa! Nagle's algorithm just kills HTTP performance. Make sure the
TCPNoDelay is set to true.
> * Stale Checking is enabled. Maybe I dont need this since I use the
> IdleConnectionMonitorThread doing connMgr.closeExpiredConnections() every 5
> seconds?
>
The stale connection check is more evil than the Church of Scientology. Just do
not use it.
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org