You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by og...@yahoo.com on 2008/04/18 06:27:44 UTC

protocol-http vs. -httpclient, HTTP 1.1 vs 1.0

Hi,

Has anyone done any performance testing with protocol-http vs. protocol-httpclient?

I just switched from protocol-http to protocol-httpclient and I switched from useHttp11=false to useHttp11=true and I *think* this resulted in higher download speed.

I also upgraded httpclient from 3.0.1 (which is in the trunk) to 3.1 (which is the latest stable 3.* release) and things seem to be working just fine.

Otis 

--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



Re: protocol-http vs. -httpclient, HTTP 1.1 vs 1.0

Posted by Andrzej Bialecki <ab...@getopt.org>.
Doğacan Güney wrote:

> That being said, I remember reading that it is better to keep a connection
> open (by using HTTP/1.1) than opening a new connection for each request. I
> am not sure, but I think it is also easier on the server too....

This is not as simple as it looks. We can reuse a connection if we don't 
open too many connections at the same time ... if a fetchlist consists 
of 64k unique hosts you will have a hard time opening that many sockets! :)

So, the benefit from using keepalive connections is mitigated by the 
fact that we need to limit the number of open sockets on our side. Also, 
consecutive requests to the same host may be separated by a long wait 
(due to all fetcher threads being busy fetching other stuff) - from the 
point of view of resource utilization, is it still better then to keep 
the connection open?

>> I also upgraded httpclient from 3.0.1 (which is in the trunk) to 3.1
>> (which is the latest stable 3.* release) and things seem to be working just
>> fine.
> 
> 
> It may be a good idea to update all external libraries before we release 1.0
> if they are stable.

I agree.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: protocol-http vs. -httpclient, HTTP 1.1 vs 1.0

Posted by Doğacan Güney <do...@gmail.com>.
Hi,

On Fri, Apr 18, 2008 at 7:27 AM, <og...@yahoo.com> wrote:

> Hi,
>
> Has anyone done any performance testing with protocol-http vs.
> protocol-httpclient?
>
> I just switched from protocol-http to protocol-httpclient and I switched
> from useHttp11=false to useHttp11=true and I *think* this resulted in higher
> download speed.


I remember changing to protocol-httpclient a while back, but I didn't notice
any speed-up. But then again, it is just too damn difficult to measure
download speed because it is not a controlled experiment (maybe when you
switch to httpclient, people start hammering those servers you are about to
fetch from, etc)

That being said, I remember reading that it is better to keep a connection
open (by using HTTP/1.1) than opening a new connection for each request. I
am not sure, but I think it is also easier on the server too....


>
>
> I also upgraded httpclient from 3.0.1 (which is in the trunk) to 3.1
> (which is the latest stable 3.* release) and things seem to be working just
> fine.


It may be a good idea to update all external libraries before we release 1.0
if they are stable.


>
>
> Otis
>
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>


-- 
Doğacan Güney