You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Claudio Martella <cl...@tis.bz.it> on 2011/01/12 11:47:18 UTC

Moving Nutch to httpclient 4

Hi,

trying to find the bug in nutch/httpclient auth methods, I had a
discussion with httpclient's developers and one of them underlined
httpclient 3.x being EOL.
Is there any plan/effort about porting nutch to httpclient 4.x? Should I
file a JIRA?


Best
Claudio

-------- Original Message --------
Subject: 	Re: httpclient 3.1 failing DIGEST authentication
Date: 	Wed, 5 Jan 2011 17:00:41 +0100
From: 	Oleg Kalnichevski <ol...@apache.org>
Reply-To: 	HttpClient User Discussion <ht...@hc.apache.org>
To: 	HttpClient User Discussion <ht...@hc.apache.org>


[...]

HttpClient 3.1 is EOL and is no longer supported. You may want to raise
this as a change request with developers of Nutch and ask them to
consider upgrading to HttpClient 4.x or 4.1.x 

Oleg




Re: Moving Nutch to httpclient 4

Posted by Alexis <al...@gmail.com>.
Hi,

I agree with you, the HTTP protocol layer and the Fetcher Job in Nutch
can be upgraded.

The web is becoming huge. You want to optimize the way web pages are downloaded.

I was interested in making it faster, yet still being respectful of
the websites being crawled, by making sure you hold a single
connection per host and maybe by forcing some throttling. The latter
is usually already done automatically on the server side. It would
need to shift from a multi-threaded model to a single-threaded one,
leveraging Non blocking IO.

This can be done with the new httpcore-nio jar from HttpComponents,
instead of httpclient.

Please see http://techvineyard.blogspot.com/2010/11/java-httpcomponents.html
for more information. The chart in the conclusion shows that the
asynchronous approach is faster than the synchronous one, which used
to be the only one available in Java before java.nio package was
introduced.

Alexis

On Wed, Jan 12, 2011 at 12:00 PM, Julien Nioche
<li...@gmail.com> wrote:
> Am pretty sure there is already one for this. IIRC this was more than an
> upgrade of the underlying library and there was a substantial amount of
> modifications to do on the plugin, which is why it has not been done so far
>
> On 12 January 2011 10:47, Claudio Martella <cl...@tis.bz.it>wrote:
>
>> Hi,
>>
>> trying to find the bug in nutch/httpclient auth methods, I had a
>> discussion with httpclient's developers and one of them underlined
>> httpclient 3.x being EOL.
>> Is there any plan/effort about porting nutch to httpclient 4.x? Should I
>> file a JIRA?
>>
>>
>> Best
>> Claudio
>>
>> -------- Original Message --------
>> Subject:        Re: httpclient 3.1 failing DIGEST authentication
>> Date:   Wed, 5 Jan 2011 17:00:41 +0100
>> From:   Oleg Kalnichevski <ol...@apache.org>
>> Reply-To:       HttpClient User Discussion <httpclient-users@hc.apache.org
>> >
>> To:     HttpClient User Discussion <ht...@hc.apache.org>
>>
>>
>> [...]
>>
>> HttpClient 3.1 is EOL and is no longer supported. You may want to raise
>> this as a change request with developers of Nutch and ask them to
>> consider upgrading to HttpClient 4.x or 4.1.x
>>
>> Oleg
>>
>>
>>
>>
>
>
> --
> *
> *Open Source Solutions for Text Engineering
>
> http://digitalpebble.blogspot.com/
> http://www.digitalpebble.com
>

Re: Moving Nutch to httpclient 4

Posted by Julien Nioche <li...@gmail.com>.
Am pretty sure there is already one for this. IIRC this was more than an
upgrade of the underlying library and there was a substantial amount of
modifications to do on the plugin, which is why it has not been done so far

On 12 January 2011 10:47, Claudio Martella <cl...@tis.bz.it>wrote:

> Hi,
>
> trying to find the bug in nutch/httpclient auth methods, I had a
> discussion with httpclient's developers and one of them underlined
> httpclient 3.x being EOL.
> Is there any plan/effort about porting nutch to httpclient 4.x? Should I
> file a JIRA?
>
>
> Best
> Claudio
>
> -------- Original Message --------
> Subject:        Re: httpclient 3.1 failing DIGEST authentication
> Date:   Wed, 5 Jan 2011 17:00:41 +0100
> From:   Oleg Kalnichevski <ol...@apache.org>
> Reply-To:       HttpClient User Discussion <httpclient-users@hc.apache.org
> >
> To:     HttpClient User Discussion <ht...@hc.apache.org>
>
>
> [...]
>
> HttpClient 3.1 is EOL and is no longer supported. You may want to raise
> this as a change request with developers of Nutch and ask them to
> consider upgrading to HttpClient 4.x or 4.1.x
>
> Oleg
>
>
>
>


-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com