[jira] Updated: (NUTCH-339) Refactor nutch to allow fetcher improvements

["chee.wu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

chee.wu updated NUTCH-339:
--------------------------

    Attachment: Fetcher2 for .81

Orginally, Fetcher2 can't work togehter with Nutch.81,here I provide a new portion for it.
Though some key improvement lik "Move robots.txt handling away from (lib-http)plugin"  is commented out, this new portion  did acheive a 80% speed increasment in my test,comprised of  orginal Fetcher.java in .81.

 

> Refactor nutch to allow fetcher improvements
> --------------------------------------------
>
>                 Key: NUTCH-339
>                 URL: https://issues.apache.org/jira/browse/NUTCH-339
>             Project: Nutch
>          Issue Type: Task
>          Components: fetcher
>    Affects Versions: 0.8
>         Environment: n/a
>            Reporter: Sami Siren
>         Assigned To: Andrzej Bialecki 
>             Fix For: 0.9.0
>
>         Attachments: Fetcher2 for .81, patch.txt, patch2.txt, patch3.txt, patch4-fixed.txt, patch4-trunk.txt
>
>
> As I (and Stefan?) see it there are two major areas the current fetcher could be
> improved (as in speed)
> 1. Politeness code and how it is implemented is the biggest
> problem of current fetcher(together with robots.txt handling).
> With a simple code changes like replacing it with a PriorityQueue
> based solution showed very promising results in increased IO.
> 2. Changing fetcher to use non blocking io (this requires great amount
> of work as we need to implement the protocols from scratch again).
> I would like to start with working towards #1 by first refactoring
> the current code (plugins actually) in following way:
> 1. Move robots.txt handling away from (lib-http)plugin.
> Even if this is related only to http, leaving it to lib-http
> does not allow other kinds of scheduling strategies to be implemented
> (it is hardcoded to fetch robots.txt from the same thread when requesting
> a page from a site from witch it hasn't tried to load robots.txt)
> 2. Move code for politeness away from (lib-http)plugin
> It is really usable outside http and also the current design limits
> changing of the implementation (to queue based)
> Where to move these, well my suggestion is the nutch core, does anybody
> see problems with this?
> These code refactoring activities are to be done in a way that none
> of the current functionality is (at least deliberately) changed leaving
> current functionality as is thus leaving room and possibility to build
> the next generation fetcher(s) without destroying the old one at same time.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.