[jira] [Updated] (NUTCH-1741) Support of Sitemaps in Nutch 2.x

["Alparslan Avcı (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NUTCH-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alparslan Avcı updated NUTCH-1741:
----------------------------------

    Attachment: NUTCH-1741.patch

I have uploaded a patch that implements the sitemap support as similar to the description in [^SitemapDevelopmentFor2x.pdf]. The sitemap crawl cycle consist of 4 steps: 
# _SitemapInjectorJob_ -> This job can inject sitemap urls from robots files of hosts in _HostDb_ or from a seed list.
# _GeneratorJob_ -> Standard _GeneratorJob_. It has to be run with _-sitemap_ argument.
# _FetcherJob_ -> Standard _FetcherJob_. (I suggest to set _http.content.limit_ (and/or _file.content.limit_) property to _52428800_, since sitemaps can be large as 50MB)
# _SitemapParserJob_ -> This job parses the sitemaps with specified parse plugins. As now, I have implemented only one plugin (_parse-sitemap_) that uses _crawler-commons_ as parser. After parse, the job writes the new urls to webpage table. 

Waiting your comments! :)

> Support of Sitemaps in Nutch 2.x
> --------------------------------
>
>                 Key: NUTCH-1741
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1741
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher, generator
>            Reporter: Alparslan Avcı
>             Fix For: 2.3
>
>         Attachments: NUTCH-1741.patch, SitemapDevelopmentFor2x.pdf
>
>
> Sitemap support has to be implemented for 2.x branch. It is being discussed in NUTCH-1465 for trunk. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)