You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Otis Gospodnetic (JIRA)" <ji...@apache.org> on 2013/11/27 06:00:36 UTC

[jira] [Commented] (NUTCH-1661) Language based crawling

    [ https://issues.apache.org/jira/browse/NUTCH-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13833459#comment-13833459 ] 

Otis Gospodnetic commented on NUTCH-1661:
-----------------------------------------

Can you describe this a bit more?  So this assumes the page was fetched once, its language was identified and stored, and the next time Nutch considers fetching this page the page may be rejected/excluded from fetching based on its language?

There is a loop in the code that may be a little nicer/faster if converted to a Set lookup.


> Language based crawling
> -----------------------
>
>                 Key: NUTCH-1661
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1661
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 2.2.1
>            Reporter: Talat UYARER
>            Priority: Minor
>              Labels: PatchAvailable
>             Fix For: 2.3
>
>         Attachments: NUTCH-1661.patch
>
>
> If you want to crawling based on specific language, Nutch has not any solution. I created a custom Fetch Scheduler. It controls page's language  



--
This message was sent by Atlassian JIRA
(v6.1#6144)