[jira] [Commented] (NUTCH-1531) URL filtering takes long time for very long URLs

["Julien Nioche (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577479#comment-13577479 ] 

Julien Nioche commented on NUTCH-1531:
--------------------------------------

See also [NUTCH-1314] and [NUTCH-1106]. Your approach would work but ideally we would like to filter on length regardless of whether the regex-based filter is used or not. [NUTCH-1314] and [NUTCH-1106] have similar shortcomings. The best approach would maybe to have that as a new plugin (urlfilter-basic) in the same way as we have a BasicURLNormalizer and add it to the list of plugins loaded by default. Can't remember whether the URLFilters are typically called before the URLNormalizers but assuming they are this would also speed up the normalisation.

Longer term we want to merge the normalization and filtering into a single logical step.
                
> URL filtering takes long time for very long URLs
> ------------------------------------------------
>
>                 Key: NUTCH-1531
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1531
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.6, 2.1, 1.7, 2.2
>            Reporter: Fırat KÜÇÜK
>            Priority: Minor
>             Fix For: 1.7, 2.2
>
>         Attachments: max_url_length.diff, test_case.txt
>
>
> Filtering very long urls (such as base64 image generators) take long time (hours). On reducing phase it locks down all the system for hours. Therefore some URL length limitation needed. We attached a little patch for this improvement.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira