You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Sebastian Nagel (JIRA)" <ji...@apache.org> on 2019/01/22 13:46:00 UTC

[jira] [Created] (NUTCH-2689) Speed up urlfilter-regex and urlfilter-automaton

Sebastian Nagel created NUTCH-2689:
--------------------------------------

             Summary: Speed up urlfilter-regex and urlfilter-automaton
                 Key: NUTCH-2689
                 URL: https://issues.apache.org/jira/browse/NUTCH-2689
             Project: Nutch
          Issue Type: Improvement
          Components: plugin
    Affects Versions: 1.15
            Reporter: Sebastian Nagel
             Fix For: 1.16


The unit tests of urlfilter-regex and urlfilter-automaton include a benchmark. After playing and benchmarking modifications the following changes seem to significantly improve the performance:
- do not extract host and domain name from the URL if not needed (no host/domain-specific rules used, cf. NUTCH-1838)
- use non-capturing groups if possible
- use {{(?i)}} to make the patterns case insensitive and remove uppercase variants 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)