[jira] [Created] (NUTCH-2527) URL filter: provide rules to exclude localhost and private address spaces

["Sebastian Nagel (JIRA)" <ji...@apache.org>]

Sebastian Nagel created NUTCH-2527:
--------------------------------------

             Summary: URL filter: provide rules to exclude localhost and private address spaces
                 Key: NUTCH-2527
                 URL: https://issues.apache.org/jira/browse/NUTCH-2527
             Project: Nutch
          Issue Type: Improvement
    Affects Versions: 1.14, 2.3.1
            Reporter: Sebastian Nagel
             Fix For: 2.4, 1.15


While checking the log files of a large web crawl, I've found hundreds of (luckily failed) requests of local or private content:
{noformat}
2018-02-18 04:48:34,022 INFO [FetcherThread] org.apache.nutch.fetcher.Fetcher: fetching http://127.0.0.42/ ...
018-02-18 04:48:34,022 INFO [FetcherThread] org.apache.nutch.fetcher.Fetcher: fetch of http://127.0.0.42/ failed with: java.net.ConnectException: Connection refused (Connection refused)
{noformat}

URLs pointing to localhost, loop-back addresses, private address spaces should be blocked for a wider web crawl where links are not controlled to avoid that information is leaked by links or redirects pointing to web interfaces of services running on the crawling machines (e.g., HDFS, Hadoop YARN).

Of course, this must be optional. For testing it's quite common to crawl your local machine.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)