You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by "Sebastian Nagel (Jira)" <ji...@apache.org> on 2020/07/11 12:05:00 UTC

[jira] [Created] (NUTCH-2807) SitemapProcessor to warn that ignoring robotst.xt affects detection of sitemaps

Sebastian Nagel created NUTCH-2807:
--------------------------------------

             Summary: SitemapProcessor to warn that ignoring robotst.xt affects detection of sitemaps
                 Key: NUTCH-2807
                 URL: https://issues.apache.org/jira/browse/NUTCH-2807
             Project: Nutch
          Issue Type: Improvement
          Components: robots, sitemap
            Reporter: Sebastian Nagel
             Fix For: 1.18


Ignoring the robots.txt causes as a site effect that no sitemaps can be detected via robots.txt.

SitemapProcessor should log a warning if robots.txt is ignored by configuration (NUTCH-1927/NUTCH-2803).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)