You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@nutch.apache.org by sn...@apache.org on 2022/01/09 12:58:00 UTC

[nutch-site] 01/01: NUTCH-1999 Add /robots.txt to Nutch site

This is an automated email from the ASF dual-hosted git repository.

snagel pushed a commit to branch NUTCH-1999-nutch-site-robots-txt
in repository https://gitbox.apache.org/repos/asf/nutch-site.git

commit 142489f5f05d955f2140bffa6a82df90cd5cc359
Author: Sebastian Nagel <sn...@apache.org>
AuthorDate: Sun Jan 9 13:53:35 2022 +0100

    NUTCH-1999 Add /robots.txt to Nutch site
---
 content/robots.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content/robots.txt b/content/robots.txt
new file mode 100644
index 0000000..c2a49f4
--- /dev/null
+++ b/content/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Allow: /