[nutch-site] branch NUTCH-1999-nutch-site-robots-txt created (now 142489f)

[sn...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

snagel pushed a change to branch NUTCH-1999-nutch-site-robots-txt
in repository https://gitbox.apache.org/repos/asf/nutch-site.git.


      at 142489f  NUTCH-1999 Add /robots.txt to Nutch site

This branch includes the following new commits:

     new 142489f  NUTCH-1999 Add /robots.txt to Nutch site

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[nutch-site] 01/01: NUTCH-1999 Add /robots.txt to Nutch site

[sn...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

snagel pushed a commit to branch NUTCH-1999-nutch-site-robots-txt
in repository https://gitbox.apache.org/repos/asf/nutch-site.git

commit 142489f5f05d955f2140bffa6a82df90cd5cc359
Author: Sebastian Nagel <sn...@apache.org>
AuthorDate: Sun Jan 9 13:53:35 2022 +0100

    NUTCH-1999 Add /robots.txt to Nutch site
---
 content/robots.txt | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/content/robots.txt b/content/robots.txt
new file mode 100644
index 0000000..c2a49f4
--- /dev/null
+++ b/content/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Allow: /