[xalan-site] 08/36: Add robots.txt to limit web crawlers access to specific content

[gg...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

ggregory pushed a commit to branch git-svn
in repository https://gitbox.apache.org/repos/asf/xalan-site.git

commit 860a77fee80c923ffe08c08992fe61c40b5f96a3
Author: Steven J. Hathaway <sh...@apache.org>
AuthorDate: Thu Jul 12 04:19:49 2012 +0000

    Add robots.txt to limit web crawlers access to specific content
---
 docs/xalan/robots.txt | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/docs/xalan/robots.txt b/docs/xalan/robots.txt
new file mode 100644
index 0000000..04adbe8
--- /dev/null
+++ b/docs/xalan/robots.txt
@@ -0,0 +1,10 @@
+# 
+# Xalan website control: robots.txt
+#
+# Keep web crawlers out of auto-generated api documentation pages
+#
+User-agent: *
+Disallow: /xalan-c/apiDocs/
+Disallow: /xalan-j/apidocs/
+Disallow: /old/xalan-c/apiDocs/
+Disallow: /old/xalan-j/apidocs/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@xalan.apache.org
For additional commands, e-mail: dev-help@xalan.apache.org