[jira] Commented: (INFRA-1106) robots.txt on wiki.apache.org blocks Google (again)

["Paul Querna (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/INFRA-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463644 ] 

Paul Querna commented on INFRA-1106:
------------------------------------

It was done in the past due to server load caused by the wiki.  Now that the wiki is on the t2000, we can likely take it off.

> robots.txt on wiki.apache.org blocks Google (again)
> ---------------------------------------------------
>
>                 Key: INFRA-1106
>                 URL: https://issues.apache.org/jira/browse/INFRA-1106
>             Project: Infrastructure
>          Issue Type: Bug
>      Security Level: public(Regular issues) 
>          Components: Wiki
>            Reporter: Justin Mason
>
> A couple of years ago, https://issues.apache.org/jira/browse/INFRA-361 was marked "fixed", allowing Google to spider the wiki sites.
> However, while searching for some Spamassassin documnetation today, I noticed that virtually no wiki hits were appearing -- especially odd since our documentation and FAQ is primarily wiki-based.
> A check of http://wiki.apache.org/robots.txt reveals the cause:
> User-agent: *
> Disallow: /
> Can this be fixed?  We rely on our user population being able to find documentation and FAQs via google. Should we move this from ASF infrastructure if there are persistent load problems?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira