[jira] Resolved: (INFRA-1578) Allow GoogleCodeBot in robots.txt

["Paul Querna (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/INFRA-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Querna resolved INFRA-1578.
--------------------------------

    Resolution: Fixed

Robots.txt has been updated.

Raw Subversion Repo is now open.

Viewvc and Snapshot directories are still blocked.

> Allow GoogleCodeBot in robots.txt
> ---------------------------------
>
>                 Key: INFRA-1578
>                 URL: https://issues.apache.org/jira/browse/INFRA-1578
>             Project: Infrastructure
>          Issue Type: Wish
>      Security Level: public(Regular issues) 
>          Components: Website
>            Reporter: Mike Aizatsky
>
> Hello,
> We, at google, has received quite a few complaints about Apache
> software source code being unavailable on Google Code Search
> (http://www.google.com/codesearch). We've investigated the issue, and
> found that you have a robots.txt file disallowing even our special
> google code crawlers (http://svn.apache.org/robots.txt):
> User-agent: *
> Disallow: /
> We do believe this was done to tell usual web crawlers to stay away
> from your svn repositories, but we have a custom,
> svn-interface-conformant crawler in codesearch. Can you relax your
> robots.txt for us and allow "GoogleCodeBot" to index your site? Or if
> you're reluctant to change your file, can you just confirm that we're
> free to index your source code?
> --
> Regards,
> Mike

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.