[jira] Commented: (INFRA-1578) Allow GoogleCodeBot in robots.txt

["Henri Yandell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/INFRA-1578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587043#action_12587043 ] 

Henri Yandell commented on INFRA-1578:
--------------------------------------

+1 to add it to the robots.txt. If it sucks bandwidth we can kick em out later.

> Allow GoogleCodeBot in robots.txt
> ---------------------------------
>
>                 Key: INFRA-1578
>                 URL: https://issues.apache.org/jira/browse/INFRA-1578
>             Project: Infrastructure
>          Issue Type: Wish
>      Security Level: public(Regular issues) 
>          Components: Website
>            Reporter: Mike Aizatsky
>
> Hello,
> We, at google, has received quite a few complaints about Apache
> software source code being unavailable on Google Code Search
> (http://www.google.com/codesearch). We've investigated the issue, and
> found that you have a robots.txt file disallowing even our special
> google code crawlers (http://svn.apache.org/robots.txt):
> User-agent: *
> Disallow: /
> We do believe this was done to tell usual web crawlers to stay away
> from your svn repositories, but we have a custom,
> svn-interface-conformant crawler in codesearch. Can you relax your
> robots.txt for us and allow "GoogleCodeBot" to index your site? Or if
> you're reluctant to change your file, can you just confirm that we're
> free to index your source code?
> --
> Regards,
> Mike

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.