[jira] Commented: (INFRA-1343) setup robots.txt and/or other access rules to prevent bots from crawling Continuum pages

["#asfinfra IRC Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/INFRA-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715395#action_12715395 ] 

#asfinfra IRC Bot commented on INFRA-1343:
------------------------------------------

<gmcdonald> I dont the need for a bot to crawl anywhere on vmbuild, so should we just set disallow=* on all of it?


> setup robots.txt and/or other access rules to prevent bots from crawling Continuum pages 
> -----------------------------------------------------------------------------------------
>
>                 Key: INFRA-1343
>                 URL: https://issues.apache.org/jira/browse/INFRA-1343
>             Project: Infrastructure
>          Issue Type: Task
>      Security Level: public(Regular issues) 
>          Components: Continuum
>            Reporter: Brett Porter
>
> We don't need search engines crawling the build pages (especially since it can navigate its way all the way through a working copy). It is picking up links from the mails sent out to mailing lists, presumably.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.