You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-issues@apache.org by "#asfinfra IRC Bot (JIRA)" <ji...@apache.org> on 2009/03/20 03:45:50 UTC
[jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683727#action_12683727 ]
#asfinfra IRC Bot commented on INFRA-1945:
------------------------------------------
<pquerna> what about the other jira instances? shouldn't they have the same disallow rules?
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.