You are viewing a plain text version of this content. The canonical link for it is here.
Posted to infrastructure-issues@apache.org by "Jeff Turner (JIRA)" <ji...@apache.org> on 2009/03/20 03:37:50 UTC
[jira] Created: (INFRA-1945) Allow crawling of JIRA issues
Allow crawling of JIRA issues
-----------------------------
Key: INFRA-1945
URL: https://issues.apache.org/jira/browse/INFRA-1945
Project: Infrastructure
Issue Type: Improvement
Security Level: public (Regular issues)
Reporter: Jeff Turner
>From email:
A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
$ curl http://issues.apache.org/robots.txt
User-agent: *
Disallow: /jira
Disallow: /scarab
Disallow: /bugzilla
Disallow: /SpamAssassin
Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
Disallow: /jira/sr/
Disallow: /jira/si/
Disallow: /jira/charts
--Jeff
Tony replied:
Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
Cheers,
Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
Posted by "#asfinfra IRC Bot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683733#action_12683733 ]
#asfinfra IRC Bot commented on INFRA-1945:
------------------------------------------
<jefft> Made the suggested change for all the JIRA prefixes
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (INFRA-1945) Allow crawling of JIRA issues
Posted by "Jeff Turner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Turner reassigned INFRA-1945:
----------------------------------
Assignee: Jeff Turner
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
Posted by "#asfinfra IRC Bot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683728#action_12683728 ]
#asfinfra IRC Bot commented on INFRA-1945:
------------------------------------------
<pquerna> also, we should let google hit bugzilla.
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (INFRA-1945) Allow crawling of JIRA issues
Posted by "#asfinfra IRC Bot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
#asfinfra IRC Bot closed INFRA-1945.
------------------------------------
Resolution: Fixed
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (INFRA-1945) Allow crawling of JIRA issues
Posted by "#asfinfra IRC Bot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/INFRA-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12683727#action_12683727 ]
#asfinfra IRC Bot commented on INFRA-1945:
------------------------------------------
<pquerna> what about the other jira instances? shouldn't they have the same disallow rules?
> Allow crawling of JIRA issues
> -----------------------------
>
> Key: INFRA-1945
> URL: https://issues.apache.org/jira/browse/INFRA-1945
> Project: Infrastructure
> Issue Type: Improvement
> Security Level: public(Regular issues)
> Reporter: Jeff Turner
> Assignee: Jeff Turner
>
> From email:
> A coworker noticed that one gets crappy results for searches like http://www.google.com.au/search?q=DAEMON-66 because our robots.txt disallows crawling:
> $ curl http://issues.apache.org/robots.txt
> User-agent: *
> Disallow: /jira
> Disallow: /scarab
> Disallow: /bugzilla
> Disallow: /SpamAssassin
> Any reason we can't allow crawling of JIRA? It should be able to handle the load if crawlers are kept away from saved searches:
> Disallow: /jira/sr/
> Disallow: /jira/si/
> Disallow: /jira/charts
> --Jeff
> Tony replied:
> Jeff once upon a time, we banned the used of crawling on a lot of sites as they absolutely caned the hosts they were indexing. I think this is a little less relaxed now, so if no one has any issues in the next day or so what I suggest you do, is either make the change yourself, if you can, or open a new JIRA with a patch and we'll do it for you.
> Cheers,
> Tony
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.