You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Peter Schuller (Created) (JIRA)" <ji...@apache.org> on 2012/03/09 22:08:57 UTC
[jira] [Created] (CASSANDRA-4035) post-effective ownership nodetool
ring returns invalid information in some circumstances
post-effective ownership nodetool ring returns invalid information in some circumstances
----------------------------------------------------------------------------------------
Key: CASSANDRA-4035
URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Peter Schuller
Fix For: 1.1.1
CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
{code}
-10.34.115.115 smf1 ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
-10.35.108.128 smf1 aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
-10.34.244.104 smf1 ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
-10.35.86.129 smf1 ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
+10.35.108.128 smf1 aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
+10.34.244.104 smf1 ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
+10.35.86.129 smf1 ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
{code}
The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
(Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4035) post-effective ownership
nodetool ring returns invalid information in some circumstances
Posted by "Vijay (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226573#comment-13226573 ]
Vijay commented on CASSANDRA-4035:
----------------------------------
Hi Peter, I couldn't reproduce it.
Except if the command is run on the replacing/Hibernating node, If yes then it is not the issue caused by CASSANDRA-3412 but because we dont set the token until the bootstrapping completes this is true for any bootstrapping node.
And bootstrapping node doesn't get reads or writes and hence it is ok. Not sure if we have to fix the way bootstrapping node displays the ring info.
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Vijay
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4035) post-effective ownership
nodetool ring returns invalid information in some circumstances
Posted by "Brandon Williams (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4035:
-------------------------------------------
Assignee: Vijay
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Vijay
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-4035) post-effective
ownership nodetool ring returns invalid information in some circumstances
Posted by "Vijay (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226573#comment-13226573 ]
Vijay edited comment on CASSANDRA-4035 at 3/9/12 11:04 PM:
-----------------------------------------------------------
Hi Peter, I couldn't reproduce it.
Except if the command is run on the replacing/Hibernating node, If yes then it is not the issue caused by CASSANDRA-3412. We dont set the token until the bootstrapping completes this is true for any bootstrapping node.
And bootstrapping node doesn't get reads or writes and hence it is ok. Not sure if we have to fix the way bootstrapping node displays the ring info.
FYI:
I did the following:
1) Took a running cassandra node and restarted it with the replace_token option.
2) Other node I did while true; do sleep 5; nt ring ; done
Address DC Rack Status State Load Effective-Owership Token
132332031580364958013534569558607324497
184.73.86.8 us-east 1c Down Normal 43.56 KB 66.67% 18904575940052136859076367081351254013
174.129.129.173 us-east 1c Up Normal 23.95 MB 66.67% 75618303760208547436305468319979289255
23.20.70.216 us-east 1c Up Normal 411.64 MB 66.67% 132332031580364958013534569558607324497
Address DC Rack Status State Load Effective-Owership Token
132332031580364958013534569558607324497
184.73.86.8 us-east 1c Up Normal 43.56 KB 66.67% 18904575940052136859076367081351254013
174.129.129.173 us-east 1c Up Normal 23.95 MB 66.67% 75618303760208547436305468319979289255
23.20.70.216 us-east 1c Up Normal 435.88 MB 66.67% 132332031580364958013534569558607324497
was (Author: vijay2win@yahoo.com):
Hi Peter, I couldn't reproduce it.
Except if the command is run on the replacing/Hibernating node, If yes then it is not the issue caused by CASSANDRA-3412 but because we dont set the token until the bootstrapping completes this is true for any bootstrapping node.
And bootstrapping node doesn't get reads or writes and hence it is ok. Not sure if we have to fix the way bootstrapping node displays the ring info.
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Vijay
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4035) post-effective ownership
nodetool ring returns invalid information in some circumstances
Posted by "Vijay (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vijay resolved CASSANDRA-4035.
------------------------------
Resolution: Not A Problem
closing it as i couldn't reproduce it, Plz let me know (I can re-open) if you want me to change the bootstrapping behavior.
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Assignee: Vijay
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4035) post-effective ownership nodetool
ring returns invalid information in some circumstances
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-4035:
--------------------------------------
Description:
CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
{code}
-10.34.115.115 rack ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
-10.35.108.128 rack aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
-10.34.244.104 rack ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
-10.35.86.129 rack ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
+10.35.108.128 rack aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
+10.34.244.104 rack ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
+10.35.86.129 rack ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
{code}
The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
(Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
was:
CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
{code}
-10.34.115.115 smf1 ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
-10.35.108.128 smf1 aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
-10.34.244.104 smf1 ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
-10.35.86.129 smf1 ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
+10.35.108.128 smf1 aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
+10.34.244.104 smf1 ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
+10.35.86.129 smf1 ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
{code}
The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
(Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 rack ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 rack aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 rack ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 rack ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 rack aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 rack ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 rack ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4035) post-effective ownership nodetool
ring returns invalid information in some circumstances
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-4035:
--------------------------------------
Description:
CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
{code}
-10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
-10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
-10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
-10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
+10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
+10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
+10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
{code}
The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
(Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
was:
CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
{code}
-10.34.115.115 rack ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
-10.35.108.128 rack aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
-10.34.244.104 rack ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
-10.35.86.129 rack ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
+10.35.108.128 rack aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
+10.34.244.104 rack ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
+10.35.86.129 rack ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
{code}
The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
(Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
> post-effective ownership nodetool ring returns invalid information in some circumstances
> ----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-4035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4035
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Peter Schuller
> Fix For: 1.1.1
>
>
> CASSANDRA-3412 broke something. We had a test cluster that I observed was unbalanced (unexpected because it wasn't supposed to be). Later, it wasn't. We realized a node was being replaced at the time it showed as unbalanced. The diff shows:
> {code}
> -10.34.115.115 dc ael Up Normal 26.32 KB 9.09% 36090554067372261276418518970036022421
> -10.35.108.128 dc aoa Up Normal 24.42 KB 9.09% 41246347505568298601621164537184025624
> -10.34.244.104 dc ajk Up Normal 27.11 KB 9.09% 46402140943764335926823810104332028827
> -10.35.86.129 dc ane Up Normal 31.67 KB 9.09% 51557934381960373252026455671480032030
> +10.35.108.128 dc aoa Up Normal 24.42 KB 12.12% 41246347505568298601621164537184025624
> +10.34.244.104 dc ajk Up Normal 27.11 KB 12.12% 46402140943764335926823810104332028827
> +10.35.86.129 dc ane Up Normal 31.67 KB 12.12% 51557934381960373252026455671480032030
> {code}
> The node that caused this was being replaced (with replace token, not regular bootstrap) into the ring either during this or in relation to this in time. The node was never removed, and if a mistake was made to do regular bootstrap it should be showing up as joining.
> Hypothesis without looking at code: Somehow nodes in HIBERNATE state are incorrectly considered?
> (Marked fix for 1.1.1 because that's the fix-for of effective-ownership.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira