You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by "Eli Reisman (JIRA)" <ji...@apache.org> on 2012/07/11 23:20:35 UTC
[jira] [Created] (GIRAPH-247) Introduce edge based partitioning for
InputSplits
Eli Reisman created GIRAPH-247:
----------------------------------
Summary: Introduce edge based partitioning for InputSplits
Key: GIRAPH-247
URL: https://issues.apache.org/jira/browse/GIRAPH-247
Project: Giraph
Issue Type: Improvement
Components: graph
Affects Versions: 0.2.0
Reporter: Eli Reisman
Assignee: Eli Reisman
Priority: Minor
Fix For: 0.2.0
Attachments: GIRAPH-247-1.patch
Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412952#comment-13412952 ]
Eli Reisman commented on GIRAPH-247:
------------------------------------
Good point, I'll look at that. I wanted to be sure the count was per-partition and was not seeing a clean way to get that from a fresh new partition without calling that. Perhaps the VertexEdgeCount would be available at that spot in the code, let me take another look. I will look at GIRAPH-249 too, finding a way to lower the memory profile per-worker is of great interest to me.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-247) Introduce edge based partitioning for
InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-247:
-------------------------------
Attachment: GIRAPH-247-3.patch
just a quick change to the patch to make it apply more nicely with GIRAPH-246 if you want to try them both (I recommend it) :)
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch, GIRAPH-247-2.patch, GIRAPH-247-3.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413235#comment-13413235 ]
Eli Reisman commented on GIRAPH-247:
------------------------------------
we might want a Configuration command line switch to choose to do this or use the vertex-centered way. Perhaps some data does not benefit from this sort of optimization. But my results were dramatic, and I think with big highly connected graphs it will prove very useful and reduce load on the system on a per-worker basis quite well. Thats what Ive seen in testing so far.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch, GIRAPH-247-2.patch, GIRAPH-247-3.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13413024#comment-13413024 ]
Alessandro Presta commented on GIRAPH-247:
------------------------------------------
Looks good to me. Let's see what the committers say.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch, GIRAPH-247-2.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414474#comment-13414474 ]
Eli Reisman commented on GIRAPH-247:
------------------------------------
UPDATE: I'm working on another issue (to be posted soon) that has given me some insight into this issue. This fix cannot delumpify the graph entirely, as it turns out that each of these Partition objects are sent to their new home when flushed as a collection of vertices to be simple combined at the other end (including locally) with an existing partition object of the same owner Id.
This means, even if you set the maxEdgesPerPartition or maxVerticesPerPartition very low, you are merely altering the granularity of the network messages (groups of vertices) being sent to their new owners after being read from InputSplits at a given local worker.
It turns out this is still very helpful, but has given me insight into why Netty fails when I ratchet up the maximum edges to what seems like a more reasonable default, and why it seems to thrive when I set it to my original lower setting (100k edges per set of vertices sent out over the wire)
In conclusion: this patch is still a very valuable way to regulate the size of outgoing groups of vertices on their way across the wire to a new home during INPUT_SUPERSTEP, but the default should be lower (perhaps even 50k total edges per message) and it further validates why checking for # of edges and then # of vertices in a collection that is to be sent out will catch oversizes messages will catch oversized messages before they get too big much more effectively than the old "10k vertices or more" check in the existing codebase.
I will upload an updated patch with a lower default that is more in keeping with the cluster metrics I'm getting in terms of optimal time and memory use during INPUT_SUPERSTEP given this new insight. I think the patch (which not doing what the title claims) is still very useful and effective for the reasons I've outlined here.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch, GIRAPH-247-2.patch, GIRAPH-247-3.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412974#comment-13412974 ]
Eli Reisman commented on GIRAPH-247:
------------------------------------
I'm going to try using the edge count coming off that round of the vertexReader and just clear my data after each partition is built. If this works I think we can avoid iteration inside the Partition.getEdgeCount() completely, good save Allesandro, thanks!
Will upload new patch ASAP
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-247) Introduce edge based partitioning for
InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-247:
-------------------------------
Attachment: GIRAPH-247-2.patch
New patch uses edge count accumulated locally rather than iterate through all edges in Partition to decide if its full and should be sent to its owner. Thanks to Allesandro, this should be MUCH faster than patch 1.
The patch itself has proven in metered load tests on cluster to be very useful in de-lumpifying large social graph data.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch, GIRAPH-247-2.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-247) Introduce edge based partitioning
for InputSplits
Posted by "Alessandro Presta (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412637#comment-13412637 ]
Alessandro Presta commented on GIRAPH-247:
------------------------------------------
Really good idea and simple implementation.
I'm experimenting with moving whole partitions out of core when the memory is low, and this would make the size of a partition more predictable.
Does the input split take considerably longer? If so, you could try replacing the call to getEdgeCount() (which iterates over all vertices) with your own counter.
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-247) Introduce edge based partitioning for
InputSplits
Posted by "Eli Reisman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/GIRAPH-247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Reisman updated GIRAPH-247:
-------------------------------
Attachment: GIRAPH-247-1.patch
> Introduce edge based partitioning for InputSplits
> -------------------------------------------------
>
> Key: GIRAPH-247
> URL: https://issues.apache.org/jira/browse/GIRAPH-247
> Project: Giraph
> Issue Type: Improvement
> Components: graph
> Affects Versions: 0.2.0
> Reporter: Eli Reisman
> Assignee: Eli Reisman
> Priority: Minor
> Labels: patch
> Fix For: 0.2.0
>
> Attachments: GIRAPH-247-1.patch
>
>
> Experiments on larger data input sets while maintaining low memory profile has revealed that typical social graph data is very lumpy and partitioning by vertices can easily overload some unlucky worker nodes who end up with partitions containing highly-connected vertices while other nodes process partitions with the same number of vertices but far fewer out-edges per vertex. This often results in cascading failures during data load-in even on tiny data sets.
> By partitioning using edges (the default I set in GiraphJob.MAX_EDGES_PER_PARTITION_DEFAULT is 200,000 per partition, or the old default # of vertices, whichever the user's input format reaches first when reading InputSplits) I have seen dramatic "de-lumpification" of data, allow the processing of 8x larger data sets before memory problems occur at a given configuration setting.
> This needs more tuning, but comes with a -Dgiraph.maxEdgesPerPartition that can be set to more edges/partition as your data sets grow or memory limitations shrink. This might be considered a first attempt, perhaps simply allowing us to default to this type of partitioning or the old version would be more compatible with existing users' needs? That would not be a hard feature to add to this. But I think this method of partition production has merit for typical large-scale graph data that Giraph is designed to process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira