You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2020/07/13 15:45:20 UTC

[GitHub] [helix] lei-xia opened a new pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

lei-xia opened a new pull request #1111:
URL: https://github.com/apache/helix/pull/1111


   ### Issue
   
   - [X] My PR addresses the following Helix issues and references them in the PR description:
   
   (#1128 - https://github.com/apache/helix/issues/1128)
   "Add metrics to report the number of messages that have not been completed after a certain of time #1128"
   
   ### Description
   
   Add Cluster and instance level metrics to report the number of messages that have not been completed after their expected completion time.
   
   Changes:
   1) Add expected completion time field to Message, set the default expected completion period to be 1 day (can be configurable via system property).
   
   2) Add Past-Due incomplete message gauge in both cluster and instance monitor.
   
   ### Tests
   
   - [X] The following tests are written for this issue:
    
   TestClusterStatusMonitor. testMessageMetrics
   
   - [X] The following is the result of the "mvn test" command on the appropriate module:
   
   [INFO]
   [INFO] Results:
   [INFO]
   [ERROR] Failures:
   [ERROR]   TestWagedRebalance.testChangeIdealState:271->validate:614 expected:<true> but was:<false>
   [ERROR]   TestClusterVerifier.testResourceSubset:225 expected:<false> but was:<true>
   [INFO]
   [ERROR] Tests run: 1146, Failures: 2, Errors: 0, Skipped: 1
   [INFO]
   [INFO] ------------------------------------------------------------------------
   [INFO] Reactor Summary:
   [INFO]
   [INFO] Apache Helix ....................................... SUCCESS [  0.626 s]
   [INFO] Apache Helix :: Metrics Common ..................... SUCCESS [  2.530 s]
   [INFO] Apache Helix :: Metadata Store Directory Common .... SUCCESS [ 14.477 s]
   [INFO] Apache Helix :: ZooKeeper API ...................... SUCCESS [ 56.517 s]
   [INFO] Apache Helix :: Helix Common ....................... SUCCESS [  1.101 s]
   [INFO] Apache Helix :: Core ............................... FAILURE [  01:17 h]
   [INFO] Apache Helix :: Admin Webapp ....................... SKIPPED
   [INFO] Apache Helix :: Restful Interface .................. SKIPPED
   [INFO] Apache Helix :: Distributed Lock ................... SKIPPED
   [INFO] Apache Helix :: HelixAgent ......................... SKIPPED
   [INFO] Apache Helix :: Recipes ............................ SKIPPED
   [INFO] Apache Helix :: Recipes :: Rabbitmq Consumer Group . SKIPPED
   [INFO] Apache Helix :: Recipes :: Rsync Replicated File Store SKIPPED
   [INFO] Apache Helix :: Recipes :: distributed lock manager  SKIPPED
   [INFO] Apache Helix :: Recipes :: distributed task execution SKIPPED
   [INFO] Apache Helix :: Recipes :: service discovery ....... SKIPPED
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD FAILURE
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time: 01:18 h
   [INFO] Finished at: 2020-06-28T21:07:23-07:00
   [INFO] Final Memory: 72M/1420M
   
   ### Commits
   
   - [X] My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - [X] My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] jiajunwang commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
jiajunwang commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r447361253



##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ClusterStatusMonitor.java
##########
@@ -289,6 +288,32 @@ public void setClusterInstanceStatus(Set<String> liveInstanceSet, Set<String> in
           }
         }
       }
+
+      // update all message related gauges

Review comment:
       nit, can we organically add the new logic to the existing one. As you can see the logic is,
   1. ensure instance mbeans created
   2. do the metric data calculation
   3. update the metrics
   
   Can we split the new logic into 2 parts and put the logic to the corresponding sections 2 and 3?
   1. We have some check regarding the monitor objects there.
   2. We want to keep the structure of the code.

##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ClusterStatusMonitor.java
##########
@@ -881,7 +910,7 @@ public String clusterBeanName() {
    * @param instanceName
    * @return instance bean name
    */
-  private String getInstanceBeanName(String instanceName) {
+  public String getInstanceBeanName(String instanceName) {

Review comment:
       I don't have strong preference here.
   - Shall protected or package-private enough here?
   - Or, on the other hand, with these two methods made to the public, we can avoid several test codes using INSTANCE_DN_KEY and RESOURCE_DN_KEY directly. Not a must in this PR, though.

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -522,12 +538,25 @@ public long getExecuteStartTimeStamp() {
 
   /**
    * Get the time that this message was created
-   * @return UNIX timestamp
+   * @return UNIX epoch timestamp
    */
   public long getCreateTimeStamp() {
     return _record.getLongField(Attributes.CREATE_TIMESTAMP.toString(), 0L);
   }
 
+  /**
+   * Get the time that the message was expected to be completed
+   * @return UNIX epoch timestamp
+   */
+  public long getCompletionDueTimeStamp() {
+    long completionDue = _record.getLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), 0L);
+    if (completionDue == 0) {
+      completionDue = getCreateTimeStamp() + MESSAGE_EXPECT_COMPLETION_PERIOD;
+    }

Review comment:
       To be safe, shall we define a constant COMPLETION_DUE_TIMESTAMP_NOT_SET = -1 for the default value?

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -173,6 +181,14 @@ public void setCreateTimeStamp(long timestamp) {
     _record.setLongField(Attributes.CREATE_TIMESTAMP.toString(), timestamp);
   }
 
+  /**
+   * Set the time that the message was expected to be completed
+   * @param timestamp a UNIX epoch timestamp
+   */
+  public void setCompletionDueTimeStamp(long timestamp) {
+    _record.setLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), timestamp);

Review comment:
       Is this method set by the controller or the participant?
   
   If controller, can we just update it when set the create time? So we don't need to consider validating.
   Or, if the messages are created with different expected run time, then we can add the expected run time as the parameter. That logic would easier to justify (and easier to valid) compared with passing the exact due timestamp.
   
   And I am thinking that it might be better if the participant updates it. Since 1. participant knows the estimation better. 2. if the participant has not read the message yet, then we probably don't want to mark it as a past-due message.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia merged pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia merged pull request #1111:
URL: https://github.com/apache/helix/pull/1111


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r448594810



##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -117,6 +120,11 @@
   // Currently, the field is only used for invalidating messages in controller's message cache.
   private boolean _expired = false;
 
+  // The expect period of time (in ms) that a message should be completed, default 1 day
+  public static final long MESSAGE_EXPECT_COMPLETION_PERIOD = HelixUtil
+      .getSystemPropertyAsLong(SystemPropertyKeys.MESSAGE_EXPECTED_COMPLETION_PERIOD,
+          24 * 60 * 60 * 1000 /* 1 day in ms */);

Review comment:
       Neat.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia closed pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia closed pull request #1111:
URL: https://github.com/apache/helix/pull/1111


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] dasahcc commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
dasahcc commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r444514270



##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ClusterStatusMonitor.java
##########
@@ -289,6 +288,32 @@ public void setClusterInstanceStatus(Set<String> liveInstanceSet, Set<String> in
           }
         }
       }
+
+      // update all message related gauges
+      long now = System.currentTimeMillis();
+      long totalMsgQueueSize = 0L;
+      long maxInstanceMsgQueueSize = 0L;
+      long totalPastdueMsgSize = 0L;
+      for (String instance : instanceMessageMap.keySet()) {
+        Set<Message> messages = instanceMessageMap.get(instance);
+        InstanceMonitor bean = _instanceMonitorMap.get(instance);
+
+        long msgQueueSize = messages.size();
+        bean.updateMessageQueueSize(msgQueueSize);
+        totalMsgQueueSize += msgQueueSize;
+        if (msgQueueSize > maxInstanceMsgQueueSize) {
+          maxInstanceMsgQueueSize = msgQueueSize;
+        }
+
+        long pastdueMsgCount = messages.stream()

Review comment:
       pastDueMsgCount

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -522,12 +538,25 @@ public long getExecuteStartTimeStamp() {
 
   /**
    * Get the time that this message was created
-   * @return UNIX timestamp
+   * @return UNIX epoch timestamp

Review comment:
       This is not epoch time. epoch time is not at milisecond but second level.

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -117,6 +120,11 @@
   // Currently, the field is only used for invalidating messages in controller's message cache.
   private boolean _expired = false;
 
+  // The expect period of time (in ms) that a message should be completed, default 1 day
+  public static final long MESSAGE_EXPECT_COMPLETION_PERIOD = HelixUtil

Review comment:
       Do we want to have this param for our internal message report or including for some user defined messages? It is hard to define the time from user perspective.

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -173,6 +181,14 @@ public void setCreateTimeStamp(long timestamp) {
     _record.setLongField(Attributes.CREATE_TIMESTAMP.toString(), timestamp);
   }
 
+  /**
+   * Set the time that the message was expected to be completed
+   * @param timestamp a UNIX epoch timestamp
+   */
+  public void setCompletionDueTimeStamp(long timestamp) {
+    _record.setLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), timestamp);

Review comment:
       Do we need any validation here? For example, it timestamp already < currentmilisecond.

##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -522,12 +538,25 @@ public long getExecuteStartTimeStamp() {
 
   /**
    * Get the time that this message was created
-   * @return UNIX timestamp
+   * @return UNIX epoch timestamp
    */
   public long getCreateTimeStamp() {
     return _record.getLongField(Attributes.CREATE_TIMESTAMP.toString(), 0L);
   }
 
+  /**
+   * Get the time that the message was expected to be completed
+   * @return UNIX epoch timestamp
+   */
+  public long getCompletionDueTimeStamp() {
+    long completionDue = _record.getLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), 0L);
+    if (completionDue == 0) {
+      completionDue = getCreateTimeStamp() + MESSAGE_EXPECT_COMPLETION_PERIOD;
+    }

Review comment:
       We can combine it as:
   
   long completionDue = _record.getLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), getCreateTimeStamp() + MESSAGE_EXPECT_COMPLETION_PERIOD);




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r446770231



##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -522,12 +538,25 @@ public long getExecuteStartTimeStamp() {
 
   /**
    * Get the time that this message was created
-   * @return UNIX timestamp
+   * @return UNIX epoch timestamp

Review comment:
       Fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on pull request #1111:
URL: https://github.com/apache/helix/pull/1111#issuecomment-657637100


   Accidentally closed the PR, trying to re-open it.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] narendly commented on pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
narendly commented on pull request #1111:
URL: https://github.com/apache/helix/pull/1111#issuecomment-649875911


   @lei-xia 
   
   Could we make sure this PR follows our PR guidelines?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r446348695



##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -117,6 +120,11 @@
   // Currently, the field is only used for invalidating messages in controller's message cache.
   private boolean _expired = false;
 
+  // The expect period of time (in ms) that a message should be completed, default 1 day
+  public static final long MESSAGE_EXPECT_COMPLETION_PERIOD = HelixUtil

Review comment:
       We have message.setExpectedCompleteTime(), if user would like to set their customer time for different message, they can do that, right? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on pull request #1111:
URL: https://github.com/apache/helix/pull/1111#issuecomment-657637624


   Rebase to current head:master


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r446355685



##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -173,6 +181,14 @@ public void setCreateTimeStamp(long timestamp) {
     _record.setLongField(Attributes.CREATE_TIMESTAMP.toString(), timestamp);
   }
 
+  /**
+   * Set the time that the message was expected to be completed
+   * @param timestamp a UNIX epoch timestamp
+   */
+  public void setCompletionDueTimeStamp(long timestamp) {
+    _record.setLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), timestamp);

Review comment:
       Is that a problem, for example, I may set an time before current time to invalid the message immediately?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r448607029



##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ClusterStatusMonitor.java
##########
@@ -881,7 +910,7 @@ public String clusterBeanName() {
    * @param instanceName
    * @return instance bean name
    */
-  private String getInstanceBeanName(String instanceName) {
+  public String getInstanceBeanName(String instanceName) {

Review comment:
       I changed all getXXXBeanName() methods to protected.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r448606862



##########
File path: helix-core/src/main/java/org/apache/helix/monitoring/mbeans/ClusterStatusMonitor.java
##########
@@ -289,6 +288,32 @@ public void setClusterInstanceStatus(Set<String> liveInstanceSet, Set<String> in
           }
         }
       }
+
+      // update all message related gauges

Review comment:
       Merged the logic into existing instance bean update loop.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org

[GitHub] [helix] lei-xia commented on a change in pull request #1111: Add Cluster and instance level metrics to report the past-due incompleted message count.

Posted by GitBox <gi...@apache.org>.
lei-xia commented on a change in pull request #1111:
URL: https://github.com/apache/helix/pull/1111#discussion_r446349305



##########
File path: helix-core/src/main/java/org/apache/helix/model/Message.java
##########
@@ -522,12 +538,25 @@ public long getExecuteStartTimeStamp() {
 
   /**
    * Get the time that this message was created
-   * @return UNIX timestamp
+   * @return UNIX epoch timestamp
    */
   public long getCreateTimeStamp() {
     return _record.getLongField(Attributes.CREATE_TIMESTAMP.toString(), 0L);
   }
 
+  /**
+   * Get the time that the message was expected to be completed
+   * @return UNIX epoch timestamp
+   */
+  public long getCompletionDueTimeStamp() {
+    long completionDue = _record.getLongField(Attributes.COMPLETION_DUE_TIMESTAMP.name(), 0L);
+    if (completionDue == 0) {
+      completionDue = getCreateTimeStamp() + MESSAGE_EXPECT_COMPLETION_PERIOD;
+    }

Review comment:
       I considered this, but think to avoid calling getCreateTimeStamp() everytime since it involves string passing, may not be big deal though.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org