[jira] [Created] (HELIX-535) Helix controller stops working with heavy configuration

["Joy (JIRA)" <ji...@apache.org>]

Joy created HELIX-535:
-------------------------

             Summary: Helix controller stops working with heavy configuration
                 Key: HELIX-535
                 URL: https://issues.apache.org/jira/browse/HELIX-535
             Project: Apache Helix
          Issue Type: Bug
          Components: helix-core
         Environment: machine:$ uname -a
Linux eat1-app373.stg 2.6.32-220.10.1.el6.x86_64 #1 SMP Fri Mar 9 12:37:51 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

JVM version: $ /export/apps/jdk/current/bin/java -version
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

            Reporter: Joy


The issue consistently comes up with heavy configuration: higher number of znodes, higher number of partitions, and higher number of databases.

The goal of our tests is to evaluate the performance of helix controller (in terms of controller latency) with increased number of nodes, databases and partitions.

In our test, we use multiple machines: one for zookeeper, one for helix controller, and the rest are for dummy processes. The configuration is as below:
        zkr <----------> helix
         ^
         |
        V
      dummy processes

We intentionally kill the master dummy processes once every 30 seconds to simulate a failure event. Everything works fine with light configuration such as: 27 nodes + 1db + 729 partitions. However, when the configuration is heavy, such as 81 nodes + 10 databases + 81 partitions for each db, the controller latency increases significantly after several failure events:
                  Control Latency (ms)
First event     : 182
Second event: 188
Third event:     200
Fourth Event:  193
Fifth event:      200
Sixth event:     185
Seventh event: 189
Eight event:      213
Ninth Event:     1082209

And then after this extremely long failure, the helix controller stop working. The controller log is as attached. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)