[jira] [Created] (ACCUMULO-862) master slow start

["Eric Newton (JIRA)" <ji...@apache.org>]

Eric Newton created ACCUMULO-862:
------------------------------------

             Summary: master slow start
                 Key: ACCUMULO-862
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-862
             Project: Accumulo
          Issue Type: Bug
          Components: master
    Affects Versions: 1.4.1, 1.4.2
         Environment: very large cluster
            Reporter: Eric Newton
            Assignee: Eric Newton
            Priority: Minor
             Fix For: 1.5.0


Master start-up was delayed by many minutes because it discovered only 3 loggers per second.  When jstack'ed, it was stuck here:

{noformat}
"master" prio=10 tid=0x00007ffff086a800 nid=0xa331 in Object.wait() [0x00007fffc565c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000bdd5f940> (a org.apache.zookeeper.ClientCnxn$Packet)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1345)
        - locked <0x00000000bdd5f940> (a org.apache.zookeeper.ClientCnxn$Packet)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1247)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1277)
        at org.apache.accumulo.core.zookeeper.ZooUtil.recursiveDelete(ZooUtil.java:104)
        at org.apache.accumulo.core.zookeeper.ZooUtil.recursiveDelete(ZooUtil.java:118)
        at org.apache.accumulo.server.zookeeper.ZooReaderWriter.recursiveDelete(ZooReaderWriter.java:67)
        at org.apache.accumulo.server.master.state.DeadServerList.delete(DeadServerList.java:66)
        at org.apache.accumulo.server.master.Master.newLogger(Master.java:2198)
        at org.apache.accumulo.server.master.TabletServerLoggers.scanZooKeeperForUpdates(TabletServerLoggers.java:93)
        - locked <0x00000000c2157b68> (a org.apache.accumulo.server.master.TabletServerLoggers)
        at org.apache.accumulo.server.master.Master.run(Master.java:2076)
        at org.apache.accumulo.server.master.Master.main(Master.java:2178)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.accumulo.start.Main$1.run(Main.java:89)
        at java.lang.Thread.run(Thread.java:662)

{noformat}

Due to many stops/start of loggers, it's possible that there were "dead logger" entries that needed to be deleted.  Future starts were not slow.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira