You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Sam Ma (JIRA)" <ji...@apache.org> on 2016/08/25 00:41:20 UTC
[jira] [Created] (AMQ-6405) "Not enough cluster members have
reported their update positions yet" after restarting master repeatedly
Sam Ma created AMQ-6405:
---------------------------
Summary: "Not enough cluster members have reported their update positions yet" after restarting master repeatedly
Key: AMQ-6405
URL: https://issues.apache.org/jira/browse/AMQ-6405
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Environment: Junit
Reporter: Sam Ma
Priority: Critical
I created the following unit test to restart master repeatedly in [ReplicatedLevelDBBrokerTest.java|https://github.com/apache/activemq/blob/master/activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ReplicatedLevelDBBrokerTest.java]. Occasionally it hung up and reported "Not enough cluster members have reported their update positions yet.". It can't be re-produced every time as it depends largely on timing.
I think it happened due to an issue of [hawtdispatch-transport|https://github.com/fusesource/hawtdispatch]. I have created a [pull request | https://github.com/fusesource/hawtdispatch/pull/33] for that. The issue caused the "next" callback of [wal_session_copy.transport.stop(next)|https://github.com/apache/activemq/blob/master/activemq-leveldb-store/src/main/scala/org/apache/activemq/leveldb/replicated/SlaveLevelDBStore.scala#L154] not being triggered.
I'm not sure when the pull request will be merged and released as hawtdispatch-transport doesn't seem to be maintained actively. That would be great if some work-around could be done in ActiveMQ.
{code:title=ReplicatedLevelDBBrokerTest.java|borderStyle=solid}
@Test(timeout = 1000*60*10)
public void test() throws Throwable {
boolean jmx = false;
try {
System.out.println("======================================");
System.out.println("1. Start 3 activemq nodes.");
System.out.println("======================================");
startBrokerAsync(createBrokerNode("node-1"));
startBrokerAsync(createBrokerNode("node-2"));
startBrokerAsync(createBrokerNode("node-3"));
BrokerService master = waitForNextMaster();
System.out.println("======================================");
System.out.println("2. Push a message to the master and browse the queue");
System.out.println("======================================");
sendMessage(master, pad("Hello World #1", 1024));
assertEquals(1, browseMessages(master, jmx).size());
int times = 200;
System.out.println("iteration " + times + " times.");
for (int i = 0; i < times; i++) {
System.out.println("======================================");
System.out.println( i + ".3. Restart master node");
System.out.println("======================================");
stop(master);
brokers.remove(master);
master = createBrokerNode(master.getBrokerName());
startBrokerAsync(master);
master = waitForNextMaster();
System.out.println("======================================");
System.out.println(i + ".4. Browse the queue on new master");
System.out.println("======================================");
assertEquals(1, browseMessages(master, jmx).size());
System.out.println("======================================");
System.out.println(i + ".5. Found");
System.out.println("======================================");
}
} catch (Throwable e) {
e.printStackTrace();
throw e;
}
}
{code}
{code:none}
Starting ZooKeeper
2016-08-25 09:01:04,848 | INFO | main | Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-08-25 09:01:04,853 | INFO | main | Server environment:host.name=192.168.1.118
2016-08-25 09:01:04,853 | INFO | main | Server environment:java.version=1.7.0_79
2016-08-25 09:01:04,853 | INFO | main | Server environment:java.vendor=Oracle Corporation
2016-08-25 09:01:04,853 | INFO | main | Server environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre
2016-08-25 09:01:04,854 | INFO | main | Server environment:java.class.path=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/classes:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-library/2.11.0/scala-library-2.11.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/classes:/Users/samuel.ma/git/lab/activemq/activemq-client/target/classes:/Users/samuel.ma/.m2/repository/javax/jmdns/jmdns/3.4.1/jmdns-3.4.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-api/1.7.13/slf4j-api-1.7.13.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.11/hawtbuf-1.11.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/Users/samuel.ma/.m2/repository/commons-net/commons-net/3.5/commons-net-3.5.jar:/Users/samuel.ma/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/samuel.ma/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar:/Users/samuel.ma/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/samuel.ma/git/lab/activemq/activemq-openwire-legacy/target/classes:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf-proto/1.11/hawtbuf-proto-1.11.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch-scala-2.11/1.22/hawtdispatch-scala-2.11-1.22.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch/1.22/hawtdispatch-1.22.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb-api/0.6/leveldb-api-0.6.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb/0.6/leveldb-0.6.jar:/Users/samuel.ma/.m2/repository/com/google/guava/guava/12.0/guava-12.0.jar:/Users/samuel.ma/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/samuel.ma/.m2/repository/org/fusesource/leveldbjni/leveldbjni/1.8/leveldbjni-1.8.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtjni/hawtjni-runtime/1.9/hawtjni-runtime-1.9.jar:/Users/samuel.ma/git/lab/hawtdispatch/hawtdispatch-transport/target/classes:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.zookeeper-impl/1.4.0/org.linkedin.zookeeper-impl-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-groovy/1.7.1/org.linkedin.util-groovy-1.7.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/jul-to-slf4j/1.5.8/jul-to-slf4j-1.5.8.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant/1.8.4/ant-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-core/1.4.0/org.linkedin.util-core-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/samuel.ma/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/samuel.ma/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.compendium/4.3.1/org.osgi.compendium-4.3.1.jar:/Users/samuel.ma/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/samuel.ma/.m2/repository/org/xerial/snappy/snappy-java/1.1.2/snappy-java-1.1.2.jar:/Users/samuel.ma/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.3/jackson-core-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.3/jackson-annotations-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.3/jackson-databind-2.6.3.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-core/1.0.0/hadoop-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/samuel.ma/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/samuel.ma/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/samuel.ma/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-kahadb-store/target/classes:/Users/samuel.ma/.m2/repository/org/apache/activemq/protobuf/activemq-protobuf/1.1/activemq-protobuf-1.1.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-test/1.0.0/hadoop-test-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/samuel.ma/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/samuel.ma/.m2/repository/org/apache/commons/commons-math/2.2/commons-math-2.2.jar:/Users/samuel.ma/.m2/repository/org/scalatest/scalatest_2.11/2.1.5/scalatest_2.11-2.1.5.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-reflect/2.11.0/scala-reflect-2.11.0.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.1/scala-parser-combinators_2.11-1.0.1.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/379/0/.cp/:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/378/0/.cp/
2016-08-25 09:01:04,854 | INFO | main | Server environment:java.library.path=/Users/samuel.ma/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2016-08-25 09:01:04,855 | INFO | main | Server environment:java.io.tmpdir=/var/folders/7k/ds05vsr900q_s75x5wq2q9g40000gp/T/
2016-08-25 09:01:04,855 | INFO | main | Server environment:java.compiler=<NA>
2016-08-25 09:01:04,855 | INFO | main | Server environment:os.name=Mac OS X
2016-08-25 09:01:04,855 | INFO | main | Server environment:os.arch=x86_64
2016-08-25 09:01:04,855 | INFO | main | Server environment:os.version=10.10.5
2016-08-25 09:01:04,856 | INFO | main | Server environment:user.name=samuel.ma
2016-08-25 09:01:04,856 | INFO | main | Server environment:user.home=/Users/samuel.ma
2016-08-25 09:01:04,856 | INFO | main | Server environment:user.dir=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store
2016-08-25 09:01:04,857 | INFO | main | tickTime set to 500
2016-08-25 09:01:04,905 | INFO | main | binding to port 0.0.0.0/0.0.0.0:0
ZooKeeper started
======================================
1. Start 3 activemq nodes.
======================================
Wait for master to start up...
2016-08-25 09:01:05,242 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:05,242 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:05,242 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:05,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:05,772 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:05,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:host.name=192.168.1.118
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.version=1.7.0_79
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.vendor=Oracle Corporation
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home/jre
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.class.path=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/classes:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-library/2.11.0/scala-library-2.11.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/classes:/Users/samuel.ma/git/lab/activemq/activemq-client/target/classes:/Users/samuel.ma/.m2/repository/javax/jmdns/jmdns/3.4.1/jmdns-3.4.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-api/1.7.13/slf4j-api-1.7.13.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-jms_1.1_spec/1.1.1/geronimo-jms_1.1_spec-1.1.1.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf/1.11/hawtbuf-1.11.jar:/Users/samuel.ma/.m2/repository/org/apache/geronimo/specs/geronimo-j2ee-management_1.1_spec/1.0.1/geronimo-j2ee-management_1.1_spec-1.0.1.jar:/Users/samuel.ma/.m2/repository/commons-net/commons-net/3.5/commons-net-3.5.jar:/Users/samuel.ma/.m2/repository/junit/junit/4.12/junit-4.12.jar:/Users/samuel.ma/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/Users/samuel.ma/.m2/repository/org/slf4j/slf4j-log4j12/1.7.13/slf4j-log4j12-1.7.13.jar:/Users/samuel.ma/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/samuel.ma/git/lab/activemq/activemq-openwire-legacy/target/classes:/Users/samuel.ma/.m2/repository/org/fusesource/hawtbuf/hawtbuf-proto/1.11/hawtbuf-proto-1.11.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch-scala-2.11/1.22/hawtdispatch-scala-2.11-1.22.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtdispatch/hawtdispatch/1.22/hawtdispatch-1.22.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb-api/0.6/leveldb-api-0.6.jar:/Users/samuel.ma/.m2/repository/org/iq80/leveldb/leveldb/0.6/leveldb-0.6.jar:/Users/samuel.ma/.m2/repository/com/google/guava/guava/12.0/guava-12.0.jar:/Users/samuel.ma/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/samuel.ma/.m2/repository/org/fusesource/leveldbjni/leveldbjni/1.8/leveldbjni-1.8.jar:/Users/samuel.ma/.m2/repository/org/fusesource/hawtjni/hawtjni-runtime/1.9/hawtjni-runtime-1.9.jar:/Users/samuel.ma/git/lab/hawtdispatch/hawtdispatch-transport/target/classes:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.zookeeper-impl/1.4.0/org.linkedin.zookeeper-impl-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-groovy/1.7.1/org.linkedin.util-groovy-1.7.1.jar:/Users/samuel.ma/.m2/repository/org/slf4j/jul-to-slf4j/1.5.8/jul-to-slf4j-1.5.8.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant/1.8.4/ant-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/apache/ant/ant-launcher/1.8.4/ant-launcher-1.8.4.jar:/Users/samuel.ma/.m2/repository/org/json/json/20090211/json-20090211.jar:/Users/samuel.ma/.m2/repository/org/linkedin/org.linkedin.util-core/1.4.0/org.linkedin.util-core-1.4.0.jar:/Users/samuel.ma/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/samuel.ma/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/samuel.ma/.m2/repository/io/netty/netty/3.7.0.Final/netty-3.7.0.Final.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.core/4.3.1/org.osgi.core-4.3.1.jar:/Users/samuel.ma/.m2/repository/org/osgi/org.osgi.compendium/4.3.1/org.osgi.compendium-4.3.1.jar:/Users/samuel.ma/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar:/Users/samuel.ma/.m2/repository/org/xerial/snappy/snappy-java/1.1.2/snappy-java-1.1.2.jar:/Users/samuel.ma/.m2/repository/org/iq80/snappy/snappy/0.2/snappy-0.2.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.3/jackson-core-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.3/jackson-annotations-2.6.3.jar:/Users/samuel.ma/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.3/jackson-databind-2.6.3.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-core/1.0.0/hadoop-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/Users/samuel.ma/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/samuel.ma/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar:/Users/samuel.ma/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar:/Users/samuel.ma/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/Users/samuel.ma/git/lab/activemq/activemq-broker/target/test-classes:/Users/samuel.ma/git/lab/activemq/activemq-kahadb-store/target/classes:/Users/samuel.ma/.m2/repository/org/apache/activemq/protobuf/activemq-protobuf/1.1/activemq-protobuf-1.1.jar:/Users/samuel.ma/.m2/repository/org/apache/hadoop/hadoop-test/1.0.0/hadoop-test-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftplet-api/1.0.0/ftplet-api-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/mina/mina-core/2.0.0-M5/mina-core-2.0.0-M5.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-core/1.0.0/ftpserver-core-1.0.0.jar:/Users/samuel.ma/.m2/repository/org/apache/ftpserver/ftpserver-deprecated/1.0.0-M2/ftpserver-deprecated-1.0.0-M2.jar:/Users/samuel.ma/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-runtime/5.5.12/jasper-runtime-5.5.12.jar:/Users/samuel.ma/.m2/repository/tomcat/jasper-compiler/5.5.12/jasper-compiler-5.5.12.jar:/Users/samuel.ma/.m2/repository/org/apache/commons/commons-math/2.2/commons-math-2.2.jar:/Users/samuel.ma/.m2/repository/org/scalatest/scalatest_2.11/2.1.5/scalatest_2.11-2.1.5.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/scala-reflect/2.11.0/scala-reflect-2.11.0.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.1/scala-parser-combinators_2.11-1.0.1.jar:/Users/samuel.ma/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.1/scala-xml_2.11-1.0.1.jar:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/379/0/.cp/:/Applications/Eclipse-Mars-4.5.app/Contents/Eclipse/configuration/org.eclipse.osgi/378/0/.cp/
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.library.path=/Users/samuel.ma/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
2016-08-25 09:01:05,780 | INFO | Starting broker node: node-3 | Client environment:java.io.tmpdir=/var/folders/7k/ds05vsr900q_s75x5wq2q9g40000gp/T/
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:java.compiler=<NA>
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.name=Mac OS X
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.arch=x86_64
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:os.version=10.10.5
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.name=samuel.ma
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.home=/Users/samuel.ma
2016-08-25 09:01:05,781 | INFO | Starting broker node: node-3 | Client environment:user.dir=/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store
2016-08-25 09:01:05,783 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@57f6eb51
2016-08-25 09:01:05,783 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@13d33ff5
2016-08-25 09:01:05,783 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@57f6eb51
2016-08-25 09:01:05,803 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:05,803 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:05,803 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:05,807 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:05,807 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54918
2016-08-25 09:01:05,808 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:05,808 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:05,812 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54919
2016-08-25 09:01:05,815 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54918
2016-08-25 09:01:05,816 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54920
2016-08-25 09:01:05,816 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54919
2016-08-25 09:01:05,817 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54920
2016-08-25 09:01:05,818 | INFO | SyncThread:0 | Creating new log file: log.1
2016-08-25 09:01:05,827 | INFO | SyncThread:0 | Established session 0x156bec917250000 with negotiated timeout 10000 for client /127.0.0.1:54918
2016-08-25 09:01:05,828 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250000, negotiated timeout = 10000
2016-08-25 09:01:05,828 | INFO | SyncThread:0 | Established session 0x156bec917250001 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54919
2016-08-25 09:01:05,828 | INFO | SyncThread:0 | Established session 0x156bec917250002 with negotiated timeout 10000 for client /127.0.0.1:54920
2016-08-25 09:01:05,829 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250001, negotiated timeout = 10000
2016-08-25 09:01:05,829 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250002, negotiated timeout = 10000
2016-08-25 09:01:05,860 | INFO | ProcessThread(sid:0 cport:-1): | Got user-level KeeperException when processing sessionid:0x156bec917250001 type:create cxid:0x3 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/default Error:KeeperErrorCode = NodeExists for /default
2016-08-25 09:01:05,861 | INFO | ProcessThread(sid:0 cport:-1): | Got user-level KeeperException when processing sessionid:0x156bec917250002 type:create cxid:0x3 zxid:0x6 txntype:-1 reqpath:n/a Error Path:/default Error:KeeperErrorCode = NodeExists for /default
2016-08-25 09:01:06,391 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:06,393 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:06,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:06,405 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:06,524 | WARN | Starting broker node: node-2-EventThread | listeners are taking too long to process the events
2016-08-25 09:01:06,524 | WARN | Starting broker node: node-2-EventThread | listeners are taking too long to process the events
2016-08-25 09:01:06,568 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:06,932 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:54921
2016-08-25 09:01:06,943 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:06,943 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:06,965 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54921
2016-08-25 09:01:06,965 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54921
2016-08-25 09:01:06,982 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:06,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:07,784 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:07,784 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:07,852 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.00/0.00 kb and 1/1 files
2016-08-25 09:01:07,852 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.00/0.00 kb and 1/1 files
2016-08-25 09:01:07,854 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:07,854 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:07,890 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:07,890 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:08,132 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is starting
2016-08-25 09:01:08,167 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54927
2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) started
2016-08-25 09:01:08,168 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:08,169 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 0 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30381 mb of usable space. - resetting to maximum available disk space: 30381 mb
2016-08-25 09:01:08,170 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 0 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30381 mb of usable space. - resetting to maximum available disk space: 30381 mb
Master started: node-2
======================================
2. Push a message to the master and browse the queue
======================================
iteration 200 times.
======================================
0.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:08,709 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is shutting down
2016-08-25 09:01:08,711 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:08,731 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:08,731 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:08,731 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250000
2016-08-25 09:01:08,732 | INFO | Time-limited test | Session: 0x156bec917250000 closed
2016-08-25 09:01:08,732 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:08,733 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54918 which had sessionid 0x156bec917250000
2016-08-25 09:01:08,737 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:08,737 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:08,738 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:08,738 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:09,193 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:09,193 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:09,194 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:09,228 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:09,234 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:09,241 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:09,325 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:09,341 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) uptime 4.193 seconds
2016-08-25 09:01:09,341 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:1) is shutdown
Wait for master to start up...
2016-08-25 09:01:09,350 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:09,358 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@22ea26c0
2016-08-25 09:01:09,358 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:09,359 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:09,360 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:09,360 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54930
2016-08-25 09:01:09,361 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54930
2016-08-25 09:01:09,362 | INFO | SyncThread:0 | Established session 0x156bec917250003 with negotiated timeout 10000 for client /127.0.0.1:54930
2016-08-25 09:01:09,362 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250003, negotiated timeout = 10000
2016-08-25 09:01:09,390 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54931
2016-08-25 09:01:09,396 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:09,396 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54931
2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54931
2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:09,410 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:09,422 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:09,765 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:09,767 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:09,769 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000002
2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000002
2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:09,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000003.log
2016-08-25 09:01:09,771 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000003.log
2016-08-25 09:01:09,771 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:09,776 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.36/2.70 kb and 1/4 files
2016-08-25 09:01:09,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.05/1.34 kb and 1/3 files
2016-08-25 09:01:09,778 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.41/2.70 kb and 2/4 files
2016-08-25 09:01:09,779 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.68/2.70 kb and 3/4 files
2016-08-25 09:01:09,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.06/1.34 kb and 2/3 files
2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.34/1.34 kb and 3/3 files
2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/2.70 kb and 4/4 files
2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:09,780 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:09,790 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:09,790 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:09,792 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is starting
2016-08-25 09:01:09,821 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54936
2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) started
2016-08-25 09:01:09,822 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:09,822 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30783 mb of usable space. - resetting to maximum available disk space: 30783 mb
2016-08-25 09:01:09,822 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30783 mb of usable space. - resetting to maximum available disk space: 30783 mb
Master started: node-3
======================================
0.4. Browse the queue on new master
======================================
======================================
0.5. Found
======================================
======================================
1.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:09,877 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is shutting down
2016-08-25 09:01:09,878 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:09,881 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:09,881 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:09,881 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250001
2016-08-25 09:01:09,882 | INFO | Time-limited test | Session: 0x156bec917250001 closed
2016-08-25 09:01:09,882 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54919 which had sessionid 0x156bec917250001
2016-08-25 09:01:09,882 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:09,882 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:09,882 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:09,883 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:09,883 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:09,917 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:09,920 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:09,924 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:09,930 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:09,990 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:09,992 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) uptime 4.846 seconds
2016-08-25 09:01:09,992 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:2) is shutdown
Wait for master to start up...
2016-08-25 09:01:09,997 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:54938
2016-08-25 09:01:10,000 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:10,006 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@58053ef7
2016-08-25 09:01:10,006 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:10,006 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,007 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:10,007 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:10,007 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54939
2016-08-25 09:01:10,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54938
2016-08-25 09:01:10,008 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54939
2016-08-25 09:01:10,008 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:10,008 | INFO | SyncThread:0 | Established session 0x156bec917250004 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54939
2016-08-25 09:01:10,009 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250004, negotiated timeout = 10000
2016-08-25 09:01:10,021 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,024 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54938
2016-08-25 09:01:10,025 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:10,027 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,031 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000004
2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:10,032 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000006.log
2016-08-25 09:01:10,035 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,037 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.10/2.25 kb and 1/4 files
2016-08-25 09:01:10,038 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.05/2.25 kb and 2/4 files
2016-08-25 09:01:10,038 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.07/2.25 kb and 3/4 files
2016-08-25 09:01:10,039 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files
2016-08-25 09:01:10,040 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:10,041 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,042 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000006.log
2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,043 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:10,044 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000004
2016-08-25 09:01:10,044 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is starting
2016-08-25 09:01:10,053 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.95/2.25 kb and 1/4 files
2016-08-25 09:01:10,053 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.13/2.25 kb and 2/4 files
2016-08-25 09:01:10,054 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.15/2.25 kb and 3/4 files
2016-08-25 09:01:10,055 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files
2016-08-25 09:01:10,056 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:10,060 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,063 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54944
2016-08-25 09:01:10,063 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:10,064 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) started
2016-08-25 09:01:10,064 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:10,064 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:10,064 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-1
======================================
1.4. Browse the queue on new master
======================================
======================================
1.5. Found
======================================
======================================
2.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:10,115 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is shutting down
2016-08-25 09:01:10,116 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:10,119 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:10,120 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250002
2016-08-25 09:01:10,120 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,120 | INFO | Time-limited test | Session: 0x156bec917250002 closed
2016-08-25 09:01:10,120 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:10,121 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54920 which had sessionid 0x156bec917250002
2016-08-25 09:01:10,122 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,122 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,124 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,125 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,130 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:10,130 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,134 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,143 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:10,148 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:10,157 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,204 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:54946
2016-08-25 09:01:10,210 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,212 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:54946
2016-08-25 09:01:10,212 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:10,229 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,240 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000006.log
2016-08-25 09:01:10,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000004
2016-08-25 09:01:10,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:10,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:10,244 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:10,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) uptime 5.100 seconds
2016-08-25 09:01:10,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:3) is shutdown
Wait for master to start up...
2016-08-25 09:01:10,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/2.25 kb and 1/4 files
2016-08-25 09:01:10,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.29/2.25 kb and 2/4 files
2016-08-25 09:01:10,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.24/2.25 kb and 3/4 files
2016-08-25 09:01:10,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.25/2.25 kb and 4/4 files
2016-08-25 09:01:10,250 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:10,263 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:10,267 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,268 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@626495c2
2016-08-25 09:01:10,268 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:10,269 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:10,270 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is starting
2016-08-25 09:01:10,270 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:10,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54949
2016-08-25 09:01:10,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54949
2016-08-25 09:01:10,279 | INFO | SyncThread:0 | Established session 0x156bec917250005 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54949
2016-08-25 09:01:10,280 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250005, negotiated timeout = 10000
2016-08-25 09:01:10,305 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54950
2016-08-25 09:01:10,305 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,306 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:10,307 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) started
2016-08-25 09:01:10,307 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:10,307 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:10,307 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:10,307 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54946
2016-08-25 09:01:10,308 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
Master started: node-2
======================================
2.4. Browse the queue on new master
======================================
2016-08-25 09:01:10,351 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:10,359 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,367 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000004
======================================
2.5. Found
======================================
======================================
3.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:10,368 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is shutting down
2016-08-25 09:01:10,368 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:10,369 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000006.log
2016-08-25 09:01:10,369 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:10,369 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:10,372 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:10,373 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250003
2016-08-25 09:01:10,373 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,374 | INFO | Time-limited test | Session: 0x156bec917250003 closed
2016-08-25 09:01:10,374 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:10,375 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54930 which had sessionid 0x156bec917250003
2016-08-25 09:01:10,377 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,378 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,380 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,381 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:10,400 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:10,400 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,404 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,407 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:10,414 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:10,419 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,455 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54954
2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54954
2016-08-25 09:01:10,461 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:10,479 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:10,480 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) uptime 1.138 seconds
2016-08-25 09:01:10,480 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:4) is shutdown
Wait for master to start up...
2016-08-25 09:01:10,488 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:10,492 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:10,492 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7171ff47
2016-08-25 09:01:10,493 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:10,494 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:10,494 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:10,495 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54956
2016-08-25 09:01:10,495 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54956
2016-08-25 09:01:10,497 | INFO | SyncThread:0 | Established session 0x156bec917250006 with negotiated timeout 10000 for client /127.0.0.1:54956
2016-08-25 09:01:10,497 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250006, negotiated timeout = 10000
2016-08-25 09:01:10,500 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,501 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:10,501 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000007
2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000009.log
2016-08-25 09:01:10,502 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:10,511 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.95/3.21 kb and 1/5 files
2016-08-25 09:01:10,511 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,511 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/3.21 kb and 2/5 files
2016-08-25 09:01:10,512 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.01/3.21 kb and 3/5 files
2016-08-25 09:01:10,513 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54954
2016-08-25 09:01:10,513 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:10,513 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files
2016-08-25 09:01:10,514 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files
2016-08-25 09:01:10,514 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:10,518 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:10,520 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is starting
2016-08-25 09:01:10,535 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54959
2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) started
2016-08-25 09:01:10,536 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:10,536 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:10,537 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:10,537 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
Master started: node-3
======================================
3.4. Browse the queue on new master
======================================
2016-08-25 09:01:10,549 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,551 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000009.log
2016-08-25 09:01:10,551 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000007
2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:10,552 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:10,573 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/3.21 kb and 1/5 files
2016-08-25 09:01:10,574 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/3.21 kb and 2/5 files
2016-08-25 09:01:10,574 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.29/3.21 kb and 3/5 files
2016-08-25 09:01:10,575 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files
2016-08-25 09:01:10,576 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files
2016-08-25 09:01:10,576 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:10,582 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
3.5. Found
======================================
======================================
4.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:10,588 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is shutting down
2016-08-25 09:01:10,589 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:10,592 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:10,593 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250004
2016-08-25 09:01:10,593 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,594 | INFO | Time-limited test | Session: 0x156bec917250004 closed
2016-08-25 09:01:10,594 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:10,594 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54939 which had sessionid 0x156bec917250004
2016-08-25 09:01:10,596 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,596 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,597 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,597 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:10,626 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:10,627 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,630 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,653 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:10,658 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:10,665 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,704 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:54962
2016-08-25 09:01:10,710 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,711 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:54962
2016-08-25 09:01:10,712 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:10,723 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:10,724 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) uptime 0.731 seconds
2016-08-25 09:01:10,724 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:5) is shutdown
Wait for master to start up...
2016-08-25 09:01:10,732 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:10,735 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@17c1fbff
2016-08-25 09:01:10,735 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:10,736 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:10,737 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:10,737 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54964
2016-08-25 09:01:10,738 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54964
2016-08-25 09:01:10,739 | INFO | SyncThread:0 | Established session 0x156bec917250007 with negotiated timeout 10000 for client /127.0.0.1:54964
2016-08-25 09:01:10,739 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250007, negotiated timeout = 10000
2016-08-25 09:01:10,742 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,765 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,765 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:10,768 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54962
2016-08-25 09:01:10,768 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000007
2016-08-25 09:01:10,770 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:10,771 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000009.log
2016-08-25 09:01:10,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/3.21 kb and 1/5 files
2016-08-25 09:01:10,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/3.21 kb and 2/5 files
2016-08-25 09:01:10,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.07/3.21 kb and 3/5 files
2016-08-25 09:01:10,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.02/3.21 kb and 4/5 files
2016-08-25 09:01:10,781 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files
2016-08-25 09:01:10,781 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:10,785 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,787 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is starting
2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54967
2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) started
2016-08-25 09:01:10,799 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:10,800 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:10,800 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
4.4. Browse the queue on new master
======================================
2016-08-25 09:01:10,857 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,867 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000007
2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000009.log
2016-08-25 09:01:10,868 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:10,869 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:10,869 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:10,878 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.15/3.21 kb and 1/5 files
2016-08-25 09:01:10,880 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/3.21 kb and 2/5 files
2016-08-25 09:01:10,881 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.24/3.21 kb and 3/5 files
2016-08-25 09:01:10,882 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.19/3.21 kb and 4/5 files
2016-08-25 09:01:10,883 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/3.21 kb and 5/5 files
2016-08-25 09:01:10,884 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:10,896 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
4.5. Found
======================================
======================================
5.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:10,927 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is shutting down
2016-08-25 09:01:10,930 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:10,935 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,936 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:10,938 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250005
2016-08-25 09:01:10,939 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54949 which had sessionid 0x156bec917250005
2016-08-25 09:01:10,939 | INFO | Time-limited test | Session: 0x156bec917250005 closed
2016-08-25 09:01:10,940 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:10,941 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,942 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:10,952 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:10,953 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:10,984 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:10,985 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:10,988 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,005 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:11,010 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:11,016 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,060 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:11,061 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) uptime 0.815 seconds
2016-08-25 09:01:11,061 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:6) is shutdown
Wait for master to start up...
2016-08-25 09:01:11,063 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:54970
2016-08-25 09:01:11,067 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:11,068 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,070 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6d2eb2e
2016-08-25 09:01:11,071 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54970
2016-08-25 09:01:11,071 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:11,071 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:11,072 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:11,072 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:11,072 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54972
2016-08-25 09:01:11,072 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54972
2016-08-25 09:01:11,073 | INFO | SyncThread:0 | Established session 0x156bec917250008 with negotiated timeout 10000 for client /127.0.0.1:54972
2016-08-25 09:01:11,074 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250008, negotiated timeout = 10000
2016-08-25 09:01:11,089 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,091 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54970
2016-08-25 09:01:11,091 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:11,108 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,117 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:11,119 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000010
2016-08-25 09:01:11,120 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:11,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000012.log
2016-08-25 09:01:11,123 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,123 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000010
2016-08-25 09:01:11,124 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:11,124 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:11,125 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000012.log
2016-08-25 09:01:11,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/4.16 kb and 1/6 files
2016-08-25 09:01:11,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/4.16 kb and 2/6 files
2016-08-25 09:01:11,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.86/4.16 kb and 3/6 files
2016-08-25 09:01:11,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.07/4.16 kb and 4/6 files
2016-08-25 09:01:11,132 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.97/4.16 kb and 5/6 files
2016-08-25 09:01:11,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files
2016-08-25 09:01:11,133 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:11,136 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,138 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.21/4.16 kb and 1/6 files
2016-08-25 09:01:11,138 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is starting
2016-08-25 09:01:11,139 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.11/4.16 kb and 2/6 files
2016-08-25 09:01:11,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.01/4.16 kb and 3/6 files
2016-08-25 09:01:11,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.96/4.16 kb and 4/6 files
2016-08-25 09:01:11,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.97/4.16 kb and 5/6 files
2016-08-25 09:01:11,142 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files
2016-08-25 09:01:11,142 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:11,146 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54976
2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) started
2016-08-25 09:01:11,148 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:11,148 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:11,149 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
5.4. Browse the queue on new master
======================================
======================================
5.5. Found
======================================
======================================
6.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:11,184 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is shutting down
2016-08-25 09:01:11,185 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:11,189 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,190 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:11,191 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250006
2016-08-25 09:01:11,192 | INFO | Time-limited test | Session: 0x156bec917250006 closed
2016-08-25 09:01:11,192 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:11,192 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54956 which had sessionid 0x156bec917250006
2016-08-25 09:01:11,196 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,196 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,197 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,197 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,243 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:11,243 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,244 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:11,246 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,250 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:11,258 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:11,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) uptime 0.826 seconds
2016-08-25 09:01:11,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:7) is shutdown
Wait for master to start up...
2016-08-25 09:01:11,313 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:11,316 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6086510a
2016-08-25 09:01:11,316 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:54978
2016-08-25 09:01:11,316 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:11,317 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:11,318 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:11,318 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54979
2016-08-25 09:01:11,318 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54979
2016-08-25 09:01:11,319 | INFO | SyncThread:0 | Established session 0x156bec917250009 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54979
2016-08-25 09:01:11,319 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250009, negotiated timeout = 10000
2016-08-25 09:01:11,323 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,324 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:54978
2016-08-25 09:01:11,324 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:11,330 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,332 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:54978
2016-08-25 09:01:11,332 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:11,346 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,350 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,351 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:11,351 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000012.log
2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000010
2016-08-25 09:01:11,352 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:11,355 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,358 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/4.16 kb and 1/6 files
2016-08-25 09:01:11,359 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/4.16 kb and 2/6 files
2016-08-25 09:01:11,359 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.82/4.16 kb and 3/6 files
2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000010
2016-08-25 09:01:11,360 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000012.log
2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/4.16 kb and 4/6 files
2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:11,361 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.21/4.16 kb and 5/6 files
2016-08-25 09:01:11,362 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files
2016-08-25 09:01:11,362 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:11,365 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,366 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is starting
2016-08-25 09:01:11,366 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.95/4.16 kb and 1/6 files
2016-08-25 09:01:11,367 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.96/4.16 kb and 2/6 files
2016-08-25 09:01:11,368 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.17/4.16 kb and 3/6 files
2016-08-25 09:01:11,369 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.35/4.16 kb and 4/6 files
2016-08-25 09:01:11,370 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.26/4.16 kb and 5/6 files
2016-08-25 09:01:11,371 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.16/4.16 kb and 6/6 files
2016-08-25 09:01:11,371 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:11,382 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,384 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54984
2016-08-25 09:01:11,384 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:11,385 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) started
2016-08-25 09:01:11,385 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:11,385 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:11,385 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
6.4. Browse the queue on new master
======================================
======================================
6.5. Found
======================================
======================================
7.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:11,421 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is shutting down
2016-08-25 09:01:11,422 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:11,426 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:11,427 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,427 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250007
2016-08-25 09:01:11,428 | INFO | Time-limited test | Session: 0x156bec917250007 closed
2016-08-25 09:01:11,428 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:11,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54964 which had sessionid 0x156bec917250007
2016-08-25 09:01:11,432 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,432 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,433 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,433 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,486 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:11,488 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,506 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:11,507 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,511 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:11,515 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:11,532 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) uptime 0.806 seconds
2016-08-25 09:01:11,532 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:8) is shutdown
Wait for master to start up...
2016-08-25 09:01:11,536 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:11,538 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@28289bdd
2016-08-25 09:01:11,539 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:11,540 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:11,540 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:11,540 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:54986
2016-08-25 09:01:11,541 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:54986
2016-08-25 09:01:11,542 | INFO | SyncThread:0 | Established session 0x156bec91725000a with negotiated timeout 10000 for client /127.0.0.1:54986
2016-08-25 09:01:11,542 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000a, negotiated timeout = 10000
2016-08-25 09:01:11,567 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:54987
2016-08-25 09:01:11,572 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,572 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:54987
2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:54987
2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:11,574 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:11,601 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,612 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,612 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,613 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000013
2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000015.log
2016-08-25 09:01:11,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000016.sst
2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:11,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/5.11 kb and 1/8 files
2016-08-25 09:01:11,623 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000015.log
2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000013
2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000016.sst
2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.16/5.11 kb and 2/8 files
2016-08-25 09:01:11,624 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:11,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.17/5.11 kb and 3/8 files
2016-08-25 09:01:11,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.36/5.11 kb and 4/8 files
2016-08-25 09:01:11,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.26/5.11 kb and 5/8 files
2016-08-25 09:01:11,629 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files
2016-08-25 09:01:11,630 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/5.11 kb and 7/8 files
2016-08-25 09:01:11,630 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files
2016-08-25 09:01:11,631 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:11,632 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/5.11 kb and 1/8 files
2016-08-25 09:01:11,633 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.44/5.11 kb and 2/8 files
2016-08-25 09:01:11,634 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.34/5.11 kb and 3/8 files
2016-08-25 09:01:11,635 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.34/5.11 kb and 4/8 files
2016-08-25 09:01:11,636 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.36/5.11 kb and 5/8 files
2016-08-25 09:01:11,638 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files
2016-08-25 09:01:11,639 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.21/5.11 kb and 7/8 files
2016-08-25 09:01:11,640 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files
2016-08-25 09:01:11,640 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:11,644 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,646 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,646 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is starting
2016-08-25 09:01:11,656 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:54992
2016-08-25 09:01:11,656 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:11,657 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) started
2016-08-25 09:01:11,657 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:11,657 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:11,657 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
7.4. Browse the queue on new master
======================================
======================================
7.5. Found
======================================
======================================
8.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:11,690 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is shutting down
2016-08-25 09:01:11,691 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:11,696 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,697 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:11,698 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250008
2016-08-25 09:01:11,699 | INFO | Time-limited test | Session: 0x156bec917250008 closed
2016-08-25 09:01:11,699 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54972 which had sessionid 0x156bec917250008
2016-08-25 09:01:11,699 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:11,704 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,704 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,704 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,705 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:11,756 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:11,760 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,779 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:11,779 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,785 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:11,790 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,819 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:11,820 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) uptime 0.758 seconds
2016-08-25 09:01:11,820 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:9) is shutdown
Wait for master to start up...
2016-08-25 09:01:11,825 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:11,830 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@15bfc674
2016-08-25 09:01:11,830 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:11,831 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:11,831 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:11,831 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:54994
2016-08-25 09:01:11,832 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:54994
2016-08-25 09:01:11,833 | INFO | SyncThread:0 | Established session 0x156bec91725000b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:54994
2016-08-25 09:01:11,833 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725000b, negotiated timeout = 10000
2016-08-25 09:01:11,874 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:54995
2016-08-25 09:01:11,879 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,879 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:54995
2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:54995
2016-08-25 09:01:11,881 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:11,896 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,906 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,907 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000013
2016-08-25 09:01:11,907 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000016.sst
2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000015.log
2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:11,908 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:11,909 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:11,914 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.26/5.11 kb and 1/8 files
2016-08-25 09:01:11,915 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.16/5.11 kb and 2/8 files
2016-08-25 09:01:11,916 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.06/5.11 kb and 3/8 files
2016-08-25 09:01:11,916 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.06/5.11 kb and 4/8 files
2016-08-25 09:01:11,917 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.24/5.11 kb and 5/8 files
2016-08-25 09:01:11,918 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.26/5.11 kb and 6/8 files
2016-08-25 09:01:11,919 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.16/5.11 kb and 7/8 files
2016-08-25 09:01:11,920 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files
2016-08-25 09:01:11,920 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:11,923 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,924 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:11,924 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is starting
2016-08-25 09:01:11,928 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000015.log
2016-08-25 09:01:11,929 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000013
2016-08-25 09:01:11,930 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000016.sst
2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55000
2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) started
2016-08-25 09:01:11,933 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:11,934 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:11,934 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
8.4. Browse the queue on new master
======================================
2016-08-25 09:01:11,938 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/5.11 kb and 1/8 files
2016-08-25 09:01:11,939 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/5.11 kb and 2/8 files
2016-08-25 09:01:11,940 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.03/5.11 kb and 3/8 files
2016-08-25 09:01:11,941 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.94/5.11 kb and 4/8 files
2016-08-25 09:01:11,942 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.84/5.11 kb and 5/8 files
2016-08-25 09:01:11,942 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.85/5.11 kb and 6/8 files
2016-08-25 09:01:11,944 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 7/8 files
2016-08-25 09:01:11,945 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.11/5.11 kb and 8/8 files
2016-08-25 09:01:11,945 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:11,950 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
8.5. Found
======================================
======================================
9.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:11,967 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is shutting down
2016-08-25 09:01:11,968 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:11,976 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:11,976 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:11,977 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,977 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250009
2016-08-25 09:01:11,978 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:11,978 | INFO | Time-limited test | Session: 0x156bec917250009 closed
2016-08-25 09:01:11,979 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:11,979 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54979 which had sessionid 0x156bec917250009
2016-08-25 09:01:11,980 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:11,980 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:12,082 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:12,083 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) uptime 0.775 seconds
2016-08-25 09:01:12,083 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:10) is shutdown
Wait for master to start up...
2016-08-25 09:01:12,088 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:12,093 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7dc082e2
2016-08-25 09:01:12,093 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:12,094 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:12,094 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:12,094 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55002
2016-08-25 09:01:12,095 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55002
2016-08-25 09:01:12,096 | INFO | SyncThread:0 | Established session 0x156bec91725000c with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55002
2016-08-25 09:01:12,096 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725000c, negotiated timeout = 10000
2016-08-25 09:01:12,110 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,125 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:12,129 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,161 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:12,162 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,167 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:12,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,225 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55003
2016-08-25 09:01:12,232 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,233 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,234 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55003
2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55003
2016-08-25 09:01:12,235 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:12,270 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:12,304 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:12,313 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:12,313 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000018.log
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:12,314 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000018.log
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000016
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000016
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:12,315 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:12,323 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files
2016-08-25 09:01:12,323 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files
2016-08-25 09:01:12,324 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/6.06 kb and 2/8 files
2016-08-25 09:01:12,324 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/6.06 kb and 2/8 files
2016-08-25 09:01:12,325 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/6.06 kb and 3/8 files
2016-08-25 09:01:12,325 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/6.06 kb and 3/8 files
2016-08-25 09:01:12,326 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/6.06 kb and 4/8 files
2016-08-25 09:01:12,326 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.72/6.06 kb and 4/8 files
2016-08-25 09:01:12,327 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/6.06 kb and 5/8 files
2016-08-25 09:01:12,327 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.31/6.06 kb and 5/8 files
2016-08-25 09:01:12,328 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.21/6.06 kb and 6/8 files
2016-08-25 09:01:12,328 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.25/6.06 kb and 6/8 files
2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.16/6.06 kb and 7/8 files
2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.15/6.06 kb and 7/8 files
2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files
2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:12,329 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files
2016-08-25 09:01:12,331 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:12,337 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:12,338 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is starting
2016-08-25 09:01:12,341 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55008
2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) started
2016-08-25 09:01:12,350 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:12,351 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:12,351 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
9.4. Browse the queue on new master
======================================
======================================
9.5. Found
======================================
======================================
10.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:12,385 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is shutting down
2016-08-25 09:01:12,386 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:12,386 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping
2016-08-25 09:01:12,387 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException
2016-08-25 09:01:12,392 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,393 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:12,394 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000a
2016-08-25 09:01:12,395 | INFO | Time-limited test | Session: 0x156bec91725000a closed
2016-08-25 09:01:12,395 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:12,395 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:54986 which had sessionid 0x156bec91725000a
2016-08-25 09:01:12,399 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:12,399 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:12,399 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:12,400 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:12,478 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:12,479 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,482 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,500 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:12,506 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:12,512 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,514 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:12,515 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) uptime 0.983 seconds
2016-08-25 09:01:12,515 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:11) is shutdown
Wait for master to start up...
2016-08-25 09:01:12,520 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:12,523 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@c09a095
2016-08-25 09:01:12,523 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:12,524 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:12,525 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55010
2016-08-25 09:01:12,525 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:12,525 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55010
2016-08-25 09:01:12,526 | INFO | SyncThread:0 | Established session 0x156bec91725000d with negotiated timeout 10000 for client /127.0.0.1:55010
2016-08-25 09:01:12,526 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000d, negotiated timeout = 10000
2016-08-25 09:01:12,592 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55011
2016-08-25 09:01:12,599 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,599 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,601 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55011
2016-08-25 09:01:12,601 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:12,602 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55011
2016-08-25 09:01:12,602 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:12,618 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:12,655 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:12,657 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000018.log
2016-08-25 09:01:12,658 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000016
2016-08-25 09:01:12,659 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:12,659 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:12,672 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files
2016-08-25 09:01:12,673 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/6.06 kb and 2/8 files
2016-08-25 09:01:12,674 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/6.06 kb and 3/8 files
2016-08-25 09:01:12,674 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/6.06 kb and 4/8 files
2016-08-25 09:01:12,675 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/6.06 kb and 5/8 files
2016-08-25 09:01:12,675 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.14/6.06 kb and 6/8 files
2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:12,676 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000018.log
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.04/6.06 kb and 7/8 files
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000016
2016-08-25 09:01:12,677 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:12,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files
2016-08-25 09:01:12,678 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:12,681 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:12,682 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is starting
2016-08-25 09:01:12,686 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.06 kb and 1/8 files
2016-08-25 09:01:12,687 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/6.06 kb and 2/8 files
2016-08-25 09:01:12,687 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/6.06 kb and 3/8 files
2016-08-25 09:01:12,688 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.89/6.06 kb and 4/8 files
2016-08-25 09:01:12,689 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/6.06 kb and 5/8 files
2016-08-25 09:01:12,690 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.74/6.06 kb and 6/8 files
2016-08-25 09:01:12,691 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.04/6.06 kb and 7/8 files
2016-08-25 09:01:12,692 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.06 kb and 8/8 files
2016-08-25 09:01:12,692 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55016
2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) started
2016-08-25 09:01:12,696 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:12,696 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:12,696 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:12,697 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
Master started: node-1
======================================
10.4. Browse the queue on new master
======================================
======================================
10.5. Found
======================================
======================================
11.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:12,734 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is shutting down
2016-08-25 09:01:12,735 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:12,738 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:12,740 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000b
2016-08-25 09:01:12,740 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,743 | INFO | Time-limited test | Session: 0x156bec91725000b closed
2016-08-25 09:01:12,743 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:12,743 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:54994 which had sessionid 0x156bec91725000b
2016-08-25 09:01:12,744 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:12,744 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:12,744 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:12,745 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:12,847 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:12,847 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) uptime 1.026 seconds
2016-08-25 09:01:12,847 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:12) is shutdown
Wait for master to start up...
2016-08-25 09:01:12,853 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:12,857 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@b52dc3
2016-08-25 09:01:12,857 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:12,858 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:12,859 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:12,859 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55018
2016-08-25 09:01:12,859 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55018
2016-08-25 09:01:12,860 | INFO | SyncThread:0 | Established session 0x156bec91725000e with negotiated timeout 10000 for client /127.0.0.1:55018
2016-08-25 09:01:12,861 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000e, negotiated timeout = 10000
2016-08-25 09:01:12,867 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:12,871 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,875 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,884 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:12,885 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:12,890 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:12,898 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,974 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55019
2016-08-25 09:01:12,980 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,980 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55019
2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55019
2016-08-25 09:01:12,982 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:13,027 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:13,044 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,044 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,045 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:13,045 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000019
2016-08-25 09:01:13,046 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000021.log
2016-08-25 09:01:13,047 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:13,048 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,049 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000021.log
2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:13,050 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000019
2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:13,051 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:13,054 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/7.01 kb and 1/9 files
2016-08-25 09:01:13,055 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/7.01 kb and 2/9 files
2016-08-25 09:01:13,056 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/7.01 kb and 3/9 files
2016-08-25 09:01:13,056 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/7.01 kb and 4/9 files
2016-08-25 09:01:13,057 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/7.01 kb and 5/9 files
2016-08-25 09:01:13,058 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/7.01 kb and 6/9 files
2016-08-25 09:01:13,059 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.92/7.01 kb and 7/9 files
2016-08-25 09:01:13,060 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.10/7.01 kb and 8/9 files
2016-08-25 09:01:13,060 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/7.01 kb and 1/9 files
2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files
2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.01 kb and 2/9 files
2016-08-25 09:01:13,061 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:13,062 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/7.01 kb and 3/9 files
2016-08-25 09:01:13,063 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/7.01 kb and 4/9 files
2016-08-25 09:01:13,064 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.79/7.01 kb and 5/9 files
2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.14/7.01 kb and 6/9 files
2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:13,065 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.09/7.01 kb and 7/9 files
2016-08-25 09:01:13,066 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files
2016-08-25 09:01:13,067 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is starting
2016-08-25 09:01:13,067 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files
2016-08-25 09:01:13,068 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:13,073 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,075 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55024
2016-08-25 09:01:13,075 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:13,076 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) started
2016-08-25 09:01:13,076 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:13,076 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:13,076 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
11.4. Browse the queue on new master
======================================
======================================
11.5. Found
======================================
======================================
12.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:13,109 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is shutting down
2016-08-25 09:01:13,110 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:13,113 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:13,114 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,114 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000c
2016-08-25 09:01:13,115 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55002 which had sessionid 0x156bec91725000c
2016-08-25 09:01:13,115 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:13,115 | INFO | Time-limited test | Session: 0x156bec91725000c closed
2016-08-25 09:01:13,121 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,121 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,121 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,122 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:13,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:13,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) uptime 1.161 seconds
2016-08-25 09:01:13,245 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:13) is shutdown
Wait for master to start up...
2016-08-25 09:01:13,249 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:13,256 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4d995fb7
2016-08-25 09:01:13,256 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:13,257 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:13,258 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:13,258 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55026
2016-08-25 09:01:13,258 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55026
2016-08-25 09:01:13,259 | INFO | SyncThread:0 | Established session 0x156bec91725000f with negotiated timeout 10000 for client /127.0.0.1:55026
2016-08-25 09:01:13,259 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725000f, negotiated timeout = 10000
2016-08-25 09:01:13,262 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:13,263 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:13,264 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,265 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,272 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:13,281 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,360 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55027
2016-08-25 09:01:13,366 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,366 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,368 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55027
2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55027
2016-08-25 09:01:13,369 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:13,427 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:13,427 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,431 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000021.log
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000019
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000021.log
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:13,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000019
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:13,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:13,440 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/7.01 kb and 1/9 files
2016-08-25 09:01:13,441 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.01 kb and 2/9 files
2016-08-25 09:01:13,441 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.44/7.01 kb and 3/9 files
2016-08-25 09:01:13,442 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.34/7.01 kb and 4/9 files
2016-08-25 09:01:13,443 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.24/7.01 kb and 5/9 files
2016-08-25 09:01:13,444 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.26/7.01 kb and 6/9 files
2016-08-25 09:01:13,445 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.21/7.01 kb and 7/9 files
2016-08-25 09:01:13,446 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files
2016-08-25 09:01:13,447 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files
2016-08-25 09:01:13,448 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:13,452 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,452 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/7.01 kb and 1/9 files
2016-08-25 09:01:13,453 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is starting
2016-08-25 09:01:13,454 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/7.01 kb and 2/9 files
2016-08-25 09:01:13,455 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/7.01 kb and 3/9 files
2016-08-25 09:01:13,456 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.00/7.01 kb and 4/9 files
2016-08-25 09:01:13,457 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/7.01 kb and 5/9 files
2016-08-25 09:01:13,458 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/7.01 kb and 6/9 files
2016-08-25 09:01:13,459 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.16/7.01 kb and 7/9 files
2016-08-25 09:01:13,460 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.11/7.01 kb and 8/9 files
2016-08-25 09:01:13,461 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.01/7.01 kb and 9/9 files
2016-08-25 09:01:13,462 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:13,462 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55032
2016-08-25 09:01:13,462 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:13,463 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) started
2016-08-25 09:01:13,463 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:13,463 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:13,463 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
12.4. Browse the queue on new master
======================================
2016-08-25 09:01:13,466 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
12.5. Found
======================================
======================================
13.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:13,484 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is shutting down
2016-08-25 09:01:13,485 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:13,489 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:13,489 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000d
2016-08-25 09:01:13,489 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,490 | INFO | Time-limited test | Session: 0x156bec91725000d closed
2016-08-25 09:01:13,490 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:13,490 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55010 which had sessionid 0x156bec91725000d
2016-08-25 09:01:13,494 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,494 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,494 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:13,494 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,594 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:13,594 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) uptime 1.078 seconds
2016-08-25 09:01:13,595 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:14) is shutdown
Wait for master to start up...
2016-08-25 09:01:13,598 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:13,609 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@41991669
2016-08-25 09:01:13,610 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:13,611 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:13,611 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:13,611 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55034
2016-08-25 09:01:13,611 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55034
2016-08-25 09:01:13,612 | INFO | SyncThread:0 | Established session 0x156bec917250010 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55034
2016-08-25 09:01:13,613 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250010, negotiated timeout = 10000
2016-08-25 09:01:13,627 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,667 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:13,672 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,678 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:13,678 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,683 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:13,688 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,771 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55035
2016-08-25 09:01:13,777 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,777 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:13,779 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55035
2016-08-25 09:01:13,779 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:13,780 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55035
2016-08-25 09:01:13,780 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:13,842 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,842 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,846 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,847 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:13,847 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000022
2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:13,848 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000022
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000024.log
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:13,849 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000024.log
2016-08-25 09:01:13,850 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:13,850 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:13,857 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.96 kb and 1/10 files
2016-08-25 09:01:13,858 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/7.96 kb and 2/10 files
2016-08-25 09:01:13,859 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/7.96 kb and 3/10 files
2016-08-25 09:01:13,859 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.11/7.96 kb and 4/10 files
2016-08-25 09:01:13,860 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.01/7.96 kb and 5/10 files
2016-08-25 09:01:13,861 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.91/7.96 kb and 6/10 files
2016-08-25 09:01:13,862 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.81/7.96 kb and 7/10 files
2016-08-25 09:01:13,863 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.82/7.96 kb and 8/10 files
2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.77/7.96 kb and 9/10 files
2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/7.96 kb and 1/10 files
2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files
2016-08-25 09:01:13,864 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/7.96 kb and 2/10 files
2016-08-25 09:01:13,865 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:13,866 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.86/7.96 kb and 3/10 files
2016-08-25 09:01:13,867 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.76/7.96 kb and 4/10 files
2016-08-25 09:01:13,868 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.66/7.96 kb and 5/10 files
2016-08-25 09:01:13,869 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.07/7.96 kb and 6/10 files
2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.97/7.96 kb and 7/10 files
2016-08-25 09:01:13,870 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.16/7.96 kb and 8/10 files
2016-08-25 09:01:13,871 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is starting
2016-08-25 09:01:13,871 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.06/7.96 kb and 9/10 files
2016-08-25 09:01:13,872 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files
2016-08-25 09:01:13,872 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:13,878 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,878 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55040
2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) started
2016-08-25 09:01:13,879 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:13,879 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:13,879 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
13.4. Browse the queue on new master
======================================
======================================
13.5. Found
======================================
======================================
14.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:13,907 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is shutting down
2016-08-25 09:01:13,908 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:13,913 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:13,914 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:13,915 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:13,916 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:13,918 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:13,919 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000e
2016-08-25 09:01:13,920 | INFO | Time-limited test | Session: 0x156bec91725000e closed
2016-08-25 09:01:13,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55018 which had sessionid 0x156bec91725000e
2016-08-25 09:01:13,920 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:14,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:14,039 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:14,039 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,041 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,045 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:14,046 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:14,047 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) uptime 1.199 seconds
2016-08-25 09:01:14,047 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:15) is shutdown
Wait for master to start up...
2016-08-25 09:01:14,050 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:14,052 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,053 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@102873a6
2016-08-25 09:01:14,053 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:14,054 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:14,055 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:14,055 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55042
2016-08-25 09:01:14,055 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55042
2016-08-25 09:01:14,064 | INFO | SyncThread:0 | Established session 0x156bec917250011 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55042
2016-08-25 09:01:14,064 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250011, negotiated timeout = 10000
2016-08-25 09:01:14,133 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55043
2016-08-25 09:01:14,139 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,139 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,140 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55043
2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55043
2016-08-25 09:01:14,141 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:14,154 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:14,165 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000022
2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:14,167 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:14,168 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000024.log
2016-08-25 09:01:14,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.41/7.96 kb and 1/10 files
2016-08-25 09:01:14,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.31/7.96 kb and 2/10 files
2016-08-25 09:01:14,176 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.33/7.96 kb and 3/10 files
2016-08-25 09:01:14,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.23/7.96 kb and 4/10 files
2016-08-25 09:01:14,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.13/7.96 kb and 5/10 files
2016-08-25 09:01:14,178 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.03/7.96 kb and 6/10 files
2016-08-25 09:01:14,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.97/7.96 kb and 7/10 files
2016-08-25 09:01:14,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.87/7.96 kb and 8/10 files
2016-08-25 09:01:14,181 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.77/7.96 kb and 9/10 files
2016-08-25 09:01:14,182 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/7.96 kb and 10/10 files
2016-08-25 09:01:14,182 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:14,185 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:14,187 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is starting
2016-08-25 09:01:14,196 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55047
2016-08-25 09:01:14,196 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:14,197 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) started
2016-08-25 09:01:14,197 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:14,197 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:01:14,197 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
14.4. Browse the queue on new master
======================================
2016-08-25 09:01:14,203 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:14,219 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000022
2016-08-25 09:01:14,221 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000024.log
2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:14,222 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
======================================
14.5. Found
======================================
======================================
15.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:14,234 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is shutting down
2016-08-25 09:01:14,234 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:14,239 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,240 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:14,240 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:14,240 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:14,241 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:14,242 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:14,242 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725000f
2016-08-25 09:01:14,253 | INFO | Time-limited test | Session: 0x156bec91725000f closed
2016-08-25 09:01:14,253 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55026 which had sessionid 0x156bec91725000f
2016-08-25 09:01:14,253 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:14,327 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:14,327 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,330 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,357 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:14,358 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) uptime 1.112 seconds
2016-08-25 09:01:14,358 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:16) is shutdown
Wait for master to start up...
2016-08-25 09:01:14,361 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:14,365 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6ac47476
2016-08-25 09:01:14,365 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:14,366 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:14,366 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:14,366 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55050
2016-08-25 09:01:14,366 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55050
2016-08-25 09:01:14,367 | INFO | SyncThread:0 | Established session 0x156bec917250012 with negotiated timeout 10000 for client /127.0.0.1:55050
2016-08-25 09:01:14,367 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250012, negotiated timeout = 10000
2016-08-25 09:01:14,376 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,425 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:14,430 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:14,436 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,519 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55051
2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55051
2016-08-25 09:01:14,525 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:14,526 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55051
2016-08-25 09:01:14,526 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:14,577 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:14,581 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:14,587 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:14,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000025
2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000027.log
2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:14,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:14,591 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:14,593 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000025
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000027.log
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:14,594 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:14,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:14,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:14,597 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files
2016-08-25 09:01:14,598 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/8.91 kb and 2/11 files
2016-08-25 09:01:14,599 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/8.91 kb and 3/11 files
2016-08-25 09:01:14,599 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/8.91 kb and 4/11 files
2016-08-25 09:01:14,600 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.62/8.91 kb and 5/11 files
2016-08-25 09:01:14,601 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.08/8.91 kb and 6/11 files
2016-08-25 09:01:14,602 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files
2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.21/8.91 kb and 8/11 files
2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files
2016-08-25 09:01:14,603 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.11/8.91 kb and 9/11 files
2016-08-25 09:01:14,604 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/8.91 kb and 2/11 files
2016-08-25 09:01:14,604 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.01/8.91 kb and 10/11 files
2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files
2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:14,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/8.91 kb and 3/11 files
2016-08-25 09:01:14,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.27/8.91 kb and 4/11 files
2016-08-25 09:01:14,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.17/8.91 kb and 5/11 files
2016-08-25 09:01:14,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.08/8.91 kb and 6/11 files
2016-08-25 09:01:14,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files
2016-08-25 09:01:14,610 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.21/8.91 kb and 8/11 files
2016-08-25 09:01:14,610 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:14,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.11/8.91 kb and 9/11 files
2016-08-25 09:01:14,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.01/8.91 kb and 10/11 files
2016-08-25 09:01:14,611 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is starting
2016-08-25 09:01:14,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files
2016-08-25 09:01:14,612 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55056
2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) started
2016-08-25 09:01:14,618 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:14,618 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:14,619 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:14,620 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
Master started: node-1
======================================
15.4. Browse the queue on new master
======================================
======================================
15.5. Found
======================================
======================================
16.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:14,646 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is shutting down
2016-08-25 09:01:14,646 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:14,652 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:14,653 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250011
2016-08-25 09:01:14,653 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:14,653 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,653 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55042 which had sessionid 0x156bec917250011
2016-08-25 09:01:14,654 | INFO | Time-limited test | Session: 0x156bec917250011 closed
2016-08-25 09:01:14,654 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:14,656 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:14,656 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:14,656 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:14,656 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:14,658 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,783 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:14,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) uptime 0.736 seconds
2016-08-25 09:01:14,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:17) is shutdown
Wait for master to start up...
2016-08-25 09:01:14,788 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:14,789 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:14,794 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,795 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@37f832df
2016-08-25 09:01:14,795 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:14,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:14,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:14,796 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:14,797 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:14,797 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55058
2016-08-25 09:01:14,797 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55058
2016-08-25 09:01:14,798 | INFO | SyncThread:0 | Established session 0x156bec917250013 with negotiated timeout 10000 for client /127.0.0.1:55058
2016-08-25 09:01:14,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250013, negotiated timeout = 10000
2016-08-25 09:01:14,802 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:14,813 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,895 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55059
2016-08-25 09:01:14,901 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,901 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:14,902 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55059
2016-08-25 09:01:14,902 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:14,903 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55059
2016-08-25 09:01:14,903 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:14,917 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:14,968 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,968 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:14,969 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:14,970 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000027.log
2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:14,971 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000025
2016-08-25 09:01:14,977 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files
2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/8.91 kb and 2/11 files
2016-08-25 09:01:14,978 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.82/8.91 kb and 3/11 files
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000025
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000027.log
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.72/8.91 kb and 4/11 files
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:14,979 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.62/8.91 kb and 5/11 files
2016-08-25 09:01:14,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.52/8.91 kb and 6/11 files
2016-08-25 09:01:14,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.42/8.91 kb and 7/11 files
2016-08-25 09:01:14,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.32/8.91 kb and 8/11 files
2016-08-25 09:01:14,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.50/8.91 kb and 9/11 files
2016-08-25 09:01:14,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/8.91 kb and 10/11 files
2016-08-25 09:01:14,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files
2016-08-25 09:01:14,983 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:14,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.91 kb and 1/11 files
2016-08-25 09:01:14,987 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:14,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.36/8.91 kb and 2/11 files
2016-08-25 09:01:14,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.26/8.91 kb and 3/11 files
2016-08-25 09:01:14,990 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is starting
2016-08-25 09:01:14,990 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.44/8.91 kb and 4/11 files
2016-08-25 09:01:14,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.34/8.91 kb and 5/11 files
2016-08-25 09:01:14,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/8.91 kb and 6/11 files
2016-08-25 09:01:14,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/8.91 kb and 7/11 files
2016-08-25 09:01:14,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/8.91 kb and 8/11 files
2016-08-25 09:01:14,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.06/8.91 kb and 9/11 files
2016-08-25 09:01:14,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/8.91 kb and 10/11 files
2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55064
2016-08-25 09:01:14,996 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.91/8.91 kb and 11/11 files
2016-08-25 09:01:14,996 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:14,996 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) started
2016-08-25 09:01:14,997 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:14,997 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:14,997 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
16.4. Browse the queue on new master
======================================
2016-08-25 09:01:15,002 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
16.5. Found
======================================
======================================
17.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:15,023 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is shutting down
2016-08-25 09:01:15,024 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:15,029 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,032 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:15,034 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250010
2016-08-25 09:01:15,035 | INFO | Time-limited test | Session: 0x156bec917250010 closed
2016-08-25 09:01:15,035 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55034 which had sessionid 0x156bec917250010
2016-08-25 09:01:15,035 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:15,037 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,037 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,037 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:15,038 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:15,140 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:15,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) uptime 1.546 seconds
2016-08-25 09:01:15,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:18) is shutdown
Wait for master to start up...
2016-08-25 09:01:15,145 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:15,150 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@184797a
2016-08-25 09:01:15,151 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:15,152 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:15,152 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:15,152 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55066
2016-08-25 09:01:15,153 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55066
2016-08-25 09:01:15,153 | INFO | SyncThread:0 | Established session 0x156bec917250014 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55066
2016-08-25 09:01:15,154 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250014, negotiated timeout = 10000
2016-08-25 09:01:15,174 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,176 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:15,179 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,232 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:15,232 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,240 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:15,245 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,337 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55067
2016-08-25 09:01:15,343 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,343 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55067
2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55067
2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:15,344 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:15,359 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:15,400 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,410 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:15,411 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:15,412 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:15,412 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:15,413 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000030.log
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000028
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000030.log
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000031.sst
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:15,414 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000031.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000028
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:15,415 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files
2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files
2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/9.86 kb and 2/13 files
2016-08-25 09:01:15,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/9.86 kb and 2/13 files
2016-08-25 09:01:15,425 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/9.86 kb and 3/13 files
2016-08-25 09:01:15,425 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.82/9.86 kb and 3/13 files
2016-08-25 09:01:15,426 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.65/9.86 kb and 4/13 files
2016-08-25 09:01:15,426 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.72/9.86 kb and 4/13 files
2016-08-25 09:01:15,427 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.90/9.86 kb and 5/13 files
2016-08-25 09:01:15,427 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.55/9.86 kb and 5/13 files
2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.56/9.86 kb and 6/13 files
2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.41/9.86 kb and 6/13 files
2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.41/9.86 kb and 7/13 files
2016-08-25 09:01:15,428 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.47/9.86 kb and 7/13 files
2016-08-25 09:01:15,429 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.37/9.86 kb and 8/13 files
2016-08-25 09:01:15,429 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.31/9.86 kb and 8/13 files
2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.55/9.86 kb and 9/13 files
2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.21/9.86 kb and 9/13 files
2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.11/9.86 kb and 10/13 files
2016-08-25 09:01:15,430 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.45/9.86 kb and 10/13 files
2016-08-25 09:01:15,431 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.06/9.86 kb and 11/13 files
2016-08-25 09:01:15,431 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.35/9.86 kb and 11/13 files
2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.96/9.86 kb and 12/13 files
2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.35/9.86 kb and 12/13 files
2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files
2016-08-25 09:01:15,432 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files
2016-08-25 09:01:15,433 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:15,433 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:15,437 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:15,437 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,438 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is starting
2016-08-25 09:01:15,444 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55072
2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) started
2016-08-25 09:01:15,452 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:15,452 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:15,453 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
17.4. Browse the queue on new master
======================================
======================================
17.5. Found
======================================
======================================
18.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:15,486 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is shutting down
2016-08-25 09:01:15,488 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:15,494 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,494 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:15,495 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250012
2016-08-25 09:01:15,496 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55050 which had sessionid 0x156bec917250012
2016-08-25 09:01:15,496 | INFO | Time-limited test | Session: 0x156bec917250012 closed
2016-08-25 09:01:15,496 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:15,500 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,500 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,501 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:15,501 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,616 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:15,616 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) uptime 1.258 seconds
2016-08-25 09:01:15,617 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:19) is shutdown
Wait for master to start up...
2016-08-25 09:01:15,620 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:15,624 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@519b1696
2016-08-25 09:01:15,624 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:15,626 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:15,626 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:15,626 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55074
2016-08-25 09:01:15,627 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55074
2016-08-25 09:01:15,628 | INFO | SyncThread:0 | Established session 0x156bec917250015 with negotiated timeout 10000 for client /127.0.0.1:55074
2016-08-25 09:01:15,628 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250015, negotiated timeout = 10000
2016-08-25 09:01:15,647 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,679 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:15,679 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,686 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,691 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:15,697 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:15,702 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,788 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55075
2016-08-25 09:01:15,792 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,794 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:15,794 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55075
2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55075
2016-08-25 09:01:15,795 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:15,858 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,870 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:15,870 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:15,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:15,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000030.log
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:15,872 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000031.sst
2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:15,873 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000028
2016-08-25 09:01:15,874 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000031.sst
2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:15,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000028
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:15,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:15,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000030.log
2016-08-25 09:01:15,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files
2016-08-25 09:01:15,883 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/9.86 kb and 2/13 files
2016-08-25 09:01:15,884 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.99/9.86 kb and 3/13 files
2016-08-25 09:01:15,884 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/9.86 kb and 4/13 files
2016-08-25 09:01:15,885 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 1/13 files
2016-08-25 09:01:15,885 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.95/9.86 kb and 5/13 files
2016-08-25 09:01:15,886 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 2/13 files
2016-08-25 09:01:15,886 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.85/9.86 kb and 6/13 files
2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/9.86 kb and 3/13 files
2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.75/9.86 kb and 7/13 files
2016-08-25 09:01:15,887 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/9.86 kb and 4/13 files
2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.65/9.86 kb and 8/13 files
2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/9.86 kb and 5/13 files
2016-08-25 09:01:15,888 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.55/9.86 kb and 9/13 files
2016-08-25 09:01:15,889 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.11/9.86 kb and 6/13 files
2016-08-25 09:01:15,889 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.55/9.86 kb and 10/13 files
2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/9.86 kb and 11/13 files
2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.01/9.86 kb and 7/13 files
2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.35/9.86 kb and 12/13 files
2016-08-25 09:01:15,890 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.91/9.86 kb and 8/13 files
2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files
2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:15,891 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.81/9.86 kb and 9/13 files
2016-08-25 09:01:15,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/9.86 kb and 10/13 files
2016-08-25 09:01:15,893 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.72/9.86 kb and 11/13 files
2016-08-25 09:01:15,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.67/9.86 kb and 12/13 files
2016-08-25 09:01:15,895 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.86/9.86 kb and 13/13 files
2016-08-25 09:01:15,895 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:15,896 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,898 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:15,898 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is starting
2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55080
2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) started
2016-08-25 09:01:15,903 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:15,904 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:15,904 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
18.4. Browse the queue on new master
======================================
======================================
18.5. Found
======================================
======================================
19.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:15,928 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is shutting down
2016-08-25 09:01:15,929 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:15,934 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:15,935 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,936 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:15,936 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:15,936 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:15,936 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:15,938 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250013
2016-08-25 09:01:15,939 | INFO | Time-limited test | Session: 0x156bec917250013 closed
2016-08-25 09:01:15,939 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55058 which had sessionid 0x156bec917250013
2016-08-25 09:01:15,939 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:16,042 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:16,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) uptime 1.258 seconds
2016-08-25 09:01:16,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:20) is shutdown
Wait for master to start up...
2016-08-25 09:01:16,046 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:16,048 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7505d7e7
2016-08-25 09:01:16,048 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:16,049 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:16,049 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:16,049 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55082
2016-08-25 09:01:16,049 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55082
2016-08-25 09:01:16,050 | INFO | SyncThread:0 | Established session 0x156bec917250016 with negotiated timeout 10000 for client /127.0.0.1:55082
2016-08-25 09:01:16,050 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250016, negotiated timeout = 10000
2016-08-25 09:01:16,068 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,140 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:16,140 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:16,141 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,144 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,147 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:16,152 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55083
2016-08-25 09:01:16,242 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,242 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,243 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55083
2016-08-25 09:01:16,243 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:16,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55083
2016-08-25 09:01:16,244 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:16,319 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:16,319 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:16,325 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:16,325 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:16,326 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000031
2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:16,328 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000033.log
2016-08-25 09:01:16,329 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:16,329 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:16,327 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:16,330 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:16,330 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000031
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:16,331 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000033.log
2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:16,332 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:16,348 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/10.81 kb and 1/13 files
2016-08-25 09:01:16,348 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files
2016-08-25 09:01:16,349 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/10.81 kb and 2/13 files
2016-08-25 09:01:16,349 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.46/10.81 kb and 2/13 files
2016-08-25 09:01:16,350 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.82/10.81 kb and 3/13 files
2016-08-25 09:01:16,350 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.36/10.81 kb and 3/13 files
2016-08-25 09:01:16,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.72/10.81 kb and 4/13 files
2016-08-25 09:01:16,351 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.26/10.81 kb and 4/13 files
2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.62/10.81 kb and 5/13 files
2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.16/10.81 kb and 5/13 files
2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.06/10.81 kb and 6/13 files
2016-08-25 09:01:16,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.52/10.81 kb and 6/13 files
2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.96/10.81 kb and 7/13 files
2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.08/10.81 kb and 7/13 files
2016-08-25 09:01:16,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.86/10.81 kb and 8/13 files
2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.98/10.81 kb and 8/13 files
2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.76/10.81 kb and 9/13 files
2016-08-25 09:01:16,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.88/10.81 kb and 9/13 files
2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.71/10.81 kb and 10/13 files
2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.78/10.81 kb and 10/13 files
2016-08-25 09:01:16,355 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/10.81 kb and 11/13 files
2016-08-25 09:01:16,356 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.96/10.81 kb and 11/13 files
2016-08-25 09:01:16,356 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.91/10.81 kb and 12/13 files
2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files
2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.91/10.81 kb and 12/13 files
2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files
2016-08-25 09:01:16,357 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:16,361 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:16,362 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is starting
2016-08-25 09:01:16,362 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55088
2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) started
2016-08-25 09:01:16,367 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:16,368 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:16,368 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
19.4. Browse the queue on new master
======================================
======================================
19.5. Found
======================================
======================================
20.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:16,389 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is shutting down
2016-08-25 09:01:16,390 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:16,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,396 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:16,398 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250014
2016-08-25 09:01:16,399 | INFO | Time-limited test | Session: 0x156bec917250014 closed
2016-08-25 09:01:16,399 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:16,399 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55066 which had sessionid 0x156bec917250014
2016-08-25 09:01:16,400 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:16,400 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:16,404 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:16,404 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:16,525 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:16,526 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) uptime 1.382 seconds
2016-08-25 09:01:16,526 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:21) is shutdown
Wait for master to start up...
2016-08-25 09:01:16,529 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:16,532 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@177a26e
2016-08-25 09:01:16,533 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:16,533 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:16,534 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:16,534 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55090
2016-08-25 09:01:16,535 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55090
2016-08-25 09:01:16,536 | INFO | SyncThread:0 | Established session 0x156bec917250017 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55090
2016-08-25 09:01:16,536 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250017, negotiated timeout = 10000
2016-08-25 09:01:16,550 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,556 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:16,556 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,561 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,578 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:16,586 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:16,592 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,675 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55091
2016-08-25 09:01:16,680 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,680 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55091
2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55091
2016-08-25 09:01:16,681 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:16,682 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:16,696 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:16,749 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:16,757 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000033.log
2016-08-25 09:01:16,758 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000031
2016-08-25 09:01:16,759 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:16,760 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:16,763 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:16,764 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:16,765 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000033.log
2016-08-25 09:01:16,766 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000031
2016-08-25 09:01:16,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files
2016-08-25 09:01:16,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/10.81 kb and 2/13 files
2016-08-25 09:01:16,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.03/10.81 kb and 3/13 files
2016-08-25 09:01:16,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.93/10.81 kb and 4/13 files
2016-08-25 09:01:16,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/10.81 kb and 5/13 files
2016-08-25 09:01:16,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/10.81 kb and 6/13 files
2016-08-25 09:01:16,775 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/10.81 kb and 7/13 files
2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/10.81 kb and 8/13 files
2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/10.81 kb and 1/13 files
2016-08-25 09:01:16,776 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/10.81 kb and 9/13 files
2016-08-25 09:01:16,777 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/10.81 kb and 10/13 files
2016-08-25 09:01:16,778 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/10.81 kb and 11/13 files
2016-08-25 09:01:16,778 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/10.81 kb and 2/13 files
2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/10.81 kb and 12/13 files
2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files
2016-08-25 09:01:16,780 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:16,779 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/10.81 kb and 3/13 files
2016-08-25 09:01:16,780 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/10.81 kb and 4/13 files
2016-08-25 09:01:16,781 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/10.81 kb and 5/13 files
2016-08-25 09:01:16,782 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.40/10.81 kb and 6/13 files
2016-08-25 09:01:16,783 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:16,783 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.30/10.81 kb and 7/13 files
2016-08-25 09:01:16,784 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.32/10.81 kb and 8/13 files
2016-08-25 09:01:16,784 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is starting
2016-08-25 09:01:16,785 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.26/10.81 kb and 9/13 files
2016-08-25 09:01:16,786 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.16/10.81 kb and 10/13 files
2016-08-25 09:01:16,786 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.06/10.81 kb and 11/13 files
2016-08-25 09:01:16,787 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.25/10.81 kb and 12/13 files
2016-08-25 09:01:16,788 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.81/10.81 kb and 13/13 files
2016-08-25 09:01:16,788 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:16,790 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55096
2016-08-25 09:01:16,790 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:16,791 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) started
2016-08-25 09:01:16,791 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:16,791 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:16,791 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:16,793 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
Master started: node-2
======================================
20.4. Browse the queue on new master
======================================
======================================
20.5. Found
======================================
======================================
21.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:16,813 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is shutting down
2016-08-25 09:01:16,814 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:16,814 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping
2016-08-25 09:01:16,815 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException
2016-08-25 09:01:16,822 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:16,827 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:16,828 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250015
2016-08-25 09:01:16,829 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55074 which had sessionid 0x156bec917250015
2016-08-25 09:01:16,830 | INFO | Time-limited test | Session: 0x156bec917250015 closed
2016-08-25 09:01:16,830 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:16,833 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:16,833 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:16,833 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:16,833 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:16,933 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:16,934 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) uptime 1.316 seconds
2016-08-25 09:01:16,934 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:22) is shutdown
Wait for master to start up...
2016-08-25 09:01:16,936 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:16,940 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6f900132
2016-08-25 09:01:16,940 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:16,943 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:16,944 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:16,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55098
2016-08-25 09:01:16,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55098
2016-08-25 09:01:16,945 | INFO | SyncThread:0 | Established session 0x156bec917250018 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55098
2016-08-25 09:01:16,945 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250018, negotiated timeout = 10000
2016-08-25 09:01:16,956 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,124 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:17,128 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,147 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:17,148 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,153 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:17,170 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,312 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55099
2016-08-25 09:01:17,317 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,317 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,319 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55099
2016-08-25 09:01:17,319 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:17,320 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55099
2016-08-25 09:01:17,321 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:17,412 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:17,431 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:17,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000037.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000036.log
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000034
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000036.log
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:17,442 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000034
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:17,443 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000037.sst
2016-08-25 09:01:17,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files
2016-08-25 09:01:17,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/11.75 kb and 2/15 files
2016-08-25 09:01:17,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/11.75 kb and 3/15 files
2016-08-25 09:01:17,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/11.75 kb and 4/15 files
2016-08-25 09:01:17,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/11.75 kb and 5/15 files
2016-08-25 09:01:17,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/11.75 kb and 6/15 files
2016-08-25 09:01:17,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/11.75 kb and 7/15 files
2016-08-25 09:01:17,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/11.75 kb and 8/15 files
2016-08-25 09:01:17,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.64/11.75 kb and 9/15 files
2016-08-25 09:01:17,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/11.75 kb and 10/15 files
2016-08-25 09:01:17,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.15/11.75 kb and 11/15 files
2016-08-25 09:01:17,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/11.75 kb and 12/15 files
2016-08-25 09:01:17,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files
2016-08-25 09:01:17,460 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files
2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files
2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/11.75 kb and 2/15 files
2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files
2016-08-25 09:01:17,461 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:17,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/11.75 kb and 3/15 files
2016-08-25 09:01:17,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.31/11.75 kb and 4/15 files
2016-08-25 09:01:17,464 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/11.75 kb and 5/15 files
2016-08-25 09:01:17,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.11/11.75 kb and 6/15 files
2016-08-25 09:01:17,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/11.75 kb and 7/15 files
2016-08-25 09:01:17,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/11.75 kb and 8/15 files
2016-08-25 09:01:17,466 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:17,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/11.75 kb and 9/15 files
2016-08-25 09:01:17,468 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is starting
2016-08-25 09:01:17,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.16/11.75 kb and 10/15 files
2016-08-25 09:01:17,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.06/11.75 kb and 11/15 files
2016-08-25 09:01:17,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/11.75 kb and 12/15 files
2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55104
2016-08-25 09:01:17,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/11.75 kb and 13/15 files
2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:17,473 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) started
2016-08-25 09:01:17,474 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:17,474 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:17,474 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 14/15 files
2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files
2016-08-25 09:01:17,475 | INFO | hawtdispatch-DEFAULT-1 | Attached
Master started: node-1
======================================
21.4. Browse the queue on new master
======================================
2016-08-25 09:01:17,481 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
21.5. Found
======================================
======================================
22.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:17,494 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is shutting down
2016-08-25 09:01:17,494 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:17,496 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:17,497 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250016
2016-08-25 09:01:17,497 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,498 | INFO | Time-limited test | Session: 0x156bec917250016 closed
2016-08-25 09:01:17,498 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:17,498 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55082 which had sessionid 0x156bec917250016
2016-08-25 09:01:17,505 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:17,505 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:17,505 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:17,505 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:17,627 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:17,628 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) uptime 1.584 seconds
2016-08-25 09:01:17,628 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:23) is shutdown
Wait for master to start up...
2016-08-25 09:01:17,631 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:17,634 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6e57f107
2016-08-25 09:01:17,634 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:17,635 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:17,636 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55106
2016-08-25 09:01:17,636 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:17,636 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55106
2016-08-25 09:01:17,637 | INFO | SyncThread:0 | Established session 0x156bec917250019 with negotiated timeout 10000 for client /127.0.0.1:55106
2016-08-25 09:01:17,637 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250019, negotiated timeout = 10000
2016-08-25 09:01:17,649 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,735 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:17,739 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,746 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:17,746 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:17,756 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:17,764 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,909 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55107
2016-08-25 09:01:17,916 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,917 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55107
2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55107
2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:17,918 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:17,933 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,014 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:18,015 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:18,015 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000034
2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:18,016 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000036.log
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000037.sst
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:18,017 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:18,018 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:18,019 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000034
2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:18,020 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000036.log
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000037.sst
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:18,021 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:18,022 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:18,027 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files
2016-08-25 09:01:18,028 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.51/11.75 kb and 2/15 files
2016-08-25 09:01:18,028 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.41/11.75 kb and 3/15 files
2016-08-25 09:01:18,029 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.31/11.75 kb and 4/15 files
2016-08-25 09:01:18,030 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.21/11.75 kb and 5/15 files
2016-08-25 09:01:18,030 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.16/11.75 kb and 6/15 files
2016-08-25 09:01:18,031 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.34/11.75 kb and 7/15 files
2016-08-25 09:01:18,032 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/11.75 kb and 8/15 files
2016-08-25 09:01:18,032 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/11.75 kb and 1/15 files
2016-08-25 09:01:18,033 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/11.75 kb and 9/15 files
2016-08-25 09:01:18,034 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.51/11.75 kb and 2/15 files
2016-08-25 09:01:18,034 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.26/11.75 kb and 10/15 files
2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.53/11.75 kb and 3/15 files
2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.16/11.75 kb and 11/15 files
2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.06/11.75 kb and 12/15 files
2016-08-25 09:01:18,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.43/11.75 kb and 4/15 files
2016-08-25 09:01:18,036 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files
2016-08-25 09:01:18,036 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/11.75 kb and 5/15 files
2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files
2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/11.75 kb and 6/15 files
2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files
2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.41/11.75 kb and 7/15 files
2016-08-25 09:01:18,037 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:18,038 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/11.75 kb and 8/15 files
2016-08-25 09:01:18,039 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/11.75 kb and 9/15 files
2016-08-25 09:01:18,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.25/11.75 kb and 10/15 files
2016-08-25 09:01:18,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.15/11.75 kb and 11/15 files
2016-08-25 09:01:18,041 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/11.75 kb and 12/15 files
2016-08-25 09:01:18,043 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/11.75 kb and 13/15 files
2016-08-25 09:01:18,044 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,044 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/11.75 kb and 14/15 files
2016-08-25 09:01:18,046 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/11.75 kb and 15/15 files
2016-08-25 09:01:18,046 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:18,047 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is starting
2016-08-25 09:01:18,052 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55112
2016-08-25 09:01:18,052 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) started
2016-08-25 09:01:18,053 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:18,053 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:18,053 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
22.4. Browse the queue on new master
======================================
======================================
22.5. Found
======================================
======================================
23.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:18,071 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is shutting down
2016-08-25 09:01:18,072 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:18,078 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,079 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:18,080 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:18,080 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:18,080 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,081 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:18,081 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250017
2016-08-25 09:01:18,082 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55090 which had sessionid 0x156bec917250017
2016-08-25 09:01:18,083 | INFO | Time-limited test | Session: 0x156bec917250017 closed
2016-08-25 09:01:18,083 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:18,188 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:18,189 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) uptime 1.662 seconds
2016-08-25 09:01:18,189 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:24) is shutdown
Wait for master to start up...
2016-08-25 09:01:18,192 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:18,194 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@727c55d0
2016-08-25 09:01:18,194 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:18,195 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:18,195 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:18,195 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55114
2016-08-25 09:01:18,196 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55114
2016-08-25 09:01:18,196 | INFO | SyncThread:0 | Established session 0x156bec91725001a with negotiated timeout 10000 for client /127.0.0.1:55114
2016-08-25 09:01:18,197 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001a, negotiated timeout = 10000
2016-08-25 09:01:18,205 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:18,340 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,348 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:18,348 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,353 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:18,358 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55115
2016-08-25 09:01:18,492 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,492 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,494 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55115
2016-08-25 09:01:18,494 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:18,495 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55115
2016-08-25 09:01:18,495 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:18,599 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:18,601 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,604 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000037
2016-08-25 09:01:18,605 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:18,606 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000039.log
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:18,607 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000039.log
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:18,608 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000037
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:18,609 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:18,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files
2016-08-25 09:01:18,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.56/12.71 kb and 2/15 files
2016-08-25 09:01:18,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.46/12.71 kb and 3/15 files
2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files
2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.36/12.71 kb and 4/15 files
2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.26/12.71 kb and 5/15 files
2016-08-25 09:01:18,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/12.71 kb and 2/15 files
2016-08-25 09:01:18,620 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/12.71 kb and 3/15 files
2016-08-25 09:01:18,620 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.16/12.71 kb and 6/15 files
2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/12.71 kb and 4/15 files
2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.06/12.71 kb and 7/15 files
2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.96/12.71 kb and 8/15 files
2016-08-25 09:01:18,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/12.71 kb and 5/15 files
2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.86/12.71 kb and 9/15 files
2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.73/12.71 kb and 6/15 files
2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.76/12.71 kb and 10/15 files
2016-08-25 09:01:18,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.63/12.71 kb and 7/15 files
2016-08-25 09:01:18,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.53/12.71 kb and 8/15 files
2016-08-25 09:01:18,623 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.66/12.71 kb and 11/15 files
2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.43/12.71 kb and 9/15 files
2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.84/12.71 kb and 12/15 files
2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/12.71 kb and 10/15 files
2016-08-25 09:01:18,624 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.74/12.71 kb and 13/15 files
2016-08-25 09:01:18,625 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.69/12.71 kb and 14/15 files
2016-08-25 09:01:18,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.22/12.71 kb and 11/15 files
2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files
2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.89/12.71 kb and 12/15 files
2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:18,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.79/12.71 kb and 13/15 files
2016-08-25 09:01:18,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.69/12.71 kb and 14/15 files
2016-08-25 09:01:18,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files
2016-08-25 09:01:18,628 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:18,630 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:18,631 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is starting
2016-08-25 09:01:18,632 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55120
2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) started
2016-08-25 09:01:18,636 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:18,637 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:18,637 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
23.4. Browse the queue on new master
======================================
======================================
23.5. Found
======================================
======================================
24.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:18,663 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is shutting down
2016-08-25 09:01:18,663 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:18,666 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,667 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:18,667 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:18,667 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250018
2016-08-25 09:01:18,667 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:18,668 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:18,668 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:18,668 | INFO | Time-limited test | Session: 0x156bec917250018 closed
2016-08-25 09:01:18,668 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:18,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55098 which had sessionid 0x156bec917250018
2016-08-25 09:01:18,783 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:18,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) uptime 1.850 seconds
2016-08-25 09:01:18,784 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:25) is shutdown
Wait for master to start up...
2016-08-25 09:01:18,787 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:18,790 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3d31af5f
2016-08-25 09:01:18,790 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:18,791 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:18,791 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:18,791 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55122
2016-08-25 09:01:18,792 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55122
2016-08-25 09:01:18,793 | INFO | SyncThread:0 | Established session 0x156bec91725001b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55122
2016-08-25 09:01:18,793 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725001b, negotiated timeout = 10000
2016-08-25 09:01:18,802 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,862 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:18,864 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,865 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:18,865 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:18,870 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:18,873 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,992 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55123
2016-08-25 09:01:18,997 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,997 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55123
2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55123
2016-08-25 09:01:18,999 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:19,016 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:19,113 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:19,115 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:19,115 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000037
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000039.log
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:19,116 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:19,117 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:19,119 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000037
2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:19,120 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000039.log
2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:19,121 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:19,122 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:19,123 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:19,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.66/12.71 kb and 1/15 files
2016-08-25 09:01:19,128 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.61/12.71 kb and 2/15 files
2016-08-25 09:01:19,129 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.51/12.71 kb and 3/15 files
2016-08-25 09:01:19,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.70/12.71 kb and 4/15 files
2016-08-25 09:01:19,130 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.59/12.71 kb and 5/15 files
2016-08-25 09:01:19,131 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.49/12.71 kb and 6/15 files
2016-08-25 09:01:19,132 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.39/12.71 kb and 7/15 files
2016-08-25 09:01:19,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.29/12.71 kb and 8/15 files
2016-08-25 09:01:19,133 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.19/12.71 kb and 9/15 files
2016-08-25 09:01:19,134 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.09/12.71 kb and 10/15 files
2016-08-25 09:01:19,135 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.99/12.71 kb and 11/15 files
2016-08-25 09:01:19,136 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.89/12.71 kb and 12/15 files
2016-08-25 09:01:19,136 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.90/12.71 kb and 13/15 files
2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/12.71 kb and 1/15 files
2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.80/12.71 kb and 14/15 files
2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.56/12.71 kb and 2/15 files
2016-08-25 09:01:19,137 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files
2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.46/12.71 kb and 3/15 files
2016-08-25 09:01:19,138 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.36/12.71 kb and 4/15 files
2016-08-25 09:01:19,139 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.54/12.71 kb and 5/15 files
2016-08-25 09:01:19,140 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.44/12.71 kb and 6/15 files
2016-08-25 09:01:19,141 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.34/12.71 kb and 7/15 files
2016-08-25 09:01:19,142 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.24/12.71 kb and 8/15 files
2016-08-25 09:01:19,142 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.14/12.71 kb and 9/15 files
2016-08-25 09:01:19,143 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.15/12.71 kb and 10/15 files
2016-08-25 09:01:19,144 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.06/12.71 kb and 11/15 files
2016-08-25 09:01:19,145 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.01/12.71 kb and 12/15 files
2016-08-25 09:01:19,145 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.91/12.71 kb and 13/15 files
2016-08-25 09:01:19,146 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.81/12.71 kb and 14/15 files
2016-08-25 09:01:19,148 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.71/12.71 kb and 15/15 files
2016-08-25 09:01:19,148 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:19,152 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:19,152 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:19,154 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is starting
2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55128
2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) started
2016-08-25 09:01:19,159 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:19,160 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:19,160 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
24.4. Browse the queue on new master
======================================
======================================
24.5. Found
======================================
======================================
25.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:19,179 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is shutting down
2016-08-25 09:01:19,179 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:19,183 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:19,183 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:19,183 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:19,184 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:19,184 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:19,185 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:19,185 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250019
2016-08-25 09:01:19,186 | INFO | Time-limited test | Session: 0x156bec917250019 closed
2016-08-25 09:01:19,186 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:19,186 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55106 which had sessionid 0x156bec917250019
2016-08-25 09:01:19,290 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:19,291 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) uptime 1.661 seconds
2016-08-25 09:01:19,291 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:26) is shutdown
Wait for master to start up...
2016-08-25 09:01:19,294 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:19,298 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@78f004c9
2016-08-25 09:01:19,298 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:19,299 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:19,299 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:19,300 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55130
2016-08-25 09:01:19,300 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55130
2016-08-25 09:01:19,301 | INFO | SyncThread:0 | Established session 0x156bec91725001c with negotiated timeout 10000 for client /127.0.0.1:55130
2016-08-25 09:01:19,301 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001c, negotiated timeout = 10000
2016-08-25 09:01:19,314 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:19,407 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:19,409 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:19,507 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:19,507 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:19,513 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:19,522 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:19,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55131
2016-08-25 09:01:19,636 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:19,636 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55131
2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55131
2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:19,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:19,720 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:19,720 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:19,724 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:19,724 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:19,725 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000042.log
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000042.log
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000040
2016-08-25 09:01:19,726 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000040
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:19,727 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:19,736 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/13.66 kb and 1/16 files
2016-08-25 09:01:19,737 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/13.66 kb and 2/16 files
2016-08-25 09:01:19,737 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/13.66 kb and 3/16 files
2016-08-25 09:01:19,738 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/13.66 kb and 4/16 files
2016-08-25 09:01:19,739 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.79/13.66 kb and 5/16 files
2016-08-25 09:01:19,739 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/13.66 kb and 6/16 files
2016-08-25 09:01:19,740 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.70/13.66 kb and 7/16 files
2016-08-25 09:01:19,741 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.65/13.66 kb and 8/16 files
2016-08-25 09:01:19,742 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.55/13.66 kb and 9/16 files
2016-08-25 09:01:19,743 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.45/13.66 kb and 10/16 files
2016-08-25 09:01:19,744 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/13.66 kb and 11/16 files
2016-08-25 09:01:19,745 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.06/13.66 kb and 12/16 files
2016-08-25 09:01:19,746 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.96/13.66 kb and 13/16 files
2016-08-25 09:01:19,747 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.86/13.66 kb and 14/16 files
2016-08-25 09:01:19,748 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files
2016-08-25 09:01:19,749 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files
2016-08-25 09:01:19,749 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:19,753 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:19,753 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/13.66 kb and 1/16 files
2016-08-25 09:01:19,754 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/13.66 kb and 2/16 files
2016-08-25 09:01:19,754 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is starting
2016-08-25 09:01:19,755 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.69/13.66 kb and 3/16 files
2016-08-25 09:01:19,755 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.59/13.66 kb and 4/16 files
2016-08-25 09:01:19,756 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/13.66 kb and 5/16 files
2016-08-25 09:01:19,757 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.40/13.66 kb and 6/16 files
2016-08-25 09:01:19,758 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.35/13.66 kb and 7/16 files
2016-08-25 09:01:19,758 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.53/13.66 kb and 8/16 files
2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55136
2016-08-25 09:01:19,759 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.24/13.66 kb and 9/16 files
2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) started
2016-08-25 09:01:19,759 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:19,759 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:19,760 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.14/13.66 kb and 10/16 files
2016-08-25 09:01:19,760 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:19,760 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.04/13.66 kb and 11/16 files
2016-08-25 09:01:19,761 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.94/13.66 kb and 12/16 files
2016-08-25 09:01:19,762 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.84/13.66 kb and 13/16 files
Master started: node-3
======================================
25.4. Browse the queue on new master
======================================
2016-08-25 09:01:19,762 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.74/13.66 kb and 14/16 files
2016-08-25 09:01:19,763 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files
2016-08-25 09:01:19,764 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files
2016-08-25 09:01:19,764 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:19,768 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
25.5. Found
======================================
======================================
26.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:19,781 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is shutting down
2016-08-25 09:01:19,783 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:19,788 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:19,789 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:19,789 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:19,789 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:19,790 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:19,790 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:19,791 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001a
2016-08-25 09:01:19,792 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55114 which had sessionid 0x156bec91725001a
2016-08-25 09:01:19,792 | INFO | Time-limited test | Session: 0x156bec91725001a closed
2016-08-25 09:01:19,792 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:19,913 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:19,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) uptime 1.723 seconds
2016-08-25 09:01:19,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:27) is shutdown
Wait for master to start up...
2016-08-25 09:01:19,916 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:19,919 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3512e6d9
2016-08-25 09:01:19,919 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:19,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:19,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:19,921 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55138
2016-08-25 09:01:19,921 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55138
2016-08-25 09:01:19,922 | INFO | SyncThread:0 | Established session 0x156bec91725001d with negotiated timeout 10000 for client /127.0.0.1:55138
2016-08-25 09:01:19,922 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001d, negotiated timeout = 10000
2016-08-25 09:01:19,932 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,033 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:20,035 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,045 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:20,045 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,050 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:20,056 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,178 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55139
2016-08-25 09:01:20,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,186 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55139
2016-08-25 09:01:20,186 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:20,187 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,190 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55139
2016-08-25 09:01:20,191 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:20,207 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:20,212 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:20,212 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000042.log
2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:20,213 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000040
2016-08-25 09:01:20,214 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:20,215 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:20,225 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/13.66 kb and 1/16 files
2016-08-25 09:01:20,226 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/13.66 kb and 2/16 files
2016-08-25 09:01:20,227 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/13.66 kb and 3/16 files
2016-08-25 09:01:20,228 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.88/13.66 kb and 4/16 files
2016-08-25 09:01:20,231 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.78/13.66 kb and 5/16 files
2016-08-25 09:01:20,232 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/13.66 kb and 6/16 files
2016-08-25 09:01:20,232 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/13.66 kb and 7/16 files
2016-08-25 09:01:20,233 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.65/13.66 kb and 8/16 files
2016-08-25 09:01:20,234 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.55/13.66 kb and 9/16 files
2016-08-25 09:01:20,235 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.45/13.66 kb and 10/16 files
2016-08-25 09:01:20,236 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.35/13.66 kb and 11/16 files
2016-08-25 09:01:20,236 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.06/13.66 kb and 12/16 files
2016-08-25 09:01:20,237 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.96/13.66 kb and 13/16 files
2016-08-25 09:01:20,238 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.86/13.66 kb and 14/16 files
2016-08-25 09:01:20,238 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.76/13.66 kb and 15/16 files
2016-08-25 09:01:20,239 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.66/13.66 kb and 16/16 files
2016-08-25 09:01:20,239 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:20,242 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:20,243 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is starting
2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55143
2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:20,248 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) started
2016-08-25 09:01:20,249 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:20,249 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:20,249 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
26.4. Browse the queue on new master
======================================
======================================
26.5. Found
======================================
======================================
27.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:20,267 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is shutting down
2016-08-25 09:01:20,268 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:20,271 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,271 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:20,272 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001b
2016-08-25 09:01:20,272 | INFO | Time-limited test | Session: 0x156bec91725001b closed
2016-08-25 09:01:20,272 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:20,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55122 which had sessionid 0x156bec91725001b
2016-08-25 09:01:20,275 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:20,275 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:20,338 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:20,377 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:20,378 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) uptime 1.593 seconds
2016-08-25 09:01:20,378 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:28) is shutdown
Wait for master to start up...
2016-08-25 09:01:20,381 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:20,383 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@72133137
2016-08-25 09:01:20,383 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:20,385 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:20,385 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:20,385 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55145
2016-08-25 09:01:20,386 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55145
2016-08-25 09:01:20,386 | INFO | SyncThread:0 | Established session 0x156bec91725001e with negotiated timeout 10000 for client /127.0.0.1:55145
2016-08-25 09:01:20,386 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725001e, negotiated timeout = 10000
2016-08-25 09:01:20,394 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,462 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:20,463 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,465 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,620 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:20,624 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:20,629 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,757 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55146
2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55146
2016-08-25 09:01:20,762 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:20,764 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:20,765 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55146
2016-08-25 09:01:20,765 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:20,866 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:20,867 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:20,873 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:20,873 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:20,874 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:20,875 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000045.log
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:20,876 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000045.log
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:20,877 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000043
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000043
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:20,878 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/14.61 kb and 1/17 files
2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/14.61 kb and 1/17 files
2016-08-25 09:01:20,891 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/14.61 kb and 2/17 files
2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/14.61 kb and 2/17 files
2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/14.61 kb and 3/17 files
2016-08-25 09:01:20,892 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/14.61 kb and 3/17 files
2016-08-25 09:01:20,893 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/14.61 kb and 4/17 files
2016-08-25 09:01:20,893 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/14.61 kb and 4/17 files
2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/14.61 kb and 5/17 files
2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/14.61 kb and 5/17 files
2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/14.61 kb and 6/17 files
2016-08-25 09:01:20,894 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/14.61 kb and 6/17 files
2016-08-25 09:01:20,895 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/14.61 kb and 7/17 files
2016-08-25 09:01:20,895 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/14.61 kb and 7/17 files
2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/14.61 kb and 8/17 files
2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/14.61 kb and 8/17 files
2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.26/14.61 kb and 9/17 files
2016-08-25 09:01:20,896 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/14.61 kb and 9/17 files
2016-08-25 09:01:20,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.45/14.61 kb and 10/17 files
2016-08-25 09:01:20,897 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.39/14.61 kb and 10/17 files
2016-08-25 09:01:20,898 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.16/14.61 kb and 11/17 files
2016-08-25 09:01:20,898 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.35/14.61 kb and 11/17 files
2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/14.61 kb and 12/17 files
2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.10/14.61 kb and 12/17 files
2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.14/14.61 kb and 13/17 files
2016-08-25 09:01:20,899 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/14.61 kb and 13/17 files
2016-08-25 09:01:20,900 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.04/14.61 kb and 14/17 files
2016-08-25 09:01:20,900 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/14.61 kb and 14/17 files
2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/14.61 kb and 15/17 files
2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/14.61 kb and 15/17 files
2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.71/14.61 kb and 16/17 files
2016-08-25 09:01:20,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/14.61 kb and 16/17 files
2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files
2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files
2016-08-25 09:01:20,902 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:20,903 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:20,908 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:20,908 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:20,909 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is starting
2016-08-25 09:01:20,914 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55151
2016-08-25 09:01:20,914 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:20,915 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) started
2016-08-25 09:01:20,915 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:20,915 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:20,915 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
27.4. Browse the queue on new master
======================================
======================================
27.5. Found
======================================
======================================
28.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:20,942 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is shutting down
2016-08-25 09:01:20,944 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:20,949 | INFO | Starting broker node: node-1-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:20,949 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,950 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:20,950 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:20,950 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:20,950 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:20,953 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:20,954 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001d
2016-08-25 09:01:20,955 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55138 which had sessionid 0x156bec91725001d
2016-08-25 09:01:20,955 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:20,955 | INFO | Time-limited test | Session: 0x156bec91725001d closed
2016-08-25 09:01:20,955 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:21,073 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:21,074 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) uptime 1.160 seconds
2016-08-25 09:01:21,074 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:29) is shutdown
Wait for master to start up...
2016-08-25 09:01:21,077 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:21,079 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4dca593e
2016-08-25 09:01:21,079 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:21,080 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:21,081 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:21,081 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55153
2016-08-25 09:01:21,081 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55153
2016-08-25 09:01:21,082 | INFO | SyncThread:0 | Established session 0x156bec91725001f with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55153
2016-08-25 09:01:21,082 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725001f, negotiated timeout = 10000
2016-08-25 09:01:21,090 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,190 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:21,190 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:21,191 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,192 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,196 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:21,199 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,331 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55154
2016-08-25 09:01:21,337 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,337 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55154
2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55154
2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:21,338 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:21,353 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:21,450 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:21,453 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000045.log
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:21,455 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000043
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:21,456 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:21,457 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:21,458 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:21,459 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000043
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:21,460 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000045.log
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:21,461 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.95/14.61 kb and 1/17 files
2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/14.61 kb and 1/17 files
2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/14.61 kb and 2/17 files
2016-08-25 09:01:21,472 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/14.61 kb and 2/17 files
2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.10/14.61 kb and 3/17 files
2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.86/14.61 kb and 3/17 files
2016-08-25 09:01:21,473 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.00/14.61 kb and 4/17 files
2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.76/14.61 kb and 4/17 files
2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.66/14.61 kb and 5/17 files
2016-08-25 09:01:21,474 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.90/14.61 kb and 5/17 files
2016-08-25 09:01:21,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/14.61 kb and 6/17 files
2016-08-25 09:01:21,475 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/14.61 kb and 6/17 files
2016-08-25 09:01:21,476 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/14.61 kb and 7/17 files
2016-08-25 09:01:21,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.46/14.61 kb and 7/17 files
2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.60/14.61 kb and 8/17 files
2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.23/14.61 kb and 8/17 files
2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.13/14.61 kb and 9/17 files
2016-08-25 09:01:21,477 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.50/14.61 kb and 9/17 files
2016-08-25 09:01:21,478 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.40/14.61 kb and 10/17 files
2016-08-25 09:01:21,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/14.61 kb and 10/17 files
2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/14.61 kb and 11/17 files
2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.29/14.61 kb and 11/17 files
2016-08-25 09:01:21,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/14.61 kb and 12/17 files
2016-08-25 09:01:21,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/14.61 kb and 13/17 files
2016-08-25 09:01:21,480 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.06/14.61 kb and 12/17 files
2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.96/14.61 kb and 13/17 files
2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/14.61 kb and 14/17 files
2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/14.61 kb and 15/17 files
2016-08-25 09:01:21,481 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.91/14.61 kb and 14/17 files
2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.70/14.61 kb and 16/17 files
2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.81/14.61 kb and 15/17 files
2016-08-25 09:01:21,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files
2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.71/14.61 kb and 16/17 files
2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.61/14.61 kb and 17/17 files
2016-08-25 09:01:21,483 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:21,486 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:21,487 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is starting
2016-08-25 09:01:21,487 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:21,491 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55159
2016-08-25 09:01:21,491 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:21,492 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) started
2016-08-25 09:01:21,492 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:21,492 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:21,492 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
28.4. Browse the queue on new master
======================================
======================================
28.5. Found
======================================
======================================
29.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:21,511 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is shutting down
2016-08-25 09:01:21,511 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:21,515 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,515 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:21,516 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001c
2016-08-25 09:01:21,516 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:21,516 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:21,517 | INFO | Time-limited test | Session: 0x156bec91725001c closed
2016-08-25 09:01:21,517 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:21,517 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55130 which had sessionid 0x156bec91725001c
2016-08-25 09:01:21,517 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:21,517 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:21,623 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:21,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) uptime 2.332 seconds
2016-08-25 09:01:21,624 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:30) is shutdown
Wait for master to start up...
2016-08-25 09:01:21,626 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:21,629 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@27b68be7
2016-08-25 09:01:21,629 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:21,630 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:21,631 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:21,631 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55161
2016-08-25 09:01:21,631 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55161
2016-08-25 09:01:21,632 | INFO | SyncThread:0 | Established session 0x156bec917250020 with negotiated timeout 10000 for client /127.0.0.1:55161
2016-08-25 09:01:21,632 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250020, negotiated timeout = 10000
2016-08-25 09:01:21,640 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,844 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:21,845 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,847 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:21,849 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:21,854 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:21,860 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,966 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55162
2016-08-25 09:01:21,971 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,971 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:21,972 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55162
2016-08-25 09:01:21,972 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:21,974 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55162
2016-08-25 09:01:21,974 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:22,064 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:22,074 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:22,074 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000048.log
2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:22,075 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000046
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:22,076 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:22,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:22,083 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:22,084 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000046
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:22,085 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000048.log
2016-08-25 09:01:22,086 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:22,094 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/15.56 kb and 1/18 files
2016-08-25 09:01:22,095 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.09/15.56 kb and 2/18 files
2016-08-25 09:01:22,095 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.90/15.56 kb and 3/18 files
2016-08-25 09:01:22,096 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.80/15.56 kb and 4/18 files
2016-08-25 09:01:22,097 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.70/15.56 kb and 5/18 files
2016-08-25 09:01:22,097 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.60/15.56 kb and 6/18 files
2016-08-25 09:01:22,098 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.50/15.56 kb and 7/18 files
2016-08-25 09:01:22,099 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.40/15.56 kb and 8/18 files
2016-08-25 09:01:22,099 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.29/15.56 kb and 9/18 files
2016-08-25 09:01:22,100 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.31/15.56 kb and 10/18 files
2016-08-25 09:01:22,101 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.26/15.56 kb and 11/18 files
2016-08-25 09:01:22,102 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.16/15.56 kb and 12/18 files
2016-08-25 09:01:22,102 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.06/15.56 kb and 13/18 files
2016-08-25 09:01:22,103 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.96/15.56 kb and 14/18 files
2016-08-25 09:01:22,104 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.86/15.56 kb and 15/18 files
2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.76/15.56 kb and 16/18 files
2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files
2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/15.56 kb and 2/18 files
2016-08-25 09:01:22,105 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files
2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/15.56 kb and 3/18 files
2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files
2016-08-25 09:01:22,106 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:22,107 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.65/15.56 kb and 4/18 files
2016-08-25 09:01:22,107 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.47/15.56 kb and 5/18 files
2016-08-25 09:01:22,108 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.37/15.56 kb and 6/18 files
2016-08-25 09:01:22,109 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.26/15.56 kb and 7/18 files
2016-08-25 09:01:22,109 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.16/15.56 kb and 8/18 files
2016-08-25 09:01:22,110 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.18/15.56 kb and 9/18 files
2016-08-25 09:01:22,110 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:22,111 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.08/15.56 kb and 10/18 files
2016-08-25 09:01:22,111 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is starting
2016-08-25 09:01:22,112 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.97/15.56 kb and 11/18 files
2016-08-25 09:01:22,112 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.87/15.56 kb and 12/18 files
2016-08-25 09:01:22,113 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.77/15.56 kb and 13/18 files
2016-08-25 09:01:22,115 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.67/15.56 kb and 14/18 files
2016-08-25 09:01:22,116 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.57/15.56 kb and 15/18 files
2016-08-25 09:01:22,117 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.47/15.56 kb and 16/18 files
2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55167
2016-08-25 09:01:22,118 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files
2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) started
2016-08-25 09:01:22,118 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:22,118 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:22,119 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files
2016-08-25 09:01:22,119 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:22,119 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
29.4. Browse the queue on new master
======================================
2016-08-25 09:01:22,124 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
29.5. Found
======================================
======================================
30.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:22,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is shutting down
2016-08-25 09:01:22,142 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:22,144 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:22,145 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,145 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001e
2016-08-25 09:01:22,146 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:22,146 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:22,146 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55145 which had sessionid 0x156bec91725001e
2016-08-25 09:01:22,146 | INFO | Time-limited test | Session: 0x156bec91725001e closed
2016-08-25 09:01:22,146 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:22,151 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:22,151 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:22,262 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:22,263 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) uptime 1.884 seconds
2016-08-25 09:01:22,263 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:31) is shutdown
Wait for master to start up...
2016-08-25 09:01:22,265 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:22,269 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4264a42e
2016-08-25 09:01:22,270 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:22,270 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:22,271 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:22,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55169
2016-08-25 09:01:22,271 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55169
2016-08-25 09:01:22,272 | INFO | SyncThread:0 | Established session 0x156bec917250021 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55169
2016-08-25 09:01:22,272 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250021, negotiated timeout = 10000
2016-08-25 09:01:22,282 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,362 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:22,366 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,368 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:22,369 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,375 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:22,384 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:22,493 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55170
2016-08-25 09:01:22,500 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:22,500 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:22,502 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55170
2016-08-25 09:01:22,502 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:22,503 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55170
2016-08-25 09:01:22,503 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:22,515 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:22,591 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:22,592 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:22,594 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000048.log
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:22,595 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000046
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000046
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:22,596 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:22,597 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000048.log
2016-08-25 09:01:22,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files
2016-08-25 09:01:22,605 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/15.56 kb and 2/18 files
2016-08-25 09:01:22,606 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/15.56 kb and 3/18 files
2016-08-25 09:01:22,606 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/15.56 kb and 4/18 files
2016-08-25 09:01:22,607 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/15.56 kb and 5/18 files
2016-08-25 09:01:22,608 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/15.56 kb and 6/18 files
2016-08-25 09:01:22,609 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/15.56 kb and 7/18 files
2016-08-25 09:01:22,609 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/15.56 kb and 8/18 files
2016-08-25 09:01:22,610 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/15.56 kb and 9/18 files
2016-08-25 09:01:22,611 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/15.56 kb and 10/18 files
2016-08-25 09:01:22,612 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.34/15.56 kb and 11/18 files
2016-08-25 09:01:22,613 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/15.56 kb and 12/18 files
2016-08-25 09:01:22,614 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.14/15.56 kb and 13/18 files
2016-08-25 09:01:22,615 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.04/15.56 kb and 14/18 files
2016-08-25 09:01:22,616 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/15.56 kb and 15/18 files
2016-08-25 09:01:22,616 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/15.56 kb and 1/18 files
2016-08-25 09:01:22,617 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/15.56 kb and 16/18 files
2016-08-25 09:01:22,617 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/15.56 kb and 2/18 files
2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.66/15.56 kb and 17/18 files
2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files
2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/15.56 kb and 3/18 files
2016-08-25 09:01:22,618 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:22,619 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/15.56 kb and 4/18 files
2016-08-25 09:01:22,620 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/15.56 kb and 5/18 files
2016-08-25 09:01:22,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.43/15.56 kb and 6/18 files
2016-08-25 09:01:22,621 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.33/15.56 kb and 7/18 files
2016-08-25 09:01:22,622 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.22/15.56 kb and 8/18 files
2016-08-25 09:01:22,622 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:22,623 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.12/15.56 kb and 9/18 files
2016-08-25 09:01:22,624 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/15.56 kb and 10/18 files
2016-08-25 09:01:22,624 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is starting
2016-08-25 09:01:22,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/15.56 kb and 11/18 files
2016-08-25 09:01:22,625 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/15.56 kb and 12/18 files
2016-08-25 09:01:22,626 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.72/15.56 kb and 13/18 files
2016-08-25 09:01:22,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.62/15.56 kb and 14/18 files
2016-08-25 09:01:22,627 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.52/15.56 kb and 15/18 files
2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55175
2016-08-25 09:01:22,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.47/15.56 kb and 16/18 files
2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) started
2016-08-25 09:01:22,628 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:22,628 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:22,628 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.37/15.56 kb and 17/18 files
2016-08-25 09:01:22,628 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:22,629 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/15.56 kb and 18/18 files
2016-08-25 09:01:22,629 | INFO | hawtdispatch-DEFAULT-1 | Attached
Master started: node-3
======================================
30.4. Browse the queue on new master
======================================
2016-08-25 09:01:22,638 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
30.5. Found
======================================
======================================
31.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:22,650 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is shutting down
2016-08-25 09:01:22,651 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:22,654 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,656 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:22,656 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725001f
2016-08-25 09:01:22,657 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55153 which had sessionid 0x156bec91725001f
2016-08-25 09:01:22,658 | INFO | Time-limited test | Session: 0x156bec91725001f closed
2016-08-25 09:01:22,658 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:22,658 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:22,658 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:22,658 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:22,658 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:22,772 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:22,773 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) uptime 1.698 seconds
2016-08-25 09:01:22,773 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:32) is shutdown
Wait for master to start up...
2016-08-25 09:01:22,780 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:22,782 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@21a086ed
2016-08-25 09:01:22,782 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:22,783 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:22,784 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:22,784 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55177
2016-08-25 09:01:22,784 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55177
2016-08-25 09:01:22,785 | INFO | SyncThread:0 | Established session 0x156bec917250022 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55177
2016-08-25 09:01:22,785 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250022, negotiated timeout = 10000
2016-08-25 09:01:22,799 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:22,980 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:22,982 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,017 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:23,017 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,023 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:23,030 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,158 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55178
2016-08-25 09:01:23,163 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,163 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,164 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55178
2016-08-25 09:01:23,165 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:23,167 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55178
2016-08-25 09:01:23,167 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:23,274 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:23,278 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:23,278 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000051.log
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000052.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:23,279 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:23,280 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000049
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:23,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:23,285 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:23,289 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000051.log
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000052.sst
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:23,290 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:23,291 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/16.51 kb and 1/20 files
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000049
2016-08-25 09:01:23,292 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/16.51 kb and 2/20 files
2016-08-25 09:01:23,293 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/16.51 kb and 3/20 files
2016-08-25 09:01:23,293 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/16.51 kb and 4/20 files
2016-08-25 09:01:23,294 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.88/16.51 kb and 5/20 files
2016-08-25 09:01:23,294 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files
2016-08-25 09:01:23,295 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files
2016-08-25 09:01:23,296 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files
2016-08-25 09:01:23,297 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files
2016-08-25 09:01:23,297 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files
2016-08-25 09:01:23,298 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.28/16.51 kb and 11/20 files
2016-08-25 09:01:23,298 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.18/16.51 kb and 12/20 files
2016-08-25 09:01:23,299 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.04/16.51 kb and 13/20 files
2016-08-25 09:01:23,300 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.99/16.51 kb and 14/20 files
2016-08-25 09:01:23,300 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.89/16.51 kb and 15/20 files
2016-08-25 09:01:23,301 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.79/16.51 kb and 16/20 files
2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.69/16.51 kb and 17/20 files
2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/16.51 kb and 1/20 files
2016-08-25 09:01:23,302 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.59/16.51 kb and 18/20 files
2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.08/16.51 kb and 2/20 files
2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.61/16.51 kb and 19/20 files
2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files
2016-08-25 09:01:23,303 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/16.51 kb and 3/20 files
2016-08-25 09:01:23,304 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:23,304 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/16.51 kb and 4/20 files
2016-08-25 09:01:23,305 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.88/16.51 kb and 5/20 files
2016-08-25 09:01:23,306 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files
2016-08-25 09:01:23,306 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files
2016-08-25 09:01:23,307 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files
2016-08-25 09:01:23,307 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files
2016-08-25 09:01:23,308 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files
2016-08-25 09:01:23,308 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:23,309 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/16.51 kb and 11/20 files
2016-08-25 09:01:23,309 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is starting
2016-08-25 09:01:23,310 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.35/16.51 kb and 12/20 files
2016-08-25 09:01:23,310 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.25/16.51 kb and 13/20 files
2016-08-25 09:01:23,311 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.15/16.51 kb and 14/20 files
2016-08-25 09:01:23,312 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/16.51 kb and 15/20 files
2016-08-25 09:01:23,313 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/16.51 kb and 16/20 files
2016-08-25 09:01:23,313 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/16.51 kb and 17/20 files
2016-08-25 09:01:23,314 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/16.51 kb and 18/20 files
2016-08-25 09:01:23,314 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55183
2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) started
2016-08-25 09:01:23,315 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:23,315 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.64/16.51 kb and 19/20 files
2016-08-25 09:01:23,315 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:23,315 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:23,316 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files
2016-08-25 09:01:23,316 | INFO | hawtdispatch-DEFAULT-1 | Attached
Master started: node-1
======================================
31.4. Browse the queue on new master
======================================
2016-08-25 09:01:23,322 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
31.5. Found
======================================
======================================
32.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:23,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is shutting down
2016-08-25 09:01:23,341 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:23,346 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,349 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:23,350 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250020
2016-08-25 09:01:23,351 | INFO | Time-limited test | Session: 0x156bec917250020 closed
2016-08-25 09:01:23,351 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55161 which had sessionid 0x156bec917250020
2016-08-25 09:01:23,351 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:23,352 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:23,352 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:23,352 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:23,353 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:23,473 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:23,474 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) uptime 1.850 seconds
2016-08-25 09:01:23,474 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:33) is shutdown
Wait for master to start up...
2016-08-25 09:01:23,476 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:23,479 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@54cc6586
2016-08-25 09:01:23,479 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:23,480 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:23,480 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:23,480 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55185
2016-08-25 09:01:23,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55185
2016-08-25 09:01:23,482 | INFO | SyncThread:0 | Established session 0x156bec917250023 with negotiated timeout 10000 for client /127.0.0.1:55185
2016-08-25 09:01:23,482 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250023, negotiated timeout = 10000
2016-08-25 09:01:23,492 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,588 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:23,590 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,595 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:23,595 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,599 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:23,603 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,728 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55186
2016-08-25 09:01:23,736 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,737 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55186
2016-08-25 09:01:23,737 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:23,738 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:23,739 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55186
2016-08-25 09:01:23,739 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:23,758 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:23,762 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000052.sst
2016-08-25 09:01:23,763 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000051.log
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:23,764 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000049
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:23,765 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:23,780 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/16.51 kb and 1/20 files
2016-08-25 09:01:23,781 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/16.51 kb and 2/20 files
2016-08-25 09:01:23,781 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/16.51 kb and 3/20 files
2016-08-25 09:01:23,782 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/16.51 kb and 4/20 files
2016-08-25 09:01:23,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/16.51 kb and 5/20 files
2016-08-25 09:01:23,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/16.51 kb and 6/20 files
2016-08-25 09:01:23,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/16.51 kb and 7/20 files
2016-08-25 09:01:23,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/16.51 kb and 8/20 files
2016-08-25 09:01:23,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/16.51 kb and 9/20 files
2016-08-25 09:01:23,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/16.51 kb and 10/20 files
2016-08-25 09:01:23,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/16.51 kb and 11/20 files
2016-08-25 09:01:23,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.23/16.51 kb and 12/20 files
2016-08-25 09:01:23,789 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.24/16.51 kb and 13/20 files
2016-08-25 09:01:23,790 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.14/16.51 kb and 14/20 files
2016-08-25 09:01:23,790 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.01/16.51 kb and 15/20 files
2016-08-25 09:01:23,791 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.91/16.51 kb and 16/20 files
2016-08-25 09:01:23,792 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.81/16.51 kb and 17/20 files
2016-08-25 09:01:23,792 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.71/16.51 kb and 18/20 files
2016-08-25 09:01:23,793 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.61/16.51 kb and 19/20 files
2016-08-25 09:01:23,794 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.51/16.51 kb and 20/20 files
2016-08-25 09:01:23,794 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:23,796 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:23,797 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is starting
2016-08-25 09:01:23,801 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55190
2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) started
2016-08-25 09:01:23,802 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:23,803 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:23,803 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
32.4. Browse the queue on new master
======================================
======================================
32.5. Found
======================================
======================================
33.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:23,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is shutting down
2016-08-25 09:01:23,826 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:23,829 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:23,829 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:23,830 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:23,830 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:23,831 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250021
2016-08-25 09:01:23,831 | INFO | Time-limited test | Session: 0x156bec917250021 closed
2016-08-25 09:01:23,831 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:23,832 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55169 which had sessionid 0x156bec917250021
2016-08-25 09:01:23,935 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:23,936 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) uptime 1.672 seconds
2016-08-25 09:01:23,936 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:34) is shutdown
Wait for master to start up...
2016-08-25 09:01:23,938 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:23,941 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@50a4a6f0
2016-08-25 09:01:23,941 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:23,942 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:23,943 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:23,943 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55192
2016-08-25 09:01:23,944 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55192
2016-08-25 09:01:23,944 | INFO | SyncThread:0 | Established session 0x156bec917250024 with negotiated timeout 10000 for client /127.0.0.1:55192
2016-08-25 09:01:23,945 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250024, negotiated timeout = 10000
2016-08-25 09:01:23,957 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,007 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:24,007 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:24,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,010 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,159 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:24,165 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:24,170 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,287 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55193
2016-08-25 09:01:24,293 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,294 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55193
2016-08-25 09:01:24,294 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:24,295 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,296 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55193
2016-08-25 09:01:24,296 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:24,389 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:24,393 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:24,394 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:24,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:24,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:24,396 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000052
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000054.log
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:24,397 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000054.log
2016-08-25 09:01:24,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000052
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:24,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:24,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:24,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files
2016-08-25 09:01:24,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/17.46 kb and 2/20 files
2016-08-25 09:01:24,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/17.46 kb and 3/20 files
2016-08-25 09:01:24,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/17.46 kb and 4/20 files
2016-08-25 09:01:24,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files
2016-08-25 09:01:24,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files
2016-08-25 09:01:24,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.29/17.46 kb and 7/20 files
2016-08-25 09:01:24,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/17.46 kb and 8/20 files
2016-08-25 09:01:24,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.11/17.46 kb and 9/20 files
2016-08-25 09:01:24,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.29/17.46 kb and 10/20 files
2016-08-25 09:01:24,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.19/17.46 kb and 11/20 files
2016-08-25 09:01:24,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/17.46 kb and 12/20 files
2016-08-25 09:01:24,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/17.46 kb and 13/20 files
2016-08-25 09:01:24,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/17.46 kb and 14/20 files
2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.90/17.46 kb and 15/20 files
2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files
2016-08-25 09:01:24,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.80/17.46 kb and 16/20 files
2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/17.46 kb and 2/20 files
2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.71/17.46 kb and 17/20 files
2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/17.46 kb and 3/20 files
2016-08-25 09:01:24,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.61/17.46 kb and 18/20 files
2016-08-25 09:01:24,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.51/17.46 kb and 19/20 files
2016-08-25 09:01:24,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/17.46 kb and 4/20 files
2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files
2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files
2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:24,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files
2016-08-25 09:01:24,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.35/17.46 kb and 7/20 files
2016-08-25 09:01:24,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.25/17.46 kb and 8/20 files
2016-08-25 09:01:24,424 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.15/17.46 kb and 9/20 files
2016-08-25 09:01:24,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/17.46 kb and 10/20 files
2016-08-25 09:01:24,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.23/17.46 kb and 11/20 files
2016-08-25 09:01:24,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.15/17.46 kb and 12/20 files
2016-08-25 09:01:24,427 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:24,427 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/17.46 kb and 13/20 files
2016-08-25 09:01:24,428 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.95/17.46 kb and 14/20 files
2016-08-25 09:01:24,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is starting
2016-08-25 09:01:24,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.85/17.46 kb and 15/20 files
2016-08-25 09:01:24,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.75/17.46 kb and 16/20 files
2016-08-25 09:01:24,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.76/17.46 kb and 17/20 files
2016-08-25 09:01:24,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.66/17.46 kb and 18/20 files
2016-08-25 09:01:24,431 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.56/17.46 kb and 19/20 files
2016-08-25 09:01:24,432 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files
2016-08-25 09:01:24,432 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55198
2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) started
2016-08-25 09:01:24,433 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:24,433 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:24,433 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
33.4. Browse the queue on new master
======================================
2016-08-25 09:01:24,438 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
33.5. Found
======================================
======================================
34.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:24,457 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is shutting down
2016-08-25 09:01:24,459 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:24,463 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:24,463 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,465 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:24,466 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:24,466 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:24,466 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:24,471 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:24,473 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,473 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250023
2016-08-25 09:01:24,474 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55185 which had sessionid 0x156bec917250023
2016-08-25 09:01:24,475 | INFO | Time-limited test | Session: 0x156bec917250023 closed
2016-08-25 09:01:24,475 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:24,591 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:24,592 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) uptime 1.118 seconds
2016-08-25 09:01:24,592 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:35) is shutdown
Wait for master to start up...
2016-08-25 09:01:24,594 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:24,597 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6442aec9
2016-08-25 09:01:24,597 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:24,598 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:24,598 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:24,598 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55200
2016-08-25 09:01:24,599 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55200
2016-08-25 09:01:24,600 | INFO | SyncThread:0 | Established session 0x156bec917250025 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55200
2016-08-25 09:01:24,600 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250025, negotiated timeout = 10000
2016-08-25 09:01:24,607 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,696 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:24,696 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,699 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:24,705 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:24,709 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:24,712 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,827 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55201
2016-08-25 09:01:24,834 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,834 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:24,835 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55201
2016-08-25 09:01:24,835 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:24,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55201
2016-08-25 09:01:24,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:24,944 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:24,944 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:24,947 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000052
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:24,948 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000052
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:24,949 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000054.log
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000054.log
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:24,950 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/17.46 kb and 1/20 files
2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/17.46 kb and 1/20 files
2016-08-25 09:01:24,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/17.46 kb and 2/20 files
2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/17.46 kb and 2/20 files
2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.83/17.46 kb and 3/20 files
2016-08-25 09:01:24,961 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/17.46 kb and 3/20 files
2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.73/17.46 kb and 4/20 files
2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/17.46 kb and 4/20 files
2016-08-25 09:01:24,962 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.63/17.46 kb and 5/20 files
2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/17.46 kb and 5/20 files
2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.53/17.46 kb and 6/20 files
2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.43/17.46 kb and 7/20 files
2016-08-25 09:01:24,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/17.46 kb and 6/20 files
2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/17.46 kb and 7/20 files
2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.38/17.46 kb and 8/20 files
2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.28/17.46 kb and 9/20 files
2016-08-25 09:01:24,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/17.46 kb and 8/20 files
2016-08-25 09:01:24,965 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.18/17.46 kb and 10/20 files
2016-08-25 09:01:24,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.14/17.46 kb and 9/20 files
2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.08/17.46 kb and 11/20 files
2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.04/17.46 kb and 10/20 files
2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/17.46 kb and 11/20 files
2016-08-25 09:01:24,966 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.98/17.46 kb and 12/20 files
2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/17.46 kb and 12/20 files
2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.87/17.46 kb and 13/20 files
2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/17.46 kb and 13/20 files
2016-08-25 09:01:24,967 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.77/17.46 kb and 14/20 files
2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/17.46 kb and 14/20 files
2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.67/17.46 kb and 15/20 files
2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.57/17.46 kb and 16/20 files
2016-08-25 09:01:24,968 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.67/17.46 kb and 15/20 files
2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.76/17.46 kb and 17/20 files
2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.66/17.46 kb and 18/20 files
2016-08-25 09:01:24,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.57/17.46 kb and 16/20 files
2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.56/17.46 kb and 19/20 files
2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.47/17.46 kb and 17/20 files
2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files
2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.37/17.46 kb and 18/20 files
2016-08-25 09:01:24,970 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.27/17.46 kb and 19/20 files
2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.46/17.46 kb and 20/20 files
2016-08-25 09:01:24,971 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:24,972 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:24,973 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is starting
2016-08-25 09:01:24,976 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:24,977 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55206
2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) started
2016-08-25 09:01:24,978 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:24,978 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:24,978 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
34.4. Browse the queue on new master
======================================
======================================
34.5. Found
======================================
======================================
35.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:25,002 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is shutting down
2016-08-25 09:01:25,003 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:25,006 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,006 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:25,007 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250022
2016-08-25 09:01:25,008 | INFO | Time-limited test | Session: 0x156bec917250022 closed
2016-08-25 09:01:25,008 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55177 which had sessionid 0x156bec917250022
2016-08-25 09:01:25,009 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:25,012 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:25,012 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:25,013 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:25,013 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:25,112 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:25,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) uptime 2.339 seconds
2016-08-25 09:01:25,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:36) is shutdown
Wait for master to start up...
2016-08-25 09:01:25,115 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:25,118 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7b4f3d95
2016-08-25 09:01:25,119 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:25,119 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:25,120 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:25,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55208
2016-08-25 09:01:25,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55208
2016-08-25 09:01:25,121 | INFO | SyncThread:0 | Established session 0x156bec917250026 with negotiated timeout 10000 for client /127.0.0.1:55208
2016-08-25 09:01:25,121 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250026, negotiated timeout = 10000
2016-08-25 09:01:25,133 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,374 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:25,377 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,379 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:25,379 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,383 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:25,386 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:25,508 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55209
2016-08-25 09:01:25,514 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:25,515 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:25,516 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55209
2016-08-25 09:01:25,516 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:25,517 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55209
2016-08-25 09:01:25,518 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:25,643 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:25,645 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:25,650 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000057.log
2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:25,651 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000057.log
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000055
2016-08-25 09:01:25,652 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:25,653 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000055
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:25,654 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:25,666 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files
2016-08-25 09:01:25,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/18.40 kb and 1/21 files
2016-08-25 09:01:25,667 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/18.40 kb and 2/21 files
2016-08-25 09:01:25,667 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files
2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/18.40 kb and 3/21 files
2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files
2016-08-25 09:01:25,668 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.60/18.40 kb and 4/21 files
2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/18.40 kb and 4/21 files
2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.50/18.40 kb and 5/21 files
2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.83/18.40 kb and 5/21 files
2016-08-25 09:01:25,669 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.40/18.40 kb and 6/21 files
2016-08-25 09:01:25,670 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.73/18.40 kb and 6/21 files
2016-08-25 09:01:25,670 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.58/18.40 kb and 7/21 files
2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.55/18.40 kb and 8/21 files
2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/18.40 kb and 7/21 files
2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/18.40 kb and 8/21 files
2016-08-25 09:01:25,671 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.45/18.40 kb and 9/21 files
2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.42/18.40 kb and 9/21 files
2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.35/18.40 kb and 10/21 files
2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.32/18.40 kb and 10/21 files
2016-08-25 09:01:25,672 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.25/18.40 kb and 11/21 files
2016-08-25 09:01:25,673 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/18.40 kb and 11/21 files
2016-08-25 09:01:25,673 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.20/18.40 kb and 12/21 files
2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.24/18.40 kb and 12/21 files
2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.10/18.40 kb and 13/21 files
2016-08-25 09:01:25,674 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.14/18.40 kb and 13/21 files
2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.99/18.40 kb and 14/21 files
2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.04/18.40 kb and 14/21 files
2016-08-25 09:01:25,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.89/18.40 kb and 15/21 files
2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.94/18.40 kb and 15/21 files
2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.79/18.40 kb and 16/21 files
2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.80/18.40 kb and 17/21 files
2016-08-25 09:01:25,676 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.84/18.40 kb and 16/21 files
2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.74/18.40 kb and 17/21 files
2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.70/18.40 kb and 18/21 files
2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.64/18.40 kb and 18/21 files
2016-08-25 09:01:25,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files
2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files
2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files
2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files
2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files
2016-08-25 09:01:25,678 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:25,679 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files
2016-08-25 09:01:25,679 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:25,683 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:25,685 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is starting
2016-08-25 09:01:25,686 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:25,690 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55214
2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) started
2016-08-25 09:01:25,691 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:25,691 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:25,691 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
35.4. Browse the queue on new master
======================================
======================================
35.5. Found
======================================
======================================
36.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:25,712 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is shutting down
2016-08-25 09:01:25,713 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:25,716 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,719 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:25,719 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:25,719 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:25,719 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:25,721 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:25,721 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250024
2016-08-25 09:01:25,722 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55192 which had sessionid 0x156bec917250024
2016-08-25 09:01:25,722 | INFO | Time-limited test | Session: 0x156bec917250024 closed
2016-08-25 09:01:25,722 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:25,844 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:25,845 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) uptime 1.909 seconds
2016-08-25 09:01:25,845 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:37) is shutdown
Wait for master to start up...
2016-08-25 09:01:25,849 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:25,853 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1210a97
2016-08-25 09:01:25,853 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:25,854 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:25,855 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:25,855 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55216
2016-08-25 09:01:25,855 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55216
2016-08-25 09:01:25,859 | INFO | SyncThread:0 | Established session 0x156bec917250027 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55216
2016-08-25 09:01:25,859 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250027, negotiated timeout = 10000
2016-08-25 09:01:25,874 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:25,983 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,986 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:25,989 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:25,994 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,121 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55217
2016-08-25 09:01:26,126 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,127 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55217
2016-08-25 09:01:26,128 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:26,129 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55217
2016-08-25 09:01:26,129 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:26,141 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:26,230 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:26,231 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:26,232 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000055
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000057.log
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:26,233 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:26,234 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000057.log
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:26,235 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:26,236 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000055
2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:26,237 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:26,243 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files
2016-08-25 09:01:26,243 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files
2016-08-25 09:01:26,244 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files
2016-08-25 09:01:26,244 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/18.40 kb and 4/21 files
2016-08-25 09:01:26,245 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/18.40 kb and 5/21 files
2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.39/18.40 kb and 6/21 files
2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/18.40 kb and 1/21 files
2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/18.40 kb and 2/21 files
2016-08-25 09:01:26,246 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/18.40 kb and 7/21 files
2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/18.40 kb and 3/21 files
2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.31/18.40 kb and 8/21 files
2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.21/18.40 kb and 9/21 files
2016-08-25 09:01:26,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/18.40 kb and 4/21 files
2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.17/18.40 kb and 10/21 files
2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.90/18.40 kb and 5/21 files
2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.07/18.40 kb and 11/21 files
2016-08-25 09:01:26,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.80/18.40 kb and 6/21 files
2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.26/18.40 kb and 12/21 files
2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/18.40 kb and 7/21 files
2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.15/18.40 kb and 13/21 files
2016-08-25 09:01:26,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/18.40 kb and 8/21 files
2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.05/18.40 kb and 14/21 files
2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/18.40 kb and 9/21 files
2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.96/18.40 kb and 15/21 files
2016-08-25 09:01:26,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.40/18.40 kb and 10/21 files
2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.30/18.40 kb and 11/21 files
2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.86/18.40 kb and 16/21 files
2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.75/18.40 kb and 17/21 files
2016-08-25 09:01:26,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.20/18.40 kb and 12/21 files
2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.70/18.40 kb and 18/21 files
2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.10/18.40 kb and 13/21 files
2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.05/18.40 kb and 14/21 files
2016-08-25 09:01:26,252 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.60/18.40 kb and 19/21 files
2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/18.40 kb and 15/21 files
2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.50/18.40 kb and 20/21 files
2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/18.40 kb and 16/21 files
2016-08-25 09:01:26,253 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files
2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/18.40 kb and 17/21 files
2016-08-25 09:01:26,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.71/18.40 kb and 18/21 files
2016-08-25 09:01:26,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.61/18.40 kb and 19/21 files
2016-08-25 09:01:26,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.51/18.40 kb and 20/21 files
2016-08-25 09:01:26,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.40/18.40 kb and 21/21 files
2016-08-25 09:01:26,256 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:26,257 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:26,258 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is starting
2016-08-25 09:01:26,263 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:26,263 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55222
2016-08-25 09:01:26,263 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:26,264 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) started
2016-08-25 09:01:26,264 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:26,264 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:26,264 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
36.4. Browse the queue on new master
======================================
======================================
36.5. Found
======================================
======================================
37.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:26,283 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is shutting down
2016-08-25 09:01:26,284 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:26,288 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:26,291 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:26,292 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250025
2016-08-25 09:01:26,293 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:26,293 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:26,293 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:26,293 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:26,294 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55200 which had sessionid 0x156bec917250025
2016-08-25 09:01:26,294 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:26,294 | INFO | Time-limited test | Session: 0x156bec917250025 closed
2016-08-25 09:01:26,398 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:26,398 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) uptime 1.806 seconds
2016-08-25 09:01:26,398 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:38) is shutdown
Wait for master to start up...
2016-08-25 09:01:26,401 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:26,404 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@60bc56c1
2016-08-25 09:01:26,404 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:26,405 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:26,405 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:26,406 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55224
2016-08-25 09:01:26,406 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55224
2016-08-25 09:01:26,407 | INFO | SyncThread:0 | Established session 0x156bec917250028 with negotiated timeout 10000 for client /127.0.0.1:55224
2016-08-25 09:01:26,407 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250028, negotiated timeout = 10000
2016-08-25 09:01:26,426 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:26,681 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:26,684 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:26,687 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:26,687 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:26,691 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:26,694 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,825 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55225
2016-08-25 09:01:26,830 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,830 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:26,831 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55225
2016-08-25 09:01:26,831 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:26,833 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55225
2016-08-25 09:01:26,833 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:26,951 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:26,961 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:26,961 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:26,962 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:26,962 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000060.log
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:26,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000058
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:26,964 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:26,965 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:26,965 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000058
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:26,966 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000060.log
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:26,967 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:26,974 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files
2016-08-25 09:01:26,974 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files
2016-08-25 09:01:26,975 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files
2016-08-25 09:01:26,976 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/19.35 kb and 4/22 files
2016-08-25 09:01:26,977 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.49/19.35 kb and 5/22 files
2016-08-25 09:01:26,978 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/19.35 kb and 6/22 files
2016-08-25 09:01:26,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/19.35 kb and 7/22 files
2016-08-25 09:01:26,979 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/19.35 kb and 8/22 files
2016-08-25 09:01:26,980 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/19.35 kb and 9/22 files
2016-08-25 09:01:26,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/19.35 kb and 10/22 files
2016-08-25 09:01:26,981 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.22/19.35 kb and 11/22 files
2016-08-25 09:01:26,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.12/19.35 kb and 12/22 files
2016-08-25 09:01:26,982 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.02/19.35 kb and 13/22 files
2016-08-25 09:01:26,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.92/19.35 kb and 14/22 files
2016-08-25 09:01:26,983 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.94/19.35 kb and 15/22 files
2016-08-25 09:01:26,984 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/19.35 kb and 16/22 files
2016-08-25 09:01:26,984 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.74/19.35 kb and 17/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.64/19.35 kb and 18/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/19.35 kb and 2/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/19.35 kb and 19/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files
2016-08-25 09:01:26,985 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files
2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files
2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/19.35 kb and 4/22 files
2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/19.35 kb and 5/22 files
2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files
2016-08-25 09:01:26,986 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:26,987 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/19.35 kb and 6/22 files
2016-08-25 09:01:26,987 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/19.35 kb and 7/22 files
2016-08-25 09:01:26,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.31/19.35 kb and 8/22 files
2016-08-25 09:01:26,988 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.21/19.35 kb and 9/22 files
2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.10/19.35 kb and 10/22 files
2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.00/19.35 kb and 11/22 files
2016-08-25 09:01:26,989 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.90/19.35 kb and 12/22 files
2016-08-25 09:01:26,990 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.80/19.35 kb and 13/22 files
2016-08-25 09:01:26,990 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:26,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.70/19.35 kb and 14/22 files
2016-08-25 09:01:26,991 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is starting
2016-08-25 09:01:26,991 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.60/19.35 kb and 15/22 files
2016-08-25 09:01:26,992 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.55/19.35 kb and 16/22 files
2016-08-25 09:01:26,992 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.45/19.35 kb and 17/22 files
2016-08-25 09:01:26,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.35/19.35 kb and 18/22 files
2016-08-25 09:01:26,993 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.53/19.35 kb and 19/22 files
2016-08-25 09:01:26,994 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files
2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files
2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files
2016-08-25 09:01:26,995 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55230
2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) started
2016-08-25 09:01:26,996 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:26,996 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:26,996 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
37.4. Browse the queue on new master
======================================
2016-08-25 09:01:27,000 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
37.5. Found
======================================
======================================
38.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:27,015 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is shutting down
2016-08-25 09:01:27,016 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:27,019 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:27,020 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:27,020 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:27,021 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:27,021 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:27,021 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:27,021 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250026
2016-08-25 09:01:27,022 | INFO | Time-limited test | Session: 0x156bec917250026 closed
2016-08-25 09:01:27,022 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:27,022 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55208 which had sessionid 0x156bec917250026
2016-08-25 09:01:27,141 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:27,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) uptime 2.028 seconds
2016-08-25 09:01:27,141 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:39) is shutdown
Wait for master to start up...
2016-08-25 09:01:27,145 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:27,147 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@2bf7a4f8
2016-08-25 09:01:27,148 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:27,149 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:27,149 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:27,150 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55232
2016-08-25 09:01:27,150 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55232
2016-08-25 09:01:27,151 | INFO | SyncThread:0 | Established session 0x156bec917250029 with negotiated timeout 10000 for client /127.0.0.1:55232
2016-08-25 09:01:27,151 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250029, negotiated timeout = 10000
2016-08-25 09:01:27,167 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:27,286 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:27,289 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:27,292 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:27,292 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:27,296 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:27,300 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:27,431 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55233
2016-08-25 09:01:27,437 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:27,437 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:27,438 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55233
2016-08-25 09:01:27,438 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:27,441 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55233
2016-08-25 09:01:27,441 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:27,452 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:27,557 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:27,557 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:27,558 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:27,558 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000060.log
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:27,559 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000058
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:27,560 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:27,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:27,562 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000060.log
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000058
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:27,563 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:27,564 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:27,564 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:27,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files
2016-08-25 09:01:27,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files
2016-08-25 09:01:27,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files
2016-08-25 09:01:27,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.64/19.35 kb and 4/22 files
2016-08-25 09:01:27,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/19.35 kb and 5/22 files
2016-08-25 09:01:27,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/19.35 kb and 6/22 files
2016-08-25 09:01:27,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/19.35 kb and 7/22 files
2016-08-25 09:01:27,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.64/19.35 kb and 8/22 files
2016-08-25 09:01:27,576 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.54/19.35 kb and 9/22 files
2016-08-25 09:01:27,577 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.44/19.35 kb and 10/22 files
2016-08-25 09:01:27,577 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.34/19.35 kb and 11/22 files
2016-08-25 09:01:27,578 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.24/19.35 kb and 12/22 files
2016-08-25 09:01:27,579 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.26/19.35 kb and 13/22 files
2016-08-25 09:01:27,579 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.16/19.35 kb and 14/22 files
2016-08-25 09:01:27,580 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.06/19.35 kb and 15/22 files
2016-08-25 09:01:27,581 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.95/19.35 kb and 16/22 files
2016-08-25 09:01:27,581 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.85/19.35 kb and 17/22 files
2016-08-25 09:01:27,582 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.75/19.35 kb and 18/22 files
2016-08-25 09:01:27,582 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.65/19.35 kb and 19/22 files
2016-08-25 09:01:27,583 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files
2016-08-25 09:01:27,583 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files
2016-08-25 09:01:27,584 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files
2016-08-25 09:01:27,584 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:27,587 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:27,588 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is starting
2016-08-25 09:01:27,589 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/19.35 kb and 1/22 files
2016-08-25 09:01:27,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/19.35 kb and 2/22 files
2016-08-25 09:01:27,591 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/19.35 kb and 3/22 files
2016-08-25 09:01:27,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/19.35 kb and 4/22 files
2016-08-25 09:01:27,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/19.35 kb and 5/22 files
2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55238
2016-08-25 09:01:27,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/19.35 kb and 6/22 files
2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) started
2016-08-25 09:01:27,594 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:27,594 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:27,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.34/19.35 kb and 7/22 files
2016-08-25 09:01:27,594 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:27,595 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/19.35 kb and 8/22 files
2016-08-25 09:01:27,596 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/19.35 kb and 9/22 files
2016-08-25 09:01:27,596 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.04/19.35 kb and 10/22 files
Master started: node-2
======================================
38.4. Browse the queue on new master
======================================
2016-08-25 09:01:27,597 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/19.35 kb and 11/22 files
2016-08-25 09:01:27,598 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.95/19.35 kb and 12/22 files
2016-08-25 09:01:27,598 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.85/19.35 kb and 13/22 files
2016-08-25 09:01:27,599 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/19.35 kb and 14/22 files
2016-08-25 09:01:27,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.94/19.35 kb and 15/22 files
2016-08-25 09:01:27,602 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.84/19.35 kb and 16/22 files
2016-08-25 09:01:27,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.74/19.35 kb and 17/22 files
2016-08-25 09:01:27,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.75/19.35 kb and 18/22 files
2016-08-25 09:01:27,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.65/19.35 kb and 19/22 files
2016-08-25 09:01:27,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.55/19.35 kb and 20/22 files
2016-08-25 09:01:27,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/19.35 kb and 21/22 files
2016-08-25 09:01:27,615 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/19.35 kb and 22/22 files
2016-08-25 09:01:27,615 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
38.5. Found
======================================
======================================
39.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:27,618 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is shutting down
2016-08-25 09:01:27,619 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:27,623 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:27,625 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:27,625 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250027
2016-08-25 09:01:27,626 | INFO | Time-limited test | Session: 0x156bec917250027 closed
2016-08-25 09:01:27,626 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55216 which had sessionid 0x156bec917250027
2016-08-25 09:01:27,627 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:27,632 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:27,632 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:27,637 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:27,637 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:27,732 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:27,733 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) uptime 1.887 seconds
2016-08-25 09:01:27,733 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:40) is shutdown
Wait for master to start up...
2016-08-25 09:01:27,735 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:27,738 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@289c0574
2016-08-25 09:01:27,739 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:27,740 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:27,740 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:27,741 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55240
2016-08-25 09:01:27,741 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55240
2016-08-25 09:01:27,742 | INFO | SyncThread:0 | Established session 0x156bec91725002a with negotiated timeout 10000 for client /127.0.0.1:55240
2016-08-25 09:01:27,742 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002a, negotiated timeout = 10000
2016-08-25 09:01:27,756 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,075 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:28,078 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,097 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:28,097 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,101 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:28,107 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:28,279 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55241
2016-08-25 09:01:28,285 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:28,285 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:28,287 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55241
2016-08-25 09:01:28,287 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:28,288 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55241
2016-08-25 09:01:28,289 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:28,441 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:28,455 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:28,456 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000061
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000063.log
2016-08-25 09:01:28,457 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:28,458 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000061
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:28,459 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000063.log
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:28,460 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:28,461 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:28,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files
2016-08-25 09:01:28,475 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files
2016-08-25 09:01:28,476 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/20.30 kb and 3/23 files
2016-08-25 09:01:28,476 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/20.30 kb and 4/23 files
2016-08-25 09:01:28,477 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/20.30 kb and 5/23 files
2016-08-25 09:01:28,477 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/20.30 kb and 6/23 files
2016-08-25 09:01:28,478 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/20.30 kb and 7/23 files
2016-08-25 09:01:28,479 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.31/20.30 kb and 8/23 files
2016-08-25 09:01:28,479 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.21/20.30 kb and 9/23 files
2016-08-25 09:01:28,480 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.11/20.30 kb and 10/23 files
2016-08-25 09:01:28,480 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.01/20.30 kb and 11/23 files
2016-08-25 09:01:28,481 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/20.30 kb and 12/23 files
2016-08-25 09:01:28,482 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/20.30 kb and 13/23 files
2016-08-25 09:01:28,482 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/20.30 kb and 14/23 files
2016-08-25 09:01:28,483 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/20.30 kb and 15/23 files
2016-08-25 09:01:28,484 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/20.30 kb and 16/23 files
2016-08-25 09:01:28,484 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/20.30 kb and 17/23 files
2016-08-25 09:01:28,485 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/20.30 kb and 18/23 files
2016-08-25 09:01:28,485 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/20.30 kb and 19/23 files
2016-08-25 09:01:28,486 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/20.30 kb and 20/23 files
2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.22/20.30 kb and 21/23 files
2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files
2016-08-25 09:01:28,487 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.40/20.30 kb and 22/23 files
2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files
2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files
2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:28,488 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.87/20.30 kb and 3/23 files
2016-08-25 09:01:28,489 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.77/20.30 kb and 4/23 files
2016-08-25 09:01:28,490 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.67/20.30 kb and 5/23 files
2016-08-25 09:01:28,490 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.56/20.30 kb and 6/23 files
2016-08-25 09:01:28,491 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.46/20.30 kb and 7/23 files
2016-08-25 09:01:28,491 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.36/20.30 kb and 8/23 files
2016-08-25 09:01:28,492 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.38/20.30 kb and 9/23 files
2016-08-25 09:01:28,493 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.56/20.30 kb and 10/23 files
2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.46/20.30 kb and 11/23 files
2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:28,494 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.36/20.30 kb and 12/23 files
2016-08-25 09:01:28,495 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is starting
2016-08-25 09:01:28,495 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.30/20.30 kb and 13/23 files
2016-08-25 09:01:28,496 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.20/20.30 kb and 14/23 files
2016-08-25 09:01:28,496 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.10/20.30 kb and 15/23 files
2016-08-25 09:01:28,497 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.00/20.30 kb and 16/23 files
2016-08-25 09:01:28,497 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.90/20.30 kb and 17/23 files
2016-08-25 09:01:28,498 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.80/20.30 kb and 18/23 files
2016-08-25 09:01:28,499 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.71/20.30 kb and 19/23 files
2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55246
2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) started
2016-08-25 09:01:28,499 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:28,499 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.60/20.30 kb and 20/23 files
2016-08-25 09:01:28,500 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:28,500 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:28,500 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.50/20.30 kb and 21/23 files
2016-08-25 09:01:28,501 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.40/20.30 kb and 22/23 files
Master started: node-1
======================================
39.4. Browse the queue on new master
======================================
2016-08-25 09:01:28,501 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files
2016-08-25 09:01:28,502 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:28,508 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
39.5. Found
======================================
======================================
40.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:28,517 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is shutting down
2016-08-25 09:01:28,518 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:28,520 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,521 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:28,521 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250028
2016-08-25 09:01:28,522 | INFO | Time-limited test | Session: 0x156bec917250028 closed
2016-08-25 09:01:28,522 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:28,522 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55224 which had sessionid 0x156bec917250028
2016-08-25 09:01:28,523 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:28,523 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:28,524 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:28,524 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:28,639 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:28,640 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) uptime 2.240 seconds
2016-08-25 09:01:28,640 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:41) is shutdown
Wait for master to start up...
2016-08-25 09:01:28,642 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:28,645 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1c7d7259
2016-08-25 09:01:28,645 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:28,646 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:28,647 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:28,647 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55248
2016-08-25 09:01:28,647 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55248
2016-08-25 09:01:28,648 | INFO | SyncThread:0 | Established session 0x156bec91725002b with negotiated timeout 10000 for client /127.0.0.1:55248
2016-08-25 09:01:28,648 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002b, negotiated timeout = 10000
2016-08-25 09:01:28,655 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:28,839 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,858 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:28,858 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:28,865 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:28,869 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,031 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55249
2016-08-25 09:01:29,036 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,037 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,038 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55249
2016-08-25 09:01:29,038 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:29,040 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55249
2016-08-25 09:01:29,040 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:29,053 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:29,056 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:29,057 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000063.log
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000061
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:29,058 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:29,059 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:29,060 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:29,074 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/20.30 kb and 1/23 files
2016-08-25 09:01:29,074 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/20.30 kb and 2/23 files
2016-08-25 09:01:29,075 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/20.30 kb and 3/23 files
2016-08-25 09:01:29,076 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/20.30 kb and 4/23 files
2016-08-25 09:01:29,077 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.96/20.30 kb and 5/23 files
2016-08-25 09:01:29,077 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.85/20.30 kb and 6/23 files
2016-08-25 09:01:29,078 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.75/20.30 kb and 7/23 files
2016-08-25 09:01:29,079 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.65/20.30 kb and 8/23 files
2016-08-25 09:01:29,079 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.55/20.30 kb and 9/23 files
2016-08-25 09:01:29,080 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.45/20.30 kb and 10/23 files
2016-08-25 09:01:29,081 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.35/20.30 kb and 11/23 files
2016-08-25 09:01:29,081 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.25/20.30 kb and 12/23 files
2016-08-25 09:01:29,082 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.15/20.30 kb and 13/23 files
2016-08-25 09:01:29,083 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.04/20.30 kb and 14/23 files
2016-08-25 09:01:29,083 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.94/20.30 kb and 15/23 files
2016-08-25 09:01:29,084 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.84/20.30 kb and 16/23 files
2016-08-25 09:01:29,085 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.86/20.30 kb and 17/23 files
2016-08-25 09:01:29,086 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.76/20.30 kb and 18/23 files
2016-08-25 09:01:29,086 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.66/20.30 kb and 19/23 files
2016-08-25 09:01:29,087 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.56/20.30 kb and 20/23 files
2016-08-25 09:01:29,088 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.45/20.30 kb and 21/23 files
2016-08-25 09:01:29,088 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.35/20.30 kb and 22/23 files
2016-08-25 09:01:29,089 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.30/20.30 kb and 23/23 files
2016-08-25 09:01:29,089 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:29,092 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:29,093 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is starting
2016-08-25 09:01:29,097 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55253
2016-08-25 09:01:29,097 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:29,098 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) started
2016-08-25 09:01:29,098 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:29,098 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:29,098 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
40.4. Browse the queue on new master
======================================
======================================
40.5. Found
======================================
======================================
41.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:29,113 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is shutting down
2016-08-25 09:01:29,113 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:29,115 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:29,116 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250029
2016-08-25 09:01:29,116 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:29,117 | INFO | Time-limited test | Session: 0x156bec917250029 closed
2016-08-25 09:01:29,117 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:29,117 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55232 which had sessionid 0x156bec917250029
2016-08-25 09:01:29,121 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:29,121 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:29,220 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:29,220 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) uptime 2.077 seconds
2016-08-25 09:01:29,220 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:42) is shutdown
Wait for master to start up...
2016-08-25 09:01:29,222 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:29,225 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1afd5978
2016-08-25 09:01:29,225 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:29,225 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:29,226 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:29,226 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55255
2016-08-25 09:01:29,226 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55255
2016-08-25 09:01:29,227 | INFO | SyncThread:0 | Established session 0x156bec91725002c with negotiated timeout 10000 for client /127.0.0.1:55255
2016-08-25 09:01:29,227 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002c, negotiated timeout = 10000
2016-08-25 09:01:29,233 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:29,363 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:29,363 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:29,364 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:29,367 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:29,580 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:29,584 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:29,587 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,735 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55256
2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55256
2016-08-25 09:01:29,743 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:29,745 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55256
2016-08-25 09:01:29,745 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:29,908 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:29,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:29,915 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:29,916 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000064
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:29,917 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000066.log
2016-08-25 09:01:29,918 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:29,920 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000066.log
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000064
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:29,921 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:29,922 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:29,923 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files
2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files
2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files
2016-08-25 09:01:29,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files
2016-08-25 09:01:29,946 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files
2016-08-25 09:01:29,946 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files
2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/21.25 kb and 4/24 files
2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/21.25 kb and 4/24 files
2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/21.25 kb and 5/24 files
2016-08-25 09:01:29,947 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/21.25 kb and 5/24 files
2016-08-25 09:01:29,948 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.51/21.25 kb and 6/24 files
2016-08-25 09:01:29,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/21.25 kb and 6/24 files
2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/21.25 kb and 7/24 files
2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.41/21.25 kb and 7/24 files
2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.31/21.25 kb and 8/24 files
2016-08-25 09:01:29,949 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/21.25 kb and 8/24 files
2016-08-25 09:01:29,950 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.21/21.25 kb and 9/24 files
2016-08-25 09:01:29,950 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.43/21.25 kb and 9/24 files
2016-08-25 09:01:29,951 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.11/21.25 kb and 10/24 files
2016-08-25 09:01:29,951 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.33/21.25 kb and 10/24 files
2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.45/21.25 kb and 11/24 files
2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.23/21.25 kb and 11/24 files
2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.13/21.25 kb and 12/24 files
2016-08-25 09:01:29,952 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.35/21.25 kb and 12/24 files
2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.03/21.25 kb and 13/24 files
2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.25/21.25 kb and 13/24 files
2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.15/21.25 kb and 14/24 files
2016-08-25 09:01:29,954 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.93/21.25 kb and 14/24 files
2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.04/21.25 kb and 15/24 files
2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.83/21.25 kb and 15/24 files
2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.78/21.25 kb and 16/24 files
2016-08-25 09:01:29,955 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.94/21.25 kb and 16/24 files
2016-08-25 09:01:29,956 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.68/21.25 kb and 17/24 files
2016-08-25 09:01:29,956 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.96/21.25 kb and 17/24 files
2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.86/21.25 kb and 18/24 files
2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.58/21.25 kb and 18/24 files
2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.76/21.25 kb and 19/24 files
2016-08-25 09:01:29,957 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.47/21.25 kb and 19/24 files
2016-08-25 09:01:29,958 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.37/21.25 kb and 20/24 files
2016-08-25 09:01:29,958 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.66/21.25 kb and 20/24 files
2016-08-25 09:01:29,959 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.56/21.25 kb and 21/24 files
2016-08-25 09:01:29,959 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.27/21.25 kb and 21/24 files
2016-08-25 09:01:29,960 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.17/21.25 kb and 22/24 files
2016-08-25 09:01:29,960 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.46/21.25 kb and 22/24 files
2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.35/21.25 kb and 23/24 files
2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.35/21.25 kb and 23/24 files
2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files
2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:29,961 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files
2016-08-25 09:01:29,962 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:29,967 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:29,968 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is starting
2016-08-25 09:01:29,970 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:29,972 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55261
2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) started
2016-08-25 09:01:29,973 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:29,973 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:29,973 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
41.4. Browse the queue on new master
======================================
======================================
41.5. Found
======================================
======================================
42.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:29,994 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is shutting down
2016-08-25 09:01:29,995 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:29,997 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:29,998 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:29,998 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:29,998 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002b
2016-08-25 09:01:29,999 | INFO | Time-limited test | Session: 0x156bec91725002b closed
2016-08-25 09:01:29,999 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:29,999 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55248 which had sessionid 0x156bec91725002b
2016-08-25 09:01:30,001 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:30,005 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:30,005 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:30,005 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:30,005 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:30,120 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:30,121 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) uptime 1.481 seconds
2016-08-25 09:01:30,121 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:43) is shutdown
Wait for master to start up...
2016-08-25 09:01:30,123 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:30,126 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5a8234a5
2016-08-25 09:01:30,126 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:30,127 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:30,128 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:30,128 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55263
2016-08-25 09:01:30,129 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55263
2016-08-25 09:01:30,129 | INFO | SyncThread:0 | Established session 0x156bec91725002d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55263
2016-08-25 09:01:30,130 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725002d, negotiated timeout = 10000
2016-08-25 09:01:30,138 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:30,328 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:30,331 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:30,333 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:30,333 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:30,337 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:30,340 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:30,525 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55264
2016-08-25 09:01:30,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:30,531 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:30,532 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55264
2016-08-25 09:01:30,532 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:30,534 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55264
2016-08-25 09:01:30,535 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:30,546 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:30,679 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:30,679 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:30,680 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000066.log
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000064
2016-08-25 09:01:30,681 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:30,682 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:30,696 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files
2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/21.25 kb and 2/24 files
2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/21.25 kb and 3/24 files
2016-08-25 09:01:30,697 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.94/21.25 kb and 4/24 files
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:30,698 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.84/21.25 kb and 5/24 files
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.74/21.25 kb and 6/24 files
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:30,699 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000064
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/21.25 kb and 7/24 files
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/21.25 kb and 8/24 files
2016-08-25 09:01:30,700 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000066.log
2016-08-25 09:01:30,701 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/21.25 kb and 9/24 files
2016-08-25 09:01:30,702 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/21.25 kb and 10/24 files
2016-08-25 09:01:30,702 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.23/21.25 kb and 11/24 files
2016-08-25 09:01:30,703 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.13/21.25 kb and 12/24 files
2016-08-25 09:01:30,704 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.25/21.25 kb and 13/24 files
2016-08-25 09:01:30,704 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.15/21.25 kb and 14/24 files
2016-08-25 09:01:30,705 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.05/21.25 kb and 15/24 files
2016-08-25 09:01:30,706 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.95/21.25 kb and 16/24 files
2016-08-25 09:01:30,707 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.85/21.25 kb and 17/24 files
2016-08-25 09:01:30,707 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.75/21.25 kb and 18/24 files
2016-08-25 09:01:30,708 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.65/21.25 kb and 19/24 files
2016-08-25 09:01:30,709 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.55/21.25 kb and 20/24 files
2016-08-25 09:01:30,709 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.45/21.25 kb and 21/24 files
2016-08-25 09:01:30,710 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.46/21.25 kb and 22/24 files
2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.36/21.25 kb and 23/24 files
2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files
2016-08-25 09:01:30,711 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:30,713 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:30,714 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/21.25 kb and 1/24 files
2016-08-25 09:01:30,715 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is starting
2016-08-25 09:01:30,715 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/21.25 kb and 2/24 files
2016-08-25 09:01:30,716 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/21.25 kb and 3/24 files
2016-08-25 09:01:30,716 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/21.25 kb and 4/24 files
2016-08-25 09:01:30,717 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/21.25 kb and 5/24 files
2016-08-25 09:01:30,717 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/21.25 kb and 6/24 files
2016-08-25 09:01:30,718 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/21.25 kb and 7/24 files
2016-08-25 09:01:30,719 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/21.25 kb and 8/24 files
2016-08-25 09:01:30,719 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/21.25 kb and 9/24 files
2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55269
2016-08-25 09:01:30,720 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/21.25 kb and 10/24 files
2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) started
2016-08-25 09:01:30,720 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:30,720 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:30,720 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:30,720 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/21.25 kb and 11/24 files
2016-08-25 09:01:30,721 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/21.25 kb and 12/24 files
2016-08-25 09:01:30,722 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/21.25 kb and 13/24 files
Master started: node-2
======================================
42.4. Browse the queue on new master
======================================
2016-08-25 09:01:30,723 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/21.25 kb and 14/24 files
2016-08-25 09:01:30,724 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.76/21.25 kb and 15/24 files
2016-08-25 09:01:30,724 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.78/21.25 kb and 16/24 files
2016-08-25 09:01:30,725 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.68/21.25 kb and 17/24 files
2016-08-25 09:01:30,726 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.58/21.25 kb and 18/24 files
2016-08-25 09:01:30,726 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.48/21.25 kb and 19/24 files
2016-08-25 09:01:30,727 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.38/21.25 kb and 20/24 files
2016-08-25 09:01:30,728 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.27/21.25 kb and 21/24 files
2016-08-25 09:01:30,728 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.17/21.25 kb and 22/24 files
2016-08-25 09:01:30,729 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/21.25 kb and 23/24 files
2016-08-25 09:01:30,730 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.25/21.25 kb and 24/24 files
2016-08-25 09:01:30,730 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:30,736 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
42.5. Found
======================================
======================================
43.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:30,739 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is shutting down
2016-08-25 09:01:30,740 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:30,743 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:30,744 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:30,745 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002a
2016-08-25 09:01:30,745 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55240 which had sessionid 0x156bec91725002a
2016-08-25 09:01:30,745 | INFO | Time-limited test | Session: 0x156bec91725002a closed
2016-08-25 09:01:30,746 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:30,748 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:30,748 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:30,748 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:30,748 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:30,850 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:30,850 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) uptime 3.116 seconds
2016-08-25 09:01:30,851 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:44) is shutdown
Wait for master to start up...
2016-08-25 09:01:30,853 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:30,856 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4b0f573b
2016-08-25 09:01:30,856 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:30,857 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:30,857 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55271
2016-08-25 09:01:30,857 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:30,858 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55271
2016-08-25 09:01:30,859 | INFO | SyncThread:0 | Established session 0x156bec91725002e with negotiated timeout 10000 for client /127.0.0.1:55271
2016-08-25 09:01:30,859 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002e, negotiated timeout = 10000
2016-08-25 09:01:30,868 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:31,230 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:31,232 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:31,253 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:31,253 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:31,257 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:31,262 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:31,450 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55272
2016-08-25 09:01:31,454 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:31,454 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:31,456 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55272
2016-08-25 09:01:31,456 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:31,458 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55272
2016-08-25 09:01:31,458 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:31,617 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:31,624 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:31,624 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:31,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000069.log
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:31,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:31,627 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000067
2016-08-25 09:01:31,628 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:31,628 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000067
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:31,629 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000069.log
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:31,630 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:31,635 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:31,636 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:31,649 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/22.20 kb and 1/25 files
2016-08-25 09:01:31,650 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/22.20 kb and 2/25 files
2016-08-25 09:01:31,650 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.81/22.20 kb and 3/25 files
2016-08-25 09:01:31,651 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.71/22.20 kb and 4/25 files
2016-08-25 09:01:31,651 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.61/22.20 kb and 5/25 files
2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.51/22.20 kb and 6/25 files
2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.95/22.20 kb and 1/25 files
2016-08-25 09:01:31,652 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.41/22.20 kb and 7/25 files
2016-08-25 09:01:31,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.85/22.20 kb and 2/25 files
2016-08-25 09:01:31,653 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.31/22.20 kb and 8/25 files
2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.75/22.20 kb and 3/25 files
2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.21/22.20 kb and 9/25 files
2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.92/22.20 kb and 4/25 files
2016-08-25 09:01:31,654 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.39/22.20 kb and 10/25 files
2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.82/22.20 kb and 5/25 files
2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.29/22.20 kb and 11/25 files
2016-08-25 09:01:31,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.72/22.20 kb and 6/25 files
2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.19/22.20 kb and 12/25 files
2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/22.20 kb and 7/25 files
2016-08-25 09:01:31,656 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.09/22.20 kb and 13/25 files
2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/22.20 kb and 8/25 files
2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.99/22.20 kb and 14/25 files
2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.42/22.20 kb and 9/25 files
2016-08-25 09:01:31,657 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.94/22.20 kb and 15/25 files
2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.84/22.20 kb and 16/25 files
2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.31/22.20 kb and 10/25 files
2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.73/22.20 kb and 17/25 files
2016-08-25 09:01:31,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.50/22.20 kb and 11/25 files
2016-08-25 09:01:31,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.40/22.20 kb and 12/25 files
2016-08-25 09:01:31,659 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.64/22.20 kb and 18/25 files
2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.30/22.20 kb and 13/25 files
2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.54/22.20 kb and 19/25 files
2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.44/22.20 kb and 20/25 files
2016-08-25 09:01:31,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.20/22.20 kb and 14/25 files
2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.10/22.20 kb and 15/25 files
2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.33/22.20 kb and 21/25 files
2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.99/22.20 kb and 16/25 files
2016-08-25 09:01:31,661 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.23/22.20 kb and 22/25 files
2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.89/22.20 kb and 17/25 files
2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.13/22.20 kb and 23/25 files
2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.79/22.20 kb and 18/25 files
2016-08-25 09:01:31,662 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.03/22.20 kb and 24/25 files
2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files
2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.69/22.20 kb and 19/25 files
2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:31,663 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.70/22.20 kb and 20/25 files
2016-08-25 09:01:31,664 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.60/22.20 kb and 21/25 files
2016-08-25 09:01:31,665 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.50/22.20 kb and 22/25 files
2016-08-25 09:01:31,665 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files
2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.30/22.20 kb and 24/25 files
2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files
2016-08-25 09:01:31,666 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:31,670 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:31,670 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:31,672 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is starting
2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55277
2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) started
2016-08-25 09:01:31,676 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:31,676 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:31,676 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-3
======================================
43.4. Browse the queue on new master
======================================
======================================
43.5. Found
======================================
======================================
44.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:31,693 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is shutting down
2016-08-25 09:01:31,694 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:31,696 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:31,696 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002c
2016-08-25 09:01:31,696 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:31,697 | INFO | Time-limited test | Session: 0x156bec91725002c closed
2016-08-25 09:01:31,697 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:31,697 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55255 which had sessionid 0x156bec91725002c
2016-08-25 09:01:31,701 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:31,701 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:31,701 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:31,701 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:31,811 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:31,812 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) uptime 2.591 seconds
2016-08-25 09:01:31,812 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:45) is shutdown
Wait for master to start up...
2016-08-25 09:01:31,815 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:31,817 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4b8e51d2
2016-08-25 09:01:31,817 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:31,818 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:31,818 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:31,818 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55279
2016-08-25 09:01:31,819 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55279
2016-08-25 09:01:31,819 | INFO | SyncThread:0 | Established session 0x156bec91725002f with negotiated timeout 10000 for client /127.0.0.1:55279
2016-08-25 09:01:31,819 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725002f, negotiated timeout = 10000
2016-08-25 09:01:31,827 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,015 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:32,017 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,020 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:32,020 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,025 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:32,029 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:32,217 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55280
2016-08-25 09:01:32,222 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:32,222 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:32,224 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55280
2016-08-25 09:01:32,224 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:32,226 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55280
2016-08-25 09:01:32,226 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:32,239 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:32,392 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:32,393 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000069.log
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000067
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:32,394 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:32,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:32,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:32,397 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:32,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:32,399 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000069.log
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000067
2016-08-25 09:01:32,400 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:32,401 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:32,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/22.20 kb and 1/25 files
2016-08-25 09:01:32,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.92/22.20 kb and 2/25 files
2016-08-25 09:01:32,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.10/22.20 kb and 3/25 files
2016-08-25 09:01:32,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/22.20 kb and 4/25 files
2016-08-25 09:01:32,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.90/22.20 kb and 5/25 files
2016-08-25 09:01:32,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.79/22.20 kb and 6/25 files
2016-08-25 09:01:32,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.69/22.20 kb and 7/25 files
2016-08-25 09:01:32,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.59/22.20 kb and 8/25 files
2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.76/22.20 kb and 9/25 files
2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/22.20 kb and 1/25 files
2016-08-25 09:01:32,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.66/22.20 kb and 10/25 files
2016-08-25 09:01:32,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/22.20 kb and 2/25 files
2016-08-25 09:01:32,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.56/22.20 kb and 11/25 files
2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/22.20 kb and 3/25 files
2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.46/22.20 kb and 12/25 files
2016-08-25 09:01:32,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/22.20 kb and 4/25 files
2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.41/22.20 kb and 13/25 files
2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/22.20 kb and 5/25 files
2016-08-25 09:01:32,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.31/22.20 kb and 14/25 files
2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/22.20 kb and 6/25 files
2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.21/22.20 kb and 15/25 files
2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/22.20 kb and 7/25 files
2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.11/22.20 kb and 16/25 files
2016-08-25 09:01:32,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/22.20 kb and 8/25 files
2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.01/22.20 kb and 17/25 files
2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/22.20 kb and 9/25 files
2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.91/22.20 kb and 18/25 files
2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/22.20 kb and 10/25 files
2016-08-25 09:01:32,421 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.81/22.20 kb and 19/25 files
2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.71/22.20 kb and 20/25 files
2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/22.20 kb and 11/25 files
2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.61/22.20 kb and 21/25 files
2016-08-25 09:01:32,422 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.12/22.20 kb and 12/25 files
2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.51/22.20 kb and 22/25 files
2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.02/22.20 kb and 13/25 files
2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.92/22.20 kb and 14/25 files
2016-08-25 09:01:32,423 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files
2016-08-25 09:01:32,424 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.30/22.20 kb and 24/25 files
2016-08-25 09:01:32,424 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.82/22.20 kb and 15/25 files
2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.72/22.20 kb and 16/25 files
2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files
2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:32,425 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.62/22.20 kb and 17/25 files
2016-08-25 09:01:32,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/22.20 kb and 18/25 files
2016-08-25 09:01:32,426 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.42/22.20 kb and 19/25 files
2016-08-25 09:01:32,427 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.32/22.20 kb and 20/25 files
2016-08-25 09:01:32,427 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:32,428 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.49/22.20 kb and 21/25 files
2016-08-25 09:01:32,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is starting
2016-08-25 09:01:32,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.51/22.20 kb and 22/25 files
2016-08-25 09:01:32,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.41/22.20 kb and 23/25 files
2016-08-25 09:01:32,430 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.31/22.20 kb and 24/25 files
2016-08-25 09:01:32,431 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.20/22.20 kb and 25/25 files
2016-08-25 09:01:32,431 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55285
2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) started
2016-08-25 09:01:32,433 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:32,433 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:32,433 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-1
======================================
44.4. Browse the queue on new master
======================================
2016-08-25 09:01:32,437 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
44.5. Found
======================================
======================================
45.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:32,448 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is shutting down
2016-08-25 09:01:32,448 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:32,451 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,452 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:32,452 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:32,452 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:32,452 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002d
2016-08-25 09:01:32,452 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:32,453 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:32,453 | INFO | Time-limited test | Session: 0x156bec91725002d closed
2016-08-25 09:01:32,453 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55263 which had sessionid 0x156bec91725002d
2016-08-25 09:01:32,453 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:32,558 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:32,559 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) uptime 2.437 seconds
2016-08-25 09:01:32,559 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:46) is shutdown
Wait for master to start up...
2016-08-25 09:01:32,562 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:32,564 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1886ff2
2016-08-25 09:01:32,564 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:32,565 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:32,565 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:32,565 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55287
2016-08-25 09:01:32,565 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55287
2016-08-25 09:01:32,566 | INFO | SyncThread:0 | Established session 0x156bec917250030 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55287
2016-08-25 09:01:32,566 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250030, negotiated timeout = 10000
2016-08-25 09:01:32,572 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,981 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:32,983 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,985 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:32,986 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:32,990 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:32,997 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:33,177 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55288
2016-08-25 09:01:33,181 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:33,181 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:33,183 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55288
2016-08-25 09:01:33,183 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:33,185 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55288
2016-08-25 09:01:33,185 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:33,365 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:33,381 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:33,381 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:33,382 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:33,383 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000070
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:33,384 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000072.log
2016-08-25 09:01:33,386 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:33,387 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000072.log
2016-08-25 09:01:33,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000070
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:33,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:33,399 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files
2016-08-25 09:01:33,400 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.85/23.15 kb and 2/26 files
2016-08-25 09:01:33,400 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.86/23.15 kb and 3/26 files
2016-08-25 09:01:33,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.76/23.15 kb and 4/26 files
2016-08-25 09:01:33,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.66/23.15 kb and 5/26 files
2016-08-25 09:01:33,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.56/23.15 kb and 6/26 files
2016-08-25 09:01:33,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.46/23.15 kb and 7/26 files
2016-08-25 09:01:33,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.36/23.15 kb and 8/26 files
2016-08-25 09:01:33,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.26/23.15 kb and 9/26 files
2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files
2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.16/23.15 kb and 10/26 files
2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/23.15 kb and 2/26 files
2016-08-25 09:01:33,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/23.15 kb and 11/26 files
2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/23.15 kb and 3/26 files
2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/23.15 kb and 12/26 files
2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/23.15 kb and 4/26 files
2016-08-25 09:01:33,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.85/23.15 kb and 13/26 files
2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/23.15 kb and 5/26 files
2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/23.15 kb and 14/26 files
2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.65/23.15 kb and 15/26 files
2016-08-25 09:01:33,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/23.15 kb and 6/26 files
2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.55/23.15 kb and 16/26 files
2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/23.15 kb and 7/26 files
2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/23.15 kb and 8/26 files
2016-08-25 09:01:33,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.45/23.15 kb and 17/26 files
2016-08-25 09:01:33,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/23.15 kb and 18/26 files
2016-08-25 09:01:33,409 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/23.15 kb and 9/26 files
2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/23.15 kb and 10/26 files
2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/23.15 kb and 19/26 files
2016-08-25 09:01:33,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/23.15 kb and 11/26 files
2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/23.15 kb and 12/26 files
2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/23.15 kb and 20/26 files
2016-08-25 09:01:33,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/23.15 kb and 13/26 files
2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/23.15 kb and 14/26 files
2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/23.15 kb and 15/26 files
2016-08-25 09:01:33,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/23.15 kb and 21/26 files
2016-08-25 09:01:33,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.27/23.15 kb and 22/26 files
2016-08-25 09:01:33,413 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/23.15 kb and 16/26 files
2016-08-25 09:01:33,414 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.63/23.15 kb and 17/26 files
2016-08-25 09:01:33,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.17/23.15 kb and 23/26 files
2016-08-25 09:01:33,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.07/23.15 kb and 24/26 files
2016-08-25 09:01:33,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.97/23.15 kb and 25/26 files
2016-08-25 09:01:33,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files
2016-08-25 09:01:33,416 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/23.15 kb and 18/26 files
2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.54/23.15 kb and 19/26 files
2016-08-25 09:01:33,417 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.44/23.15 kb and 20/26 files
2016-08-25 09:01:33,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.33/23.15 kb and 21/26 files
2016-08-25 09:01:33,418 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.23/23.15 kb and 22/26 files
2016-08-25 09:01:33,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.46/23.15 kb and 23/26 files
2016-08-25 09:01:33,419 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.36/23.15 kb and 24/26 files
2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files
2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files
2016-08-25 09:01:33,420 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:33,421 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:33,422 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is starting
2016-08-25 09:01:33,426 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:33,426 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55293
2016-08-25 09:01:33,426 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:33,427 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) started
2016-08-25 09:01:33,427 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:33,427 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
2016-08-25 09:01:33,427 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30788 mb of usable space. - resetting to maximum available disk space: 30788 mb
Master started: node-2
======================================
45.4. Browse the queue on new master
======================================
======================================
45.5. Found
======================================
======================================
46.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:33,445 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is shutting down
2016-08-25 09:01:33,446 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:33,448 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:33,448 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002e
2016-08-25 09:01:33,448 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:33,449 | INFO | Time-limited test | Session: 0x156bec91725002e closed
2016-08-25 09:01:33,449 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:33,449 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55271 which had sessionid 0x156bec91725002e
2016-08-25 09:01:33,451 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:33,451 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:33,451 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:33,451 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:33,565 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:33,566 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) uptime 2.714 seconds
2016-08-25 09:01:33,566 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:47) is shutdown
Wait for master to start up...
2016-08-25 09:01:33,568 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:33,570 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7f8574aa
2016-08-25 09:01:33,570 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:33,570 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:33,571 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:33,571 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55295
2016-08-25 09:01:33,571 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55295
2016-08-25 09:01:33,572 | INFO | SyncThread:0 | Established session 0x156bec917250031 with negotiated timeout 10000 for client /127.0.0.1:55295
2016-08-25 09:01:33,572 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250031, negotiated timeout = 10000
2016-08-25 09:01:33,580 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:33,824 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:33,826 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:33,829 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:33,829 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:33,834 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:33,837 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:34,046 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55296
2016-08-25 09:01:34,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:34,051 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:34,052 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55296
2016-08-25 09:01:34,052 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:34,054 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55296
2016-08-25 09:01:34,054 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:34,067 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:34,237 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:34,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000072.log
2016-08-25 09:01:34,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000070
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:34,240 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:34,241 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:34,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000072.log
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:34,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000070
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:34,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:34,255 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files
2016-08-25 09:01:34,256 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/23.15 kb and 2/26 files
2016-08-25 09:01:34,257 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/23.15 kb and 3/26 files
2016-08-25 09:01:34,258 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.64/23.15 kb and 4/26 files
2016-08-25 09:01:34,259 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/23.15 kb and 5/26 files
2016-08-25 09:01:34,259 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.44/23.15 kb and 6/26 files
2016-08-25 09:01:34,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/23.15 kb and 7/26 files
2016-08-25 09:01:34,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/23.15 kb and 8/26 files
2016-08-25 09:01:34,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.15/23.15 kb and 9/26 files
2016-08-25 09:01:34,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.16/23.15 kb and 10/26 files
2016-08-25 09:01:34,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.06/23.15 kb and 11/26 files
2016-08-25 09:01:34,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.96/23.15 kb and 12/26 files
2016-08-25 09:01:34,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/23.15 kb and 13/26 files
2016-08-25 09:01:34,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/23.15 kb and 14/26 files
2016-08-25 09:01:34,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/23.15 kb and 15/26 files
2016-08-25 09:01:34,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.84/23.15 kb and 16/26 files
2016-08-25 09:01:34,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.74/23.15 kb and 17/26 files
2016-08-25 09:01:34,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.64/23.15 kb and 18/26 files
2016-08-25 09:01:34,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/23.15 kb and 19/26 files
2016-08-25 09:01:34,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.76/23.15 kb and 20/26 files
2016-08-25 09:01:34,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.66/23.15 kb and 21/26 files
2016-08-25 09:01:34,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.56/23.15 kb and 22/26 files
2016-08-25 09:01:34,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.46/23.15 kb and 23/26 files
2016-08-25 09:01:34,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.36/23.15 kb and 24/26 files
2016-08-25 09:01:34,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files
2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/23.15 kb and 1/26 files
2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files
2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:34,272 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/23.15 kb and 2/26 files
2016-08-25 09:01:34,273 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/23.15 kb and 3/26 files
2016-08-25 09:01:34,274 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/23.15 kb and 4/26 files
2016-08-25 09:01:34,275 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/23.15 kb and 5/26 files
2016-08-25 09:01:34,275 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:34,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/23.15 kb and 6/26 files
2016-08-25 09:01:34,276 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is starting
2016-08-25 09:01:34,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/23.15 kb and 7/26 files
2016-08-25 09:01:34,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/23.15 kb and 8/26 files
2016-08-25 09:01:34,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/23.15 kb and 9/26 files
2016-08-25 09:01:34,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.99/23.15 kb and 10/26 files
2016-08-25 09:01:34,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/23.15 kb and 11/26 files
2016-08-25 09:01:34,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.84/23.15 kb and 12/26 files
2016-08-25 09:01:34,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.02/23.15 kb and 13/26 files
2016-08-25 09:01:34,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.92/23.15 kb and 14/26 files
2016-08-25 09:01:34,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.82/23.15 kb and 15/26 files
2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55301
2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) started
2016-08-25 09:01:34,281 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:34,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.72/23.15 kb and 16/26 files
2016-08-25 09:01:34,281 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:34,281 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:34,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.62/23.15 kb and 17/26 files
2016-08-25 09:01:34,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.52/23.15 kb and 18/26 files
Master started: node-3
======================================
46.4. Browse the queue on new master
======================================
2016-08-25 09:01:34,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.42/23.15 kb and 19/26 files
2016-08-25 09:01:34,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.32/23.15 kb and 20/26 files
2016-08-25 09:01:34,285 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.54/23.15 kb and 21/26 files
2016-08-25 09:01:34,286 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.44/23.15 kb and 22/26 files
2016-08-25 09:01:34,286 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.34/23.15 kb and 23/26 files
2016-08-25 09:01:34,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.24/23.15 kb and 24/26 files
2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.25/23.15 kb and 25/26 files
2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.15/23.15 kb and 26/26 files
2016-08-25 09:01:34,288 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
46.5. Found
======================================
======================================
47.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:34,295 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is shutting down
2016-08-25 09:01:34,296 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:34,296 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:34,298 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:34,300 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:34,300 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:34,300 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725002f
2016-08-25 09:01:34,300 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:34,300 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:34,300 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:34,301 | INFO | Time-limited test | Session: 0x156bec91725002f closed
2016-08-25 09:01:34,301 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:34,301 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55279 which had sessionid 0x156bec91725002f
2016-08-25 09:01:34,404 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:34,405 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) uptime 2.592 seconds
2016-08-25 09:01:34,405 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:48) is shutdown
Wait for master to start up...
2016-08-25 09:01:34,407 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:34,409 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@100a494b
2016-08-25 09:01:34,409 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:34,410 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:34,410 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:34,410 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55303
2016-08-25 09:01:34,410 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55303
2016-08-25 09:01:34,411 | INFO | SyncThread:0 | Established session 0x156bec917250032 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55303
2016-08-25 09:01:34,411 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250032, negotiated timeout = 10000
2016-08-25 09:01:34,418 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:34,852 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:34,854 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:34,875 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:34,876 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:34,880 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:34,883 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,079 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55304
2016-08-25 09:01:35,084 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,085 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,087 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55304
2016-08-25 09:01:35,087 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:35,088 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55304
2016-08-25 09:01:35,088 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:35,269 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:35,277 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:35,278 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000073
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:35,279 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000075.log
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:35,280 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:35,281 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000073
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:35,282 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000075.log
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:35,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:35,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:35,297 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files
2016-08-25 09:01:35,298 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files
2016-08-25 09:01:35,298 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files
2016-08-25 09:01:35,299 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/24.10 kb and 4/27 files
2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.61/24.10 kb and 5/27 files
2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.51/24.10 kb and 6/27 files
2016-08-25 09:01:35,300 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/24.10 kb and 7/27 files
2016-08-25 09:01:35,301 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.68/24.10 kb and 8/27 files
2016-08-25 09:01:35,301 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.58/24.10 kb and 9/27 files
2016-08-25 09:01:35,302 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.48/24.10 kb and 10/27 files
2016-08-25 09:01:35,303 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.43/24.10 kb and 11/27 files
2016-08-25 09:01:35,303 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.33/24.10 kb and 12/27 files
2016-08-25 09:01:35,304 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.52/24.10 kb and 13/27 files
2016-08-25 09:01:35,305 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.42/24.10 kb and 14/27 files
2016-08-25 09:01:35,305 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.32/24.10 kb and 15/27 files
2016-08-25 09:01:35,306 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.21/24.10 kb and 16/27 files
2016-08-25 09:01:35,306 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.12/24.10 kb and 17/27 files
2016-08-25 09:01:35,307 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.02/24.10 kb and 18/27 files
2016-08-25 09:01:35,308 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.92/24.10 kb and 19/27 files
2016-08-25 09:01:35,308 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.82/24.10 kb and 20/27 files
2016-08-25 09:01:35,309 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.71/24.10 kb and 21/27 files
2016-08-25 09:01:35,309 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.61/24.10 kb and 22/27 files
2016-08-25 09:01:35,310 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.51/24.10 kb and 23/27 files
2016-08-25 09:01:35,310 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.41/24.10 kb and 24/27 files
2016-08-25 09:01:35,311 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files
2016-08-25 09:01:35,311 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files
2016-08-25 09:01:35,312 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files
2016-08-25 09:01:35,312 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:35,317 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.27/24.10 kb and 1/27 files
2016-08-25 09:01:35,318 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:35,318 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.17/24.10 kb and 2/27 files
2016-08-25 09:01:35,320 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.07/24.10 kb and 3/27 files
2016-08-25 09:01:35,321 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is starting
2016-08-25 09:01:35,322 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.97/24.10 kb and 4/27 files
2016-08-25 09:01:35,323 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.87/24.10 kb and 5/27 files
2016-08-25 09:01:35,323 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.77/24.10 kb and 6/27 files
2016-08-25 09:01:35,324 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.67/24.10 kb and 7/27 files
2016-08-25 09:01:35,324 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.62/24.10 kb and 8/27 files
2016-08-25 09:01:35,325 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.52/24.10 kb and 9/27 files
2016-08-25 09:01:35,326 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.42/24.10 kb and 10/27 files
2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55309
2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:35,326 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.60/24.10 kb and 11/27 files
2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) started
2016-08-25 09:01:35,326 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:35,326 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:35,327 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:35,327 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.50/24.10 kb and 12/27 files
2016-08-25 09:01:35,327 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.40/24.10 kb and 13/27 files
2016-08-25 09:01:35,328 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.30/24.10 kb and 14/27 files
Master started: node-1
======================================
47.4. Browse the queue on new master
======================================
2016-08-25 09:01:35,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.20/24.10 kb and 15/27 files
2016-08-25 09:01:35,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.09/24.10 kb and 16/27 files
2016-08-25 09:01:35,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.99/24.10 kb and 17/27 files
2016-08-25 09:01:35,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.89/24.10 kb and 18/27 files
2016-08-25 09:01:35,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.79/24.10 kb and 19/27 files
2016-08-25 09:01:35,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.69/24.10 kb and 20/27 files
2016-08-25 09:01:35,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.59/24.10 kb and 21/27 files
2016-08-25 09:01:35,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.49/24.10 kb and 22/27 files
2016-08-25 09:01:35,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.39/24.10 kb and 23/27 files
2016-08-25 09:01:35,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.29/24.10 kb and 24/27 files
2016-08-25 09:01:35,336 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files
2016-08-25 09:01:35,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files
2016-08-25 09:01:35,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files
2016-08-25 09:01:35,338 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:35,345 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
47.5. Found
======================================
======================================
48.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:35,351 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is shutting down
2016-08-25 09:01:35,351 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:35,355 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:35,357 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:35,357 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:35,357 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:35,357 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:35,360 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:35,361 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250030
2016-08-25 09:01:35,362 | INFO | Time-limited test | Session: 0x156bec917250030 closed
2016-08-25 09:01:35,362 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:35,362 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55287 which had sessionid 0x156bec917250030
2016-08-25 09:01:35,482 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:35,482 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) uptime 2.922 seconds
2016-08-25 09:01:35,482 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:49) is shutdown
Wait for master to start up...
2016-08-25 09:01:35,484 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:35,486 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7e333bfc
2016-08-25 09:01:35,487 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:35,487 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:35,488 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:35,488 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55311
2016-08-25 09:01:35,488 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55311
2016-08-25 09:01:35,489 | INFO | SyncThread:0 | Established session 0x156bec917250033 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55311
2016-08-25 09:01:35,489 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250033, negotiated timeout = 10000
2016-08-25 09:01:35,496 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:35,745 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:35,749 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:35,772 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:35,772 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:35,776 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:35,780 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55312
2016-08-25 09:01:35,979 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,979 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:35,980 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55312
2016-08-25 09:01:35,981 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:35,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55312
2016-08-25 09:01:35,983 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:35,997 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:36,151 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:36,152 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:36,153 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:36,153 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000073
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:36,154 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000075.log
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:36,155 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:36,156 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:36,157 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000075.log
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000073
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:36,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:36,159 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:36,160 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:36,173 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files
2016-08-25 09:01:36,174 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files
2016-08-25 09:01:36,175 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files
2016-08-25 09:01:36,175 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.65/24.10 kb and 4/27 files
2016-08-25 09:01:36,176 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.55/24.10 kb and 5/27 files
2016-08-25 09:01:36,177 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.45/24.10 kb and 6/27 files
2016-08-25 09:01:36,178 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.35/24.10 kb and 7/27 files
2016-08-25 09:01:36,178 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/24.10 kb and 8/27 files
2016-08-25 09:01:36,179 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.15/24.10 kb and 9/27 files
2016-08-25 09:01:36,179 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.42/24.10 kb and 10/27 files
2016-08-25 09:01:36,180 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.32/24.10 kb and 11/27 files
2016-08-25 09:01:36,181 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.22/24.10 kb and 12/27 files
2016-08-25 09:01:36,181 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.12/24.10 kb and 13/27 files
2016-08-25 09:01:36,182 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.01/24.10 kb and 14/27 files
2016-08-25 09:01:36,183 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.91/24.10 kb and 15/27 files
2016-08-25 09:01:36,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.81/24.10 kb and 16/27 files
2016-08-25 09:01:36,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.71/24.10 kb and 17/27 files
2016-08-25 09:01:36,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.73/24.10 kb and 18/27 files
2016-08-25 09:01:36,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.63/24.10 kb and 19/27 files
2016-08-25 09:01:36,186 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.52/24.10 kb and 20/27 files
2016-08-25 09:01:36,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.42/24.10 kb and 21/27 files
2016-08-25 09:01:36,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.61/24.10 kb and 22/27 files
2016-08-25 09:01:36,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.51/24.10 kb and 23/27 files
2016-08-25 09:01:36,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.41/24.10 kb and 24/27 files
2016-08-25 09:01:36,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files
2016-08-25 09:01:36,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files
2016-08-25 09:01:36,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files
2016-08-25 09:01:36,191 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:36,194 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:36,195 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is starting
2016-08-25 09:01:36,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/24.10 kb and 1/27 files
2016-08-25 09:01:36,199 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55317
2016-08-25 09:01:36,199 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:36,200 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) started
2016-08-25 09:01:36,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/24.10 kb and 2/27 files
2016-08-25 09:01:36,200 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:36,200 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:36,200 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:36,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/24.10 kb and 3/27 files
2016-08-25 09:01:36,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/24.10 kb and 4/27 files
2016-08-25 09:01:36,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/24.10 kb and 5/27 files
Master started: node-2
======================================
48.4. Browse the queue on new master
======================================
2016-08-25 09:01:36,202 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/24.10 kb and 6/27 files
2016-08-25 09:01:36,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.41/24.10 kb and 7/27 files
2016-08-25 09:01:36,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/24.10 kb and 8/27 files
2016-08-25 09:01:36,204 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.87/24.10 kb and 9/27 files
2016-08-25 09:01:36,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.77/24.10 kb and 10/27 files
2016-08-25 09:01:36,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.67/24.10 kb and 11/27 files
2016-08-25 09:01:36,206 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.57/24.10 kb and 12/27 files
2016-08-25 09:01:36,207 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.47/24.10 kb and 13/27 files
2016-08-25 09:01:36,208 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.37/24.10 kb and 14/27 files
2016-08-25 09:01:36,209 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.26/24.10 kb and 15/27 files
2016-08-25 09:01:36,210 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.16/24.10 kb and 16/27 files
2016-08-25 09:01:36,211 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.06/24.10 kb and 17/27 files
2016-08-25 09:01:36,212 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.01/24.10 kb and 18/27 files
2016-08-25 09:01:36,213 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.91/24.10 kb and 19/27 files
2016-08-25 09:01:36,213 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.80/24.10 kb and 20/27 files
2016-08-25 09:01:36,214 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.70/24.10 kb and 21/27 files
2016-08-25 09:01:36,215 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.60/24.10 kb and 22/27 files
2016-08-25 09:01:36,216 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.50/24.10 kb and 23/27 files
2016-08-25 09:01:36,216 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.40/24.10 kb and 24/27 files
2016-08-25 09:01:36,217 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/24.10 kb and 25/27 files
2016-08-25 09:01:36,218 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/24.10 kb and 26/27 files
2016-08-25 09:01:36,219 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.10/24.10 kb and 27/27 files
2016-08-25 09:01:36,219 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
48.5. Found
======================================
======================================
49.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:36,222 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is shutting down
2016-08-25 09:01:36,223 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:36,229 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:36,230 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:36,230 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250031
2016-08-25 09:01:36,232 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55295 which had sessionid 0x156bec917250031
2016-08-25 09:01:36,233 | INFO | Time-limited test | Session: 0x156bec917250031 closed
2016-08-25 09:01:36,234 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:36,238 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:36,238 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:36,238 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:36,239 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:36,338 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:36,338 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) uptime 2.772 seconds
2016-08-25 09:01:36,338 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:50) is shutdown
Wait for master to start up...
2016-08-25 09:01:36,340 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:36,343 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@39d47789
2016-08-25 09:01:36,343 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:36,344 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:36,344 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:36,344 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55319
2016-08-25 09:01:36,345 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55319
2016-08-25 09:01:36,345 | INFO | SyncThread:0 | Established session 0x156bec917250034 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55319
2016-08-25 09:01:36,345 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250034, negotiated timeout = 10000
2016-08-25 09:01:36,352 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:36,841 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:36,843 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:36,856 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:36,857 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:36,860 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:36,872 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:37,082 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55320
2016-08-25 09:01:37,087 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:37,088 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55320
2016-08-25 09:01:37,088 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:37,087 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:37,092 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55320
2016-08-25 09:01:37,092 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:37,293 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:37,306 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:37,306 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000076
2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:37,307 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:37,308 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000078.log
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:37,309 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:37,310 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000076
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:37,311 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:37,312 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000078.log
2016-08-25 09:01:37,327 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files
2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/25.06 kb and 2/28 files
2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files
2016-08-25 09:01:37,328 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.12/25.06 kb and 3/28 files
2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.02/25.06 kb and 4/28 files
2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/25.06 kb and 2/28 files
2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.92/25.06 kb and 5/28 files
2016-08-25 09:01:37,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/25.06 kb and 3/28 files
2016-08-25 09:01:37,330 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.82/25.06 kb and 6/28 files
2016-08-25 09:01:37,330 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.72/25.06 kb and 7/28 files
2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/25.06 kb and 4/28 files
2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.62/25.06 kb and 8/28 files
2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.49/25.06 kb and 5/28 files
2016-08-25 09:01:37,331 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.51/25.06 kb and 9/28 files
2016-08-25 09:01:37,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.39/25.06 kb and 6/28 files
2016-08-25 09:01:37,332 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.41/25.06 kb and 10/28 files
2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.31/25.06 kb and 11/28 files
2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.29/25.06 kb and 7/28 files
2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.21/25.06 kb and 12/28 files
2016-08-25 09:01:37,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.19/25.06 kb and 8/28 files
2016-08-25 09:01:37,334 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.16/25.06 kb and 13/28 files
2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.06/25.06 kb and 14/28 files
2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.09/25.06 kb and 9/28 files
2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.96/25.06 kb and 15/28 files
2016-08-25 09:01:37,335 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/25.06 kb and 10/28 files
2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.15/25.06 kb and 16/28 files
2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/25.06 kb and 11/28 files
2016-08-25 09:01:37,336 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.04/25.06 kb and 17/28 files
2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/25.06 kb and 12/28 files
2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.06/25.06 kb and 18/28 files
2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/25.06 kb and 13/28 files
2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.70/25.06 kb and 14/28 files
2016-08-25 09:01:37,337 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.96/25.06 kb and 19/28 files
2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.60/25.06 kb and 15/28 files
2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.86/25.06 kb and 20/28 files
2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.76/25.06 kb and 21/28 files
2016-08-25 09:01:37,338 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.50/25.06 kb and 16/28 files
2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.66/25.06 kb and 22/28 files
2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.40/25.06 kb and 17/28 files
2016-08-25 09:01:37,339 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.56/25.06 kb and 23/28 files
2016-08-25 09:01:37,340 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.30/25.06 kb and 18/28 files
2016-08-25 09:01:37,340 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.46/25.06 kb and 24/28 files
2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.20/25.06 kb and 19/28 files
2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.36/25.06 kb and 25/28 files
2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.10/25.06 kb and 20/28 files
2016-08-25 09:01:37,341 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.25/25.06 kb and 26/28 files
2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files
2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/25.06 kb and 21/28 files
2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.90/25.06 kb and 22/28 files
2016-08-25 09:01:37,342 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files
2016-08-25 09:01:37,343 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:37,343 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.22/25.06 kb and 23/28 files
2016-08-25 09:01:37,344 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.17/25.06 kb and 24/28 files
2016-08-25 09:01:37,346 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.07/25.06 kb and 25/28 files
2016-08-25 09:01:37,347 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.97/25.06 kb and 26/28 files
2016-08-25 09:01:37,347 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.87/25.06 kb and 27/28 files
2016-08-25 09:01:37,348 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files
2016-08-25 09:01:37,348 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:37,351 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:37,352 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is starting
2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55325
2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) started
2016-08-25 09:01:37,356 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:37,356 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:37,357 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-3
======================================
2016-08-25 09:01:37,358 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
49.4. Browse the queue on new master
======================================
======================================
49.5. Found
======================================
======================================
50.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:37,373 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is shutting down
2016-08-25 09:01:37,373 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:37,375 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:37,376 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:37,376 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250032
2016-08-25 09:01:37,377 | INFO | Time-limited test | Session: 0x156bec917250032 closed
2016-08-25 09:01:37,377 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:37,377 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55303 which had sessionid 0x156bec917250032
2016-08-25 09:01:37,379 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:37,379 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:37,379 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:37,379 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:37,496 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:37,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) uptime 3.091 seconds
2016-08-25 09:01:37,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:51) is shutdown
Wait for master to start up...
2016-08-25 09:01:37,499 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:37,501 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@566822b0
2016-08-25 09:01:37,501 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:37,502 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:37,502 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:37,503 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55327
2016-08-25 09:01:37,503 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55327
2016-08-25 09:01:37,503 | INFO | SyncThread:0 | Established session 0x156bec917250035 with negotiated timeout 10000 for client /127.0.0.1:55327
2016-08-25 09:01:37,504 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250035, negotiated timeout = 10000
2016-08-25 09:01:37,510 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:37,784 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:37,787 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:37,794 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:37,794 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:37,798 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:37,804 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:38,000 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55328
2016-08-25 09:01:38,006 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:38,007 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:38,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55328
2016-08-25 09:01:38,008 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:38,010 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55328
2016-08-25 09:01:38,010 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:38,024 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:38,166 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:38,167 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:38,167 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000076
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:38,168 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000078.log
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:38,169 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:38,170 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000076
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:38,171 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:38,172 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000078.log
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:38,173 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:38,174 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:38,174 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:38,184 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files
2016-08-25 09:01:38,185 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/25.06 kb and 2/28 files
2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.69/25.06 kb and 3/28 files
2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.59/25.06 kb and 4/28 files
2016-08-25 09:01:38,187 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/25.06 kb and 1/28 files
2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.92/25.06 kb and 2/28 files
2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.49/25.06 kb and 5/28 files
2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.39/25.06 kb and 6/28 files
2016-08-25 09:01:38,188 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/25.06 kb and 3/28 files
2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/25.06 kb and 4/28 files
2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.29/25.06 kb and 7/28 files
2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/25.06 kb and 5/28 files
2016-08-25 09:01:38,189 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.19/25.06 kb and 8/28 files
2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.56/25.06 kb and 6/28 files
2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.09/25.06 kb and 9/28 files
2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.89/25.06 kb and 7/28 files
2016-08-25 09:01:38,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.99/25.06 kb and 10/28 files
2016-08-25 09:01:38,191 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.79/25.06 kb and 8/28 files
2016-08-25 09:01:38,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.31/25.06 kb and 11/28 files
2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.68/25.06 kb and 9/28 files
2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.21/25.06 kb and 12/28 files
2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.11/25.06 kb and 13/28 files
2016-08-25 09:01:38,192 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.58/25.06 kb and 10/28 files
2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.48/25.06 kb and 11/28 files
2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.01/25.06 kb and 14/28 files
2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.38/25.06 kb and 12/28 files
2016-08-25 09:01:38,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.91/25.06 kb and 15/28 files
2016-08-25 09:01:38,194 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.28/25.06 kb and 13/28 files
2016-08-25 09:01:38,194 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.81/25.06 kb and 16/28 files
2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.18/25.06 kb and 14/28 files
2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.71/25.06 kb and 17/28 files
2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.36/25.06 kb and 15/28 files
2016-08-25 09:01:38,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.61/25.06 kb and 18/28 files
2016-08-25 09:01:38,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.62/25.06 kb and 19/28 files
2016-08-25 09:01:38,196 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.26/25.06 kb and 16/28 files
2016-08-25 09:01:38,197 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.81/25.06 kb and 20/28 files
2016-08-25 09:01:38,197 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.16/25.06 kb and 17/28 files
2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.71/25.06 kb and 21/28 files
2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.06/25.06 kb and 18/28 files
2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.96/25.06 kb and 19/28 files
2016-08-25 09:01:38,198 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.60/25.06 kb and 22/28 files
2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.86/25.06 kb and 20/28 files
2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.50/25.06 kb and 23/28 files
2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.40/25.06 kb and 24/28 files
2016-08-25 09:01:38,199 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.76/25.06 kb and 21/28 files
2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.66/25.06 kb and 22/28 files
2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.31/25.06 kb and 25/28 files
2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.26/25.06 kb and 26/28 files
2016-08-25 09:01:38,200 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.56/25.06 kb and 23/28 files
2016-08-25 09:01:38,201 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.46/25.06 kb and 24/28 files
2016-08-25 09:01:38,201 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files
2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.36/25.06 kb and 25/28 files
2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files
2016-08-25 09:01:38,202 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:38,203 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.26/25.06 kb and 26/28 files
2016-08-25 09:01:38,204 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.16/25.06 kb and 27/28 files
2016-08-25 09:01:38,205 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.06/25.06 kb and 28/28 files
2016-08-25 09:01:38,205 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:38,206 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:38,208 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is starting
2016-08-25 09:01:38,211 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55333
2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) started
2016-08-25 09:01:38,212 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:38,212 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:38,212 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-1
======================================
50.4. Browse the queue on new master
======================================
======================================
50.5. Found
======================================
======================================
51.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:38,229 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is shutting down
2016-08-25 09:01:38,229 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:38,232 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:38,234 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:38,234 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250033
2016-08-25 09:01:38,235 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55311 which had sessionid 0x156bec917250033
2016-08-25 09:01:38,235 | INFO | Time-limited test | Session: 0x156bec917250033 closed
2016-08-25 09:01:38,235 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:38,237 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:38,237 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:38,238 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:38,238 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:38,339 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:38,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) uptime 2.857 seconds
2016-08-25 09:01:38,340 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:52) is shutdown
Wait for master to start up...
2016-08-25 09:01:38,343 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:38,346 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6fac659
2016-08-25 09:01:38,346 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:38,347 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:38,347 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:38,348 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55335
2016-08-25 09:01:38,348 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55335
2016-08-25 09:01:38,349 | INFO | SyncThread:0 | Established session 0x156bec917250036 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55335
2016-08-25 09:01:38,349 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250036, negotiated timeout = 10000
2016-08-25 09:01:38,361 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:38,796 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:38,798 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:38,798 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:38,799 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:38,802 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:38,806 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:39,025 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55336
2016-08-25 09:01:39,030 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:39,030 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:39,032 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55336
2016-08-25 09:01:39,032 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:39,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55336
2016-08-25 09:01:39,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:39,219 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:39,236 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:39,236 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000081.log
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:39,237 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000079
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:39,238 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:39,239 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:39,240 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:39,240 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000079
2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:39,241 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:39,242 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:39,243 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:39,244 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000081.log
2016-08-25 09:01:39,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files
2016-08-25 09:01:39,260 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files
2016-08-25 09:01:39,261 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.75/26.01 kb and 3/29 files
2016-08-25 09:01:39,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/26.01 kb and 4/29 files
2016-08-25 09:01:39,262 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/26.01 kb and 5/29 files
2016-08-25 09:01:39,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/26.01 kb and 6/29 files
2016-08-25 09:01:39,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/26.01 kb and 7/29 files
2016-08-25 09:01:39,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.53/26.01 kb and 8/29 files
2016-08-25 09:01:39,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.43/26.01 kb and 9/29 files
2016-08-25 09:01:39,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.33/26.01 kb and 10/29 files
2016-08-25 09:01:39,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.23/26.01 kb and 11/29 files
2016-08-25 09:01:39,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.13/26.01 kb and 12/29 files
2016-08-25 09:01:39,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.03/26.01 kb and 13/29 files
2016-08-25 09:01:39,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.92/26.01 kb and 14/29 files
2016-08-25 09:01:39,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.82/26.01 kb and 15/29 files
2016-08-25 09:01:39,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.72/26.01 kb and 16/29 files
2016-08-25 09:01:39,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.62/26.01 kb and 17/29 files
2016-08-25 09:01:39,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.99/26.01 kb and 18/29 files
2016-08-25 09:01:39,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.89/26.01 kb and 19/29 files
2016-08-25 09:01:39,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.79/26.01 kb and 20/29 files
2016-08-25 09:01:39,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.69/26.01 kb and 21/29 files
2016-08-25 09:01:39,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.59/26.01 kb and 22/29 files
2016-08-25 09:01:39,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.49/26.01 kb and 23/29 files
2016-08-25 09:01:39,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.39/26.01 kb and 24/29 files
2016-08-25 09:01:39,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.29/26.01 kb and 25/29 files
2016-08-25 09:01:39,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.19/26.01 kb and 26/29 files
2016-08-25 09:01:39,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.09/26.01 kb and 27/29 files
2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files
2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files
2016-08-25 09:01:39,276 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:39,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files
2016-08-25 09:01:39,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files
2016-08-25 09:01:39,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/26.01 kb and 3/29 files
2016-08-25 09:01:39,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.07/26.01 kb and 4/29 files
2016-08-25 09:01:39,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.97/26.01 kb and 5/29 files
2016-08-25 09:01:39,280 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.87/26.01 kb and 6/29 files
2016-08-25 09:01:39,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.77/26.01 kb and 7/29 files
2016-08-25 09:01:39,281 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.67/26.01 kb and 8/29 files
2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.57/26.01 kb and 9/29 files
2016-08-25 09:01:39,282 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.47/26.01 kb and 10/29 files
2016-08-25 09:01:39,282 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is starting
2016-08-25 09:01:39,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.37/26.01 kb and 11/29 files
2016-08-25 09:01:39,283 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.27/26.01 kb and 12/29 files
2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.17/26.01 kb and 13/29 files
2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.07/26.01 kb and 14/29 files
2016-08-25 09:01:39,284 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.02/26.01 kb and 15/29 files
2016-08-25 09:01:39,285 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.92/26.01 kb and 16/29 files
2016-08-25 09:01:39,286 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55341
2016-08-25 09:01:39,286 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:39,287 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) started
2016-08-25 09:01:39,287 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:39,287 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:39,287 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:39,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.82/26.01 kb and 17/29 files
2016-08-25 09:01:39,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.72/26.01 kb and 18/29 files
2016-08-25 09:01:39,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.61/26.01 kb and 19/29 files
Master started: node-2
======================================
51.4. Browse the queue on new master
======================================
2016-08-25 09:01:39,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.51/26.01 kb and 20/29 files
2016-08-25 09:01:39,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.41/26.01 kb and 21/29 files
2016-08-25 09:01:39,292 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.31/26.01 kb and 22/29 files
2016-08-25 09:01:39,293 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.21/26.01 kb and 23/29 files
2016-08-25 09:01:39,293 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.11/26.01 kb and 24/29 files
2016-08-25 09:01:39,294 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.01/26.01 kb and 25/29 files
2016-08-25 09:01:39,295 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.02/26.01 kb and 26/29 files
2016-08-25 09:01:39,295 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.92/26.01 kb and 27/29 files
2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.82/26.01 kb and 28/29 files
2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files
2016-08-25 09:01:39,296 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
51.5. Found
======================================
======================================
52.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:39,306 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is shutting down
2016-08-25 09:01:39,306 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:39,306 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:39,308 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:39,308 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250034
2016-08-25 09:01:39,309 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:39,309 | INFO | Time-limited test | Session: 0x156bec917250034 closed
2016-08-25 09:01:39,309 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:39,309 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55319 which had sessionid 0x156bec917250034
2016-08-25 09:01:39,312 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:39,312 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:39,312 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:39,312 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:39,437 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:39,438 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) uptime 3.099 seconds
2016-08-25 09:01:39,438 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:53) is shutdown
Wait for master to start up...
2016-08-25 09:01:39,440 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:39,445 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4ceb4f17
2016-08-25 09:01:39,445 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:39,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:39,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:39,446 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55343
2016-08-25 09:01:39,446 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55343
2016-08-25 09:01:39,447 | INFO | SyncThread:0 | Established session 0x156bec917250037 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55343
2016-08-25 09:01:39,447 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250037, negotiated timeout = 10000
2016-08-25 09:01:39,454 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:39,760 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:39,762 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:39,767 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:39,767 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:39,771 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:39,776 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:39,998 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55344
2016-08-25 09:01:40,005 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:40,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:40,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55344
2016-08-25 09:01:40,007 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:40,010 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55344
2016-08-25 09:01:40,010 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:40,021 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:40,206 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000081.log
2016-08-25 09:01:40,207 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000079
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:40,208 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:40,209 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:40,210 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:40,211 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:40,211 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000081.log
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000079
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:40,212 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:40,213 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:40,214 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:40,215 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:40,223 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files
2016-08-25 09:01:40,224 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/26.01 kb and 2/29 files
2016-08-25 09:01:40,225 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/26.01 kb and 3/29 files
2016-08-25 09:01:40,225 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/26.01 kb and 4/29 files
2016-08-25 09:01:40,226 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/26.01 kb and 5/29 files
2016-08-25 09:01:40,227 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/26.01 kb and 6/29 files
2016-08-25 09:01:40,228 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.63/26.01 kb and 7/29 files
2016-08-25 09:01:40,229 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.01/26.01 kb and 8/29 files
2016-08-25 09:01:40,230 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.91/26.01 kb and 9/29 files
2016-08-25 09:01:40,230 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.80/26.01 kb and 10/29 files
2016-08-25 09:01:40,231 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.70/26.01 kb and 11/29 files
2016-08-25 09:01:40,231 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.60/26.01 kb and 12/29 files
2016-08-25 09:01:40,232 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.50/26.01 kb and 13/29 files
2016-08-25 09:01:40,233 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.40/26.01 kb and 14/29 files
2016-08-25 09:01:40,233 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.30/26.01 kb and 15/29 files
2016-08-25 09:01:40,234 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.20/26.01 kb and 16/29 files
2016-08-25 09:01:40,234 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.22/26.01 kb and 17/29 files
2016-08-25 09:01:40,235 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.12/26.01 kb and 18/29 files
2016-08-25 09:01:40,236 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.02/26.01 kb and 19/29 files
2016-08-25 09:01:40,236 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.92/26.01 kb and 20/29 files
2016-08-25 09:01:40,237 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.81/26.01 kb and 21/29 files
2016-08-25 09:01:40,237 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.71/26.01 kb and 22/29 files
2016-08-25 09:01:40,238 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.61/26.01 kb and 23/29 files
2016-08-25 09:01:40,239 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.51/26.01 kb and 24/29 files
2016-08-25 09:01:40,239 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.41/26.01 kb and 25/29 files
2016-08-25 09:01:40,240 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.31/26.01 kb and 26/29 files
2016-08-25 09:01:40,240 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.21/26.01 kb and 27/29 files
2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.01 kb and 1/29 files
2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files
2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.01 kb and 2/29 files
2016-08-25 09:01:40,241 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files
2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.99/26.01 kb and 3/29 files
2016-08-25 09:01:40,242 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/26.01 kb and 4/29 files
2016-08-25 09:01:40,243 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/26.01 kb and 5/29 files
2016-08-25 09:01:40,250 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:40,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/26.01 kb and 6/29 files
2016-08-25 09:01:40,251 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is starting
2016-08-25 09:01:40,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.06/26.01 kb and 7/29 files
2016-08-25 09:01:40,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.96/26.01 kb and 8/29 files
2016-08-25 09:01:40,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.85/26.01 kb and 9/29 files
2016-08-25 09:01:40,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.80/26.01 kb and 10/29 files
2016-08-25 09:01:40,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.70/26.01 kb and 11/29 files
2016-08-25 09:01:40,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.60/26.01 kb and 12/29 files
2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55349
2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) started
2016-08-25 09:01:40,255 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:40,255 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:40,255 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:40,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.62/26.01 kb and 13/29 files
2016-08-25 09:01:40,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.52/26.01 kb and 14/29 files
Master started: node-3
======================================
52.4. Browse the queue on new master
======================================
2016-08-25 09:01:40,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.42/26.01 kb and 15/29 files
2016-08-25 09:01:40,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.32/26.01 kb and 16/29 files
2016-08-25 09:01:40,260 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.21/26.01 kb and 17/29 files
2016-08-25 09:01:40,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.11/26.01 kb and 18/29 files
2016-08-25 09:01:40,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.01/26.01 kb and 19/29 files
2016-08-25 09:01:40,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.91/26.01 kb and 20/29 files
2016-08-25 09:01:40,264 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.81/26.01 kb and 21/29 files
2016-08-25 09:01:40,265 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.71/26.01 kb and 22/29 files
2016-08-25 09:01:40,266 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.61/26.01 kb and 23/29 files
2016-08-25 09:01:40,266 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.51/26.01 kb and 24/29 files
======================================
52.5. Found
======================================
======================================
53.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:40,269 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.41/26.01 kb and 25/29 files
2016-08-25 09:01:40,271 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is shutting down
2016-08-25 09:01:40,272 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:40,274 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:40,275 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:40,275 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250035
2016-08-25 09:01:40,275 | INFO | Time-limited test | Session: 0x156bec917250035 closed
2016-08-25 09:01:40,276 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55327 which had sessionid 0x156bec917250035
2016-08-25 09:01:40,276 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:40,276 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:40,276 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:40,276 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.31/26.01 kb and 26/29 files
2016-08-25 09:01:40,277 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.21/26.01 kb and 27/29 files
2016-08-25 09:01:40,278 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.11/26.01 kb and 28/29 files
2016-08-25 09:01:40,279 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.01/26.01 kb and 29/29 files
2016-08-25 09:01:40,279 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:40,289 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:40,289 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:40,379 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:40,380 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) uptime 2.883 seconds
2016-08-25 09:01:40,380 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:54) is shutdown
Wait for master to start up...
2016-08-25 09:01:40,382 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:40,385 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4e326f04
2016-08-25 09:01:40,385 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:40,386 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:40,386 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:40,386 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55351
2016-08-25 09:01:40,387 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55351
2016-08-25 09:01:40,387 | INFO | SyncThread:0 | Established session 0x156bec917250038 with negotiated timeout 10000 for client /127.0.0.1:55351
2016-08-25 09:01:40,387 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250038, negotiated timeout = 10000
2016-08-25 09:01:40,395 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:40,884 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:40,887 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:40,917 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:40,917 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:40,921 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:40,925 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:41,154 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55352
2016-08-25 09:01:41,158 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:41,158 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:41,160 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55352
2016-08-25 09:01:41,160 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:41,162 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55352
2016-08-25 09:01:41,162 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:41,339 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:41,350 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:41,351 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000084.log
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:41,352 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000082
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:41,353 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:41,354 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000084.log
2016-08-25 09:01:41,355 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000082
2016-08-25 09:01:41,356 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:41,357 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:41,367 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files
2016-08-25 09:01:41,367 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files
2016-08-25 09:01:41,368 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/26.96 kb and 3/30 files
2016-08-25 09:01:41,368 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/26.96 kb and 4/30 files
2016-08-25 09:01:41,369 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/26.96 kb and 5/30 files
2016-08-25 09:01:41,369 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/26.96 kb and 6/30 files
2016-08-25 09:01:41,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/26.96 kb and 7/30 files
2016-08-25 09:01:41,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/26.96 kb and 8/30 files
2016-08-25 09:01:41,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/26.96 kb and 9/30 files
2016-08-25 09:01:41,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.99/26.96 kb and 10/30 files
2016-08-25 09:01:41,372 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/26.96 kb and 11/30 files
2016-08-25 09:01:41,373 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/26.96 kb and 12/30 files
2016-08-25 09:01:41,373 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.69/26.96 kb and 13/30 files
2016-08-25 09:01:41,374 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.59/26.96 kb and 14/30 files
2016-08-25 09:01:41,374 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.49/26.96 kb and 15/30 files
2016-08-25 09:01:41,375 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.39/26.96 kb and 16/30 files
2016-08-25 09:01:41,375 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.29/26.96 kb and 17/30 files
2016-08-25 09:01:41,376 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.19/26.96 kb and 18/30 files
2016-08-25 09:01:41,377 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.37/26.96 kb and 19/30 files
2016-08-25 09:01:41,377 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.27/26.96 kb and 20/30 files
2016-08-25 09:01:41,378 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.17/26.96 kb and 21/30 files
2016-08-25 09:01:41,378 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.07/26.96 kb and 22/30 files
2016-08-25 09:01:41,379 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.97/26.96 kb and 23/30 files
2016-08-25 09:01:41,379 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.87/26.96 kb and 24/30 files
2016-08-25 09:01:41,380 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.29/26.96 kb and 25/30 files
2016-08-25 09:01:41,381 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.31/26.96 kb and 26/30 files
2016-08-25 09:01:41,381 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.26/26.96 kb and 27/30 files
2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.16/26.96 kb and 28/30 files
2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files
2016-08-25 09:01:41,382 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files
2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files
2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files
2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:41,383 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.98/26.96 kb and 3/30 files
2016-08-25 09:01:41,384 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.88/26.96 kb and 4/30 files
2016-08-25 09:01:41,384 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.78/26.96 kb and 5/30 files
2016-08-25 09:01:41,385 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/26.96 kb and 6/30 files
2016-08-25 09:01:41,385 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/26.96 kb and 7/30 files
2016-08-25 09:01:41,386 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/26.96 kb and 8/30 files
2016-08-25 09:01:41,387 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/26.96 kb and 9/30 files
2016-08-25 09:01:41,387 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.28/26.96 kb and 10/30 files
2016-08-25 09:01:41,388 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.18/26.96 kb and 11/30 files
2016-08-25 09:01:41,388 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.08/26.96 kb and 12/30 files
2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.98/26.96 kb and 13/30 files
2016-08-25 09:01:41,389 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.88/26.96 kb and 14/30 files
2016-08-25 09:01:41,390 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is starting
2016-08-25 09:01:41,390 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.78/26.96 kb and 15/30 files
2016-08-25 09:01:41,391 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.68/26.96 kb and 16/30 files
2016-08-25 09:01:41,391 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.58/26.96 kb and 17/30 files
2016-08-25 09:01:41,392 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.00/26.96 kb and 18/30 files
2016-08-25 09:01:41,392 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.90/26.96 kb and 19/30 files
2016-08-25 09:01:41,393 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.80/26.96 kb and 20/30 files
2016-08-25 09:01:41,394 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.70/26.96 kb and 21/30 files
2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55357
2016-08-25 09:01:41,395 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.59/26.96 kb and 22/30 files
2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) started
2016-08-25 09:01:41,395 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:41,395 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:41,395 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:41,396 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.61/26.96 kb and 23/30 files
2016-08-25 09:01:41,396 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.51/26.96 kb and 24/30 files
2016-08-25 09:01:41,397 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.41/26.96 kb and 25/30 files
Master started: node-1
======================================
53.4. Browse the queue on new master
======================================
2016-08-25 09:01:41,398 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.31/26.96 kb and 26/30 files
2016-08-25 09:01:41,398 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.21/26.96 kb and 27/30 files
2016-08-25 09:01:41,399 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.11/26.96 kb and 28/30 files
2016-08-25 09:01:41,400 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files
2016-08-25 09:01:41,400 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files
2016-08-25 09:01:41,401 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:41,407 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
53.5. Found
======================================
======================================
54.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:41,416 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is shutting down
2016-08-25 09:01:41,416 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:41,420 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:41,421 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:41,421 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:41,421 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:41,421 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:41,422 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:41,422 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250036
2016-08-25 09:01:41,423 | INFO | Time-limited test | Session: 0x156bec917250036 closed
2016-08-25 09:01:41,423 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55335 which had sessionid 0x156bec917250036
2016-08-25 09:01:41,423 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:41,546 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:41,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) uptime 3.205 seconds
2016-08-25 09:01:41,547 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:55) is shutdown
Wait for master to start up...
2016-08-25 09:01:41,549 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:41,551 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@25d73bfd
2016-08-25 09:01:41,551 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:41,552 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:41,553 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:41,553 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55359
2016-08-25 09:01:41,553 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55359
2016-08-25 09:01:41,554 | INFO | SyncThread:0 | Established session 0x156bec917250039 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55359
2016-08-25 09:01:41,554 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250039, negotiated timeout = 10000
2016-08-25 09:01:41,561 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:41,843 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:41,845 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:41,856 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:41,857 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:41,860 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:41,868 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:42,097 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55360
2016-08-25 09:01:42,103 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:42,103 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:42,105 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55360
2016-08-25 09:01:42,105 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:42,108 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55360
2016-08-25 09:01:42,108 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:42,122 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:42,125 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:42,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:42,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000084.log
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000082
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:42,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:42,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:42,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/26.96 kb and 1/30 files
2016-08-25 09:01:42,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/26.96 kb and 2/30 files
2016-08-25 09:01:42,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/26.96 kb and 3/30 files
2016-08-25 09:01:42,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/26.96 kb and 4/30 files
2016-08-25 09:01:42,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/26.96 kb and 5/30 files
2016-08-25 09:01:42,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/26.96 kb and 6/30 files
2016-08-25 09:01:42,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/26.96 kb and 7/30 files
2016-08-25 09:01:42,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.32/26.96 kb and 8/30 files
2016-08-25 09:01:42,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.27/26.96 kb and 9/30 files
2016-08-25 09:01:42,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.17/26.96 kb and 10/30 files
2016-08-25 09:01:42,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.06/26.96 kb and 11/30 files
2016-08-25 09:01:42,151 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.96/26.96 kb and 12/30 files
2016-08-25 09:01:42,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.86/26.96 kb and 13/30 files
2016-08-25 09:01:42,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.76/26.96 kb and 14/30 files
2016-08-25 09:01:42,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.66/26.96 kb and 15/30 files
2016-08-25 09:01:42,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.56/26.96 kb and 16/30 files
2016-08-25 09:01:42,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.75/26.96 kb and 17/30 files
2016-08-25 09:01:42,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.64/26.96 kb and 18/30 files
2016-08-25 09:01:42,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.54/26.96 kb and 19/30 files
2016-08-25 09:01:42,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.44/26.96 kb and 20/30 files
2016-08-25 09:01:42,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.34/26.96 kb and 21/30 files
2016-08-25 09:01:42,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.24/26.96 kb and 22/30 files
2016-08-25 09:01:42,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.66/26.96 kb and 23/30 files
2016-08-25 09:01:42,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.56/26.96 kb and 24/30 files
2016-08-25 09:01:42,160 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.46/26.96 kb and 25/30 files
2016-08-25 09:01:42,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.36/26.96 kb and 26/30 files
2016-08-25 09:01:42,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.26/26.96 kb and 27/30 files
2016-08-25 09:01:42,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.16/26.96 kb and 28/30 files
2016-08-25 09:01:42,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.06/26.96 kb and 29/30 files
2016-08-25 09:01:42,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.96/26.96 kb and 30/30 files
2016-08-25 09:01:42,163 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:42,165 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:42,166 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is starting
2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55364
2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) started
2016-08-25 09:01:42,170 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:42,170 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:42,171 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-2
======================================
54.4. Browse the queue on new master
======================================
======================================
54.5. Found
======================================
======================================
55.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:42,188 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is shutting down
2016-08-25 09:01:42,188 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:42,190 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:42,191 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:42,191 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:42,192 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:42,192 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250037
2016-08-25 09:01:42,192 | INFO | Time-limited test | Session: 0x156bec917250037 closed
2016-08-25 09:01:42,193 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55343 which had sessionid 0x156bec917250037
2016-08-25 09:01:42,193 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:42,298 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:42,298 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) uptime 2.859 seconds
2016-08-25 09:01:42,299 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:56) is shutdown
Wait for master to start up...
2016-08-25 09:01:42,301 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:42,303 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@77bac550
2016-08-25 09:01:42,303 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:42,304 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:42,305 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:42,305 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55366
2016-08-25 09:01:42,305 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55366
2016-08-25 09:01:42,307 | INFO | SyncThread:0 | Established session 0x156bec91725003a with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55366
2016-08-25 09:01:42,307 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003a, negotiated timeout = 10000
2016-08-25 09:01:42,313 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:42,532 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:42,532 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:42,532 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:42,534 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:42,784 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:42,788 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:42,792 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:42,999 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55367
2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55367
2016-08-25 09:01:43,004 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:43,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55367
2016-08-25 09:01:43,005 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:43,224 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:43,227 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000085
2016-08-25 09:01:43,228 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000087.log
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:43,229 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:43,230 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:43,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/27.91 kb and 1/31 files
2016-08-25 09:01:43,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/27.91 kb and 2/31 files
2016-08-25 09:01:43,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/27.91 kb and 3/31 files
2016-08-25 09:01:43,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/27.91 kb and 4/31 files
2016-08-25 09:01:43,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/27.91 kb and 5/31 files
2016-08-25 09:01:43,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.02/27.91 kb and 6/31 files
2016-08-25 09:01:43,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.92/27.91 kb and 7/31 files
2016-08-25 09:01:43,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.82/27.91 kb and 8/31 files
2016-08-25 09:01:43,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.72/27.91 kb and 9/31 files
2016-08-25 09:01:43,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/27.91 kb and 10/31 files
2016-08-25 09:01:43,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.80/27.91 kb and 11/31 files
2016-08-25 09:01:43,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.70/27.91 kb and 12/31 files
2016-08-25 09:01:43,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.60/27.91 kb and 13/31 files
2016-08-25 09:01:43,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.50/27.91 kb and 14/31 files
2016-08-25 09:01:43,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.40/27.91 kb and 15/31 files
2016-08-25 09:01:43,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.30/27.91 kb and 16/31 files
2016-08-25 09:01:43,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.20/27.91 kb and 17/31 files
2016-08-25 09:01:43,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.10/27.91 kb and 18/31 files
2016-08-25 09:01:43,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/27.91 kb and 19/31 files
2016-08-25 09:01:43,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.90/27.91 kb and 20/31 files
2016-08-25 09:01:43,258 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.80/27.91 kb and 21/31 files
2016-08-25 09:01:43,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.70/27.91 kb and 22/31 files
2016-08-25 09:01:43,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.59/27.91 kb and 23/31 files
2016-08-25 09:01:43,260 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.49/27.91 kb and 24/31 files
2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.39/27.91 kb and 25/31 files
2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.29/27.91 kb and 26/31 files
2016-08-25 09:01:43,261 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.19/27.91 kb and 27/31 files
2016-08-25 09:01:43,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.09/27.91 kb and 28/31 files
2016-08-25 09:01:43,262 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.99/27.91 kb and 29/31 files
2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.01/27.91 kb and 30/31 files
2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.91/27.91 kb and 31/31 files
2016-08-25 09:01:43,263 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:43,267 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:43,268 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:43,268 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is starting
2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:43,269 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000087.log
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:43,270 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000085
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:43,271 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:43,272 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55372
2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) started
2016-08-25 09:01:43,273 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:43,273 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:43,273 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-1
======================================
55.4. Browse the queue on new master
======================================
======================================
55.5. Found
======================================
======================================
56.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:43,289 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is shutting down
2016-08-25 09:01:43,290 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:43,294 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:43,294 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:43,294 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:43,295 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250039
2016-08-25 09:01:43,296 | INFO | Time-limited test | Session: 0x156bec917250039 closed
2016-08-25 09:01:43,296 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55359 which had sessionid 0x156bec917250039
2016-08-25 09:01:43,296 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:43,298 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:43,298 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:43,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:43,299 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:43,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:43,420 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:43,420 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) uptime 1.873 seconds
2016-08-25 09:01:43,420 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:57) is shutdown
Wait for master to start up...
2016-08-25 09:01:43,423 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:43,425 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6fdbce7c
2016-08-25 09:01:43,425 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:43,426 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:43,427 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:43,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55374
2016-08-25 09:01:43,428 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55374
2016-08-25 09:01:43,429 | INFO | SyncThread:0 | Established session 0x156bec91725003b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55374
2016-08-25 09:01:43,429 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003b, negotiated timeout = 10000
2016-08-25 09:01:43,437 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:43,522 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:43,525 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:43,727 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:43,727 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:43,731 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:43,734 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:43,928 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55375
2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55375
2016-08-25 09:01:43,934 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:43,936 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55375
2016-08-25 09:01:43,936 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:43,952 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:43,956 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000085
2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:43,957 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000087.log
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:43,958 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:43,959 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:43,981 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.48/27.91 kb and 1/31 files
2016-08-25 09:01:43,982 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.38/27.91 kb and 2/31 files
2016-08-25 09:01:43,982 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.27/27.91 kb and 3/31 files
2016-08-25 09:01:43,983 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.17/27.91 kb and 4/31 files
2016-08-25 09:01:43,984 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.07/27.91 kb and 5/31 files
2016-08-25 09:01:43,984 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.97/27.91 kb and 6/31 files
2016-08-25 09:01:43,985 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.87/27.91 kb and 7/31 files
2016-08-25 09:01:43,985 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.77/27.91 kb and 8/31 files
2016-08-25 09:01:43,986 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.67/27.91 kb and 9/31 files
2016-08-25 09:01:43,986 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.57/27.91 kb and 10/31 files
2016-08-25 09:01:43,987 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.47/27.91 kb and 11/31 files
2016-08-25 09:01:43,987 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.65/27.91 kb and 12/31 files
2016-08-25 09:01:43,988 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.56/27.91 kb and 13/31 files
2016-08-25 09:01:43,988 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.57/27.91 kb and 14/31 files
2016-08-25 09:01:43,989 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.47/27.91 kb and 15/31 files
2016-08-25 09:01:43,989 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.37/27.91 kb and 16/31 files
2016-08-25 09:01:43,990 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.27/27.91 kb and 17/31 files
2016-08-25 09:01:43,991 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.17/27.91 kb and 18/31 files
2016-08-25 09:01:43,991 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.07/27.91 kb and 19/31 files
2016-08-25 09:01:43,992 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.96/27.91 kb and 20/31 files
2016-08-25 09:01:43,993 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.86/27.91 kb and 21/31 files
2016-08-25 09:01:43,993 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.76/27.91 kb and 22/31 files
2016-08-25 09:01:43,994 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.66/27.91 kb and 23/31 files
2016-08-25 09:01:43,995 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.56/27.91 kb and 24/31 files
2016-08-25 09:01:43,995 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.46/27.91 kb and 25/31 files
2016-08-25 09:01:43,996 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.36/27.91 kb and 26/31 files
2016-08-25 09:01:43,996 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.26/27.91 kb and 27/31 files
2016-08-25 09:01:43,997 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.16/27.91 kb and 28/31 files
2016-08-25 09:01:43,997 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.06/27.91 kb and 29/31 files
2016-08-25 09:01:43,998 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.01/27.91 kb and 30/31 files
2016-08-25 09:01:43,999 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.91/27.91 kb and 31/31 files
2016-08-25 09:01:43,999 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:44,001 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:44,002 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is starting
2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55379
2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) started
2016-08-25 09:01:44,006 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:44,006 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:44,006 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-3
======================================
56.4. Browse the queue on new master
======================================
======================================
56.5. Found
======================================
======================================
57.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:44,026 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is shutting down
2016-08-25 09:01:44,027 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:44,030 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:44,031 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:44,032 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:44,032 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:44,032 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250038
2016-08-25 09:01:44,033 | INFO | Time-limited test | Session: 0x156bec917250038 closed
2016-08-25 09:01:44,033 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:44,033 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55351 which had sessionid 0x156bec917250038
2016-08-25 09:01:44,138 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:44,139 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) uptime 3.759 seconds
2016-08-25 09:01:44,139 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:58) is shutdown
Wait for master to start up...
2016-08-25 09:01:44,141 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:44,144 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3d2bbb19
2016-08-25 09:01:44,144 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:44,145 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:44,145 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:44,145 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55381
2016-08-25 09:01:44,145 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55381
2016-08-25 09:01:44,146 | INFO | SyncThread:0 | Established session 0x156bec91725003c with negotiated timeout 10000 for client /127.0.0.1:55381
2016-08-25 09:01:44,146 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725003c, negotiated timeout = 10000
2016-08-25 09:01:44,155 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:44,373 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:44,373 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:44,373 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:44,375 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:44,669 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:44,673 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:44,677 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:44,881 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55382
2016-08-25 09:01:44,886 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55382
2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:44,887 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:44,888 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55382
2016-08-25 09:01:44,888 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:45,096 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:45,110 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:45,110 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:45,112 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:45,113 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:45,114 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000088
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000090.log
2016-08-25 09:01:45,115 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:45,116 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000088
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000090.log
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:45,117 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:45,118 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:45,128 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files
2016-08-25 09:01:45,128 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files
2016-08-25 09:01:45,129 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files
2016-08-25 09:01:45,129 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files
2016-08-25 09:01:45,130 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files
2016-08-25 09:01:45,130 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files
2016-08-25 09:01:45,131 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files
2016-08-25 09:01:45,132 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files
2016-08-25 09:01:45,132 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/28.86 kb and 9/32 files
2016-08-25 09:01:45,133 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.00/28.86 kb and 10/32 files
2016-08-25 09:01:45,133 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.01/28.86 kb and 11/32 files
2016-08-25 09:01:45,134 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.91/28.86 kb and 12/32 files
2016-08-25 09:01:45,135 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.86/28.86 kb and 13/32 files
2016-08-25 09:01:45,135 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.76/28.86 kb and 14/32 files
2016-08-25 09:01:45,136 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/28.86 kb and 15/32 files
2016-08-25 09:01:45,136 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/28.86 kb and 16/32 files
2016-08-25 09:01:45,137 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files
2016-08-25 09:01:45,137 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.36/28.86 kb and 18/32 files
2016-08-25 09:01:45,138 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.26/28.86 kb and 19/32 files
2016-08-25 09:01:45,139 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.16/28.86 kb and 20/32 files
2016-08-25 09:01:45,139 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files
2016-08-25 09:01:45,140 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.95/28.86 kb and 22/32 files
2016-08-25 09:01:45,140 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.85/28.86 kb and 23/32 files
2016-08-25 09:01:45,141 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.75/28.86 kb and 24/32 files
2016-08-25 09:01:45,141 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.65/28.86 kb and 25/32 files
2016-08-25 09:01:45,142 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.55/28.86 kb and 26/32 files
2016-08-25 09:01:45,142 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.45/28.86 kb and 27/32 files
2016-08-25 09:01:45,143 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.35/28.86 kb and 28/32 files
2016-08-25 09:01:45,144 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.25/28.86 kb and 29/32 files
2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.77/28.86 kb and 30/32 files
2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files
2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files
2016-08-25 09:01:45,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.67/28.86 kb and 31/32 files
2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files
2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files
2016-08-25 09:01:45,146 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:45,147 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files
2016-08-25 09:01:45,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files
2016-08-25 09:01:45,148 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files
2016-08-25 09:01:45,149 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files
2016-08-25 09:01:45,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files
2016-08-25 09:01:45,150 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/28.86 kb and 9/32 files
2016-08-25 09:01:45,151 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/28.86 kb and 10/32 files
2016-08-25 09:01:45,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.90/28.86 kb and 11/32 files
2016-08-25 09:01:45,152 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/28.86 kb and 12/32 files
2016-08-25 09:01:45,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.74/28.86 kb and 13/32 files
2016-08-25 09:01:45,153 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/28.86 kb and 14/32 files
2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/28.86 kb and 15/32 files
2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:45,154 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/28.86 kb and 16/32 files
2016-08-25 09:01:45,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files
2016-08-25 09:01:45,155 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is starting
2016-08-25 09:01:45,155 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.36/28.86 kb and 18/32 files
2016-08-25 09:01:45,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.88/28.86 kb and 19/32 files
2016-08-25 09:01:45,156 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.78/28.86 kb and 20/32 files
2016-08-25 09:01:45,157 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.68/28.86 kb and 21/32 files
2016-08-25 09:01:45,158 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.58/28.86 kb and 22/32 files
2016-08-25 09:01:45,158 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.48/28.86 kb and 23/32 files
2016-08-25 09:01:45,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.66/28.86 kb and 24/32 files
2016-08-25 09:01:45,159 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55387
2016-08-25 09:01:45,159 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.56/28.86 kb and 25/32 files
2016-08-25 09:01:45,159 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:45,160 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) started
2016-08-25 09:01:45,160 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:45,160 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:45,160 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:45,160 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files
2016-08-25 09:01:45,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files
2016-08-25 09:01:45,161 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files
Master started: node-1
======================================
57.4. Browse the queue on new master
======================================
2016-08-25 09:01:45,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files
2016-08-25 09:01:45,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files
2016-08-25 09:01:45,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files
2016-08-25 09:01:45,164 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files
2016-08-25 09:01:45,164 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:45,171 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
57.5. Found
======================================
======================================
58.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:45,179 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is shutting down
2016-08-25 09:01:45,180 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:45,182 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:45,182 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:45,184 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:45,185 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003b
2016-08-25 09:01:45,185 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:45,185 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55374 which had sessionid 0x156bec91725003b
2016-08-25 09:01:45,185 | INFO | Time-limited test | Session: 0x156bec91725003b closed
2016-08-25 09:01:45,186 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:45,189 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:45,189 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:45,189 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:45,189 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:45,304 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:45,305 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) uptime 1.884 seconds
2016-08-25 09:01:45,305 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:59) is shutdown
Wait for master to start up...
2016-08-25 09:01:45,307 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:45,309 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@dabaed5
2016-08-25 09:01:45,310 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:45,310 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:45,311 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:45,311 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55389
2016-08-25 09:01:45,311 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55389
2016-08-25 09:01:45,311 | INFO | SyncThread:0 | Established session 0x156bec91725003d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55389
2016-08-25 09:01:45,312 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003d, negotiated timeout = 10000
2016-08-25 09:01:45,321 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:45,634 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:45,634 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:45,636 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:45,644 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:45,649 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:45,653 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:45,898 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55390
2016-08-25 09:01:45,903 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:45,904 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:45,906 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55390
2016-08-25 09:01:45,906 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:45,907 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55390
2016-08-25 09:01:45,907 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:45,917 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:46,121 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:46,122 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000090.log
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000088
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:46,123 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:46,124 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:46,126 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:46,127 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:46,128 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000090.log
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000088
2016-08-25 09:01:46,129 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:46,130 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:46,131 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:46,144 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files
2016-08-25 09:01:46,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files
2016-08-25 09:01:46,145 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files
2016-08-25 09:01:46,146 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files
2016-08-25 09:01:46,147 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.55/28.86 kb and 5/32 files
2016-08-25 09:01:46,147 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.45/28.86 kb and 6/32 files
2016-08-25 09:01:46,148 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.35/28.86 kb and 7/32 files
2016-08-25 09:01:46,148 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.25/28.86 kb and 8/32 files
2016-08-25 09:01:46,149 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.14/28.86 kb and 9/32 files
2016-08-25 09:01:46,149 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.05/28.86 kb and 10/32 files
2016-08-25 09:01:46,150 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.95/28.86 kb and 11/32 files
2016-08-25 09:01:46,151 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.85/28.86 kb and 12/32 files
2016-08-25 09:01:46,151 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.75/28.86 kb and 13/32 files
2016-08-25 09:01:46,152 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.65/28.86 kb and 14/32 files
2016-08-25 09:01:46,152 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.66/28.86 kb and 15/32 files
2016-08-25 09:01:46,153 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.56/28.86 kb and 16/32 files
2016-08-25 09:01:46,153 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.46/28.86 kb and 17/32 files
2016-08-25 09:01:46,154 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.35/28.86 kb and 18/32 files
2016-08-25 09:01:46,155 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.25/28.86 kb and 19/32 files
2016-08-25 09:01:46,155 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.15/28.86 kb and 20/32 files
2016-08-25 09:01:46,156 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files
2016-08-25 09:01:46,156 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.95/28.86 kb and 22/32 files
2016-08-25 09:01:46,157 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.13/28.86 kb and 23/32 files
2016-08-25 09:01:46,157 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.03/28.86 kb and 24/32 files
2016-08-25 09:01:46,158 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.56/28.86 kb and 25/32 files
2016-08-25 09:01:46,158 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files
2016-08-25 09:01:46,159 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files
2016-08-25 09:01:46,160 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files
2016-08-25 09:01:46,160 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files
2016-08-25 09:01:46,161 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files
2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files
2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files
2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/28.86 kb and 1/32 files
2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:46,162 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/28.86 kb and 2/32 files
2016-08-25 09:01:46,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/28.86 kb and 3/32 files
2016-08-25 09:01:46,163 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/28.86 kb and 4/32 files
2016-08-25 09:01:46,164 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/28.86 kb and 5/32 files
2016-08-25 09:01:46,164 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:46,165 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/28.86 kb and 6/32 files
2016-08-25 09:01:46,165 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/28.86 kb and 7/32 files
2016-08-25 09:01:46,165 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is starting
2016-08-25 09:01:46,166 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/28.86 kb and 8/32 files
2016-08-25 09:01:46,166 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/28.86 kb and 9/32 files
2016-08-25 09:01:46,167 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.05/28.86 kb and 10/32 files
2016-08-25 09:01:46,168 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.95/28.86 kb and 11/32 files
2016-08-25 09:01:46,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.85/28.86 kb and 12/32 files
2016-08-25 09:01:46,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.75/28.86 kb and 13/32 files
2016-08-25 09:01:46,169 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55395
2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) started
2016-08-25 09:01:46,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/28.86 kb and 14/32 files
2016-08-25 09:01:46,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/28.86 kb and 15/32 files
2016-08-25 09:01:46,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/28.86 kb and 16/32 files
2016-08-25 09:01:46,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.34/28.86 kb and 17/32 files
2016-08-25 09:01:46,170 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:46,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.24/28.86 kb and 18/32 files
2016-08-25 09:01:46,173 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:46,173 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:46,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.14/28.86 kb and 19/32 files
2016-08-25 09:01:46,174 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.15/28.86 kb and 20/32 files
2016-08-25 09:01:46,174 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.05/28.86 kb and 21/32 files
Master started: node-2
======================================
58.4. Browse the queue on new master
======================================
2016-08-25 09:01:46,175 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.24/28.86 kb and 22/32 files
2016-08-25 09:01:46,176 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.14/28.86 kb and 23/32 files
2016-08-25 09:01:46,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.04/28.86 kb and 24/32 files
2016-08-25 09:01:46,177 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.94/28.86 kb and 25/32 files
2016-08-25 09:01:46,178 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.46/28.86 kb and 26/32 files
2016-08-25 09:01:46,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.36/28.86 kb and 27/32 files
2016-08-25 09:01:46,179 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.26/28.86 kb and 28/32 files
2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.16/28.86 kb and 29/32 files
2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.06/28.86 kb and 30/32 files
2016-08-25 09:01:46,180 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.96/28.86 kb and 31/32 files
2016-08-25 09:01:46,181 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.86/28.86 kb and 32/32 files
2016-08-25 09:01:46,181 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:46,187 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
58.5. Found
======================================
======================================
59.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:46,194 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is shutting down
2016-08-25 09:01:46,194 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:46,197 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:46,198 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:46,199 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:46,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:46,199 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:46,199 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:46,200 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003a
2016-08-25 09:01:46,201 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55366 which had sessionid 0x156bec91725003a
2016-08-25 09:01:46,202 | INFO | Time-limited test | Session: 0x156bec91725003a closed
2016-08-25 09:01:46,202 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:46,307 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:46,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) uptime 4.008 seconds
2016-08-25 09:01:46,307 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:60) is shutdown
Wait for master to start up...
2016-08-25 09:01:46,310 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:46,312 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5f238a57
2016-08-25 09:01:46,312 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:46,313 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:46,313 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:46,313 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55397
2016-08-25 09:01:46,314 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55397
2016-08-25 09:01:46,314 | INFO | SyncThread:0 | Established session 0x156bec91725003e with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55397
2016-08-25 09:01:46,315 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003e, negotiated timeout = 10000
2016-08-25 09:01:46,322 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:46,901 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:46,904 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:46,914 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:46,914 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:46,917 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:46,920 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:47,179 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55398
2016-08-25 09:01:47,186 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:47,186 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:47,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55398
2016-08-25 09:01:47,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:47,190 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55398
2016-08-25 09:01:47,191 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:47,418 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:47,426 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:47,427 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000091
2016-08-25 09:01:47,428 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:47,429 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000093.log
2016-08-25 09:01:47,430 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:47,431 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000091
2016-08-25 09:01:47,432 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000093.log
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:47,433 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:47,447 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files
2016-08-25 09:01:47,448 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files
2016-08-25 09:01:47,448 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files
2016-08-25 09:01:47,449 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files
2016-08-25 09:01:47,449 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files
2016-08-25 09:01:47,450 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files
2016-08-25 09:01:47,451 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/29.81 kb and 7/33 files
2016-08-25 09:01:47,451 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/29.81 kb and 8/33 files
2016-08-25 09:01:47,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.10/29.81 kb and 9/33 files
2016-08-25 09:01:47,452 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.00/29.81 kb and 10/33 files
2016-08-25 09:01:47,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.90/29.81 kb and 11/33 files
2016-08-25 09:01:47,453 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.80/29.81 kb and 12/33 files
2016-08-25 09:01:47,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.70/29.81 kb and 13/33 files
2016-08-25 09:01:47,454 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.60/29.81 kb and 14/33 files
2016-08-25 09:01:47,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.49/29.81 kb and 15/33 files
2016-08-25 09:01:47,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.39/29.81 kb and 16/33 files
2016-08-25 09:01:47,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.29/29.81 kb and 17/33 files
2016-08-25 09:01:47,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.24/29.81 kb and 18/33 files
2016-08-25 09:01:47,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.14/29.81 kb and 19/33 files
2016-08-25 09:01:47,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.04/29.81 kb and 20/33 files
2016-08-25 09:01:47,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.62/29.81 kb and 21/33 files
2016-08-25 09:01:47,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.52/29.81 kb and 22/33 files
2016-08-25 09:01:47,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.42/29.81 kb and 23/33 files
2016-08-25 09:01:47,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.31/29.81 kb and 24/33 files
2016-08-25 09:01:47,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.21/29.81 kb and 25/33 files
2016-08-25 09:01:47,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.11/29.81 kb and 26/33 files
2016-08-25 09:01:47,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.13/29.81 kb and 27/33 files
2016-08-25 09:01:47,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.03/29.81 kb and 28/33 files
2016-08-25 09:01:47,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.93/29.81 kb and 29/33 files
2016-08-25 09:01:47,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.83/29.81 kb and 30/33 files
2016-08-25 09:01:47,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.73/29.81 kb and 31/33 files
2016-08-25 09:01:47,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.62/29.81 kb and 32/33 files
2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.95/29.81 kb and 1/33 files
2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files
2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:47,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/29.81 kb and 2/33 files
2016-08-25 09:01:47,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.75/29.81 kb and 3/33 files
2016-08-25 09:01:47,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/29.81 kb and 4/33 files
2016-08-25 09:01:47,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/29.81 kb and 5/33 files
2016-08-25 09:01:47,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/29.81 kb and 6/33 files
2016-08-25 09:01:47,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.34/29.81 kb and 7/33 files
2016-08-25 09:01:47,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.24/29.81 kb and 8/33 files
2016-08-25 09:01:47,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.14/29.81 kb and 9/33 files
2016-08-25 09:01:47,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.04/29.81 kb and 10/33 files
2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.94/29.81 kb and 11/33 files
2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:47,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.83/29.81 kb and 12/33 files
2016-08-25 09:01:47,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.73/29.81 kb and 13/33 files
2016-08-25 09:01:47,471 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is starting
2016-08-25 09:01:47,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.64/29.81 kb and 14/33 files
2016-08-25 09:01:47,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.54/29.81 kb and 15/33 files
2016-08-25 09:01:47,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.44/29.81 kb and 16/33 files
2016-08-25 09:01:47,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.34/29.81 kb and 17/33 files
2016-08-25 09:01:47,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.24/29.81 kb and 18/33 files
2016-08-25 09:01:47,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.14/29.81 kb and 19/33 files
2016-08-25 09:01:47,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.04/29.81 kb and 20/33 files
2016-08-25 09:01:47,476 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55403
2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:47,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.93/29.81 kb and 21/33 files
2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) started
2016-08-25 09:01:47,477 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:47,477 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:47,477 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.83/29.81 kb and 22/33 files
2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.41/29.81 kb and 23/33 files
2016-08-25 09:01:47,478 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.31/29.81 kb and 24/33 files
Master started: node-3
======================================
59.4. Browse the queue on new master
======================================
2016-08-25 09:01:47,479 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.21/29.81 kb and 25/33 files
2016-08-25 09:01:47,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.11/29.81 kb and 26/33 files
2016-08-25 09:01:47,480 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.01/29.81 kb and 27/33 files
2016-08-25 09:01:47,481 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.91/29.81 kb and 28/33 files
2016-08-25 09:01:47,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.81/29.81 kb and 29/33 files
2016-08-25 09:01:47,482 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.99/29.81 kb and 30/33 files
2016-08-25 09:01:47,483 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.89/29.81 kb and 31/33 files
2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.79/29.81 kb and 32/33 files
2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files
2016-08-25 09:01:47,484 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:47,492 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
59.5. Found
======================================
======================================
60.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:47,497 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is shutting down
2016-08-25 09:01:47,498 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:47,501 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:47,501 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:47,502 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003c
2016-08-25 09:01:47,502 | INFO | Time-limited test | Session: 0x156bec91725003c closed
2016-08-25 09:01:47,502 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:47,502 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55381 which had sessionid 0x156bec91725003c
2016-08-25 09:01:47,505 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:47,505 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:47,505 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:47,506 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:47,622 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:47,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) uptime 3.484 seconds
2016-08-25 09:01:47,623 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:61) is shutdown
Wait for master to start up...
2016-08-25 09:01:47,625 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:47,628 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@369fb2ed
2016-08-25 09:01:47,628 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:47,629 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:47,630 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:47,630 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55405
2016-08-25 09:01:47,630 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55405
2016-08-25 09:01:47,631 | INFO | SyncThread:0 | Established session 0x156bec91725003f with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55405
2016-08-25 09:01:47,631 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725003f, negotiated timeout = 10000
2016-08-25 09:01:47,638 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:47,947 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:47,950 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:47,969 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:47,970 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:47,973 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:47,976 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:48,204 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55406
2016-08-25 09:01:48,209 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:48,209 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:48,210 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55406
2016-08-25 09:01:48,210 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:48,213 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55406
2016-08-25 09:01:48,213 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:48,223 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:48,384 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:48,384 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:48,385 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000093.log
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:48,386 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000091
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:48,387 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:48,388 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000091
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:48,389 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:48,390 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000093.log
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:48,391 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:48,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files
2016-08-25 09:01:48,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files
2016-08-25 09:01:48,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files
2016-08-25 09:01:48,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files
2016-08-25 09:01:48,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files
2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files
2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.29/29.81 kb and 7/33 files
2016-08-25 09:01:48,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.19/29.81 kb and 8/33 files
2016-08-25 09:01:48,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/29.81 kb and 9/33 files
2016-08-25 09:01:48,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/29.81 kb and 10/33 files
2016-08-25 09:01:48,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/29.81 kb and 11/33 files
2016-08-25 09:01:48,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/29.81 kb and 12/33 files
2016-08-25 09:01:48,407 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/29.81 kb and 13/33 files
2016-08-25 09:01:48,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/29.81 kb and 14/33 files
2016-08-25 09:01:48,408 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/29.81 kb and 15/33 files
2016-08-25 09:01:48,409 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.79/29.81 kb and 16/33 files
2016-08-25 09:01:48,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.69/29.81 kb and 17/33 files
2016-08-25 09:01:48,410 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/29.81 kb and 18/33 files
2016-08-25 09:01:48,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.54/29.81 kb and 19/33 files
2016-08-25 09:01:48,411 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.44/29.81 kb and 20/33 files
2016-08-25 09:01:48,412 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.02/29.81 kb and 21/33 files
2016-08-25 09:01:48,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.92/29.81 kb and 22/33 files
2016-08-25 09:01:48,413 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.82/29.81 kb and 23/33 files
2016-08-25 09:01:48,414 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.71/29.81 kb and 24/33 files
2016-08-25 09:01:48,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.61/29.81 kb and 25/33 files
2016-08-25 09:01:48,415 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.51/29.81 kb and 26/33 files
2016-08-25 09:01:48,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.41/29.81 kb and 27/33 files
2016-08-25 09:01:48,416 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.31/29.81 kb and 28/33 files
2016-08-25 09:01:48,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.21/29.81 kb and 29/33 files
2016-08-25 09:01:48,417 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.11/29.81 kb and 30/33 files
2016-08-25 09:01:48,418 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.01/29.81 kb and 31/33 files
2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.91/29.81 kb and 32/33 files
2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files
2016-08-25 09:01:48,419 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:48,422 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:48,424 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is starting
2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55411
2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) started
2016-08-25 09:01:48,428 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:48,428 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:48,428 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:48,429 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/29.81 kb and 1/33 files
2016-08-25 09:01:48,434 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/29.81 kb and 2/33 files
2016-08-25 09:01:48,434 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/29.81 kb and 3/33 files
Master started: node-1
======================================
60.4. Browse the queue on new master
======================================
2016-08-25 09:01:48,435 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/29.81 kb and 4/33 files
2016-08-25 09:01:48,436 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/29.81 kb and 5/33 files
2016-08-25 09:01:48,436 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/29.81 kb and 6/33 files
2016-08-25 09:01:48,437 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.97/29.81 kb and 7/33 files
2016-08-25 09:01:48,437 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.87/29.81 kb and 8/33 files
2016-08-25 09:01:48,438 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.77/29.81 kb and 9/33 files
2016-08-25 09:01:48,439 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.67/29.81 kb and 10/33 files
2016-08-25 09:01:48,440 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.57/29.81 kb and 11/33 files
2016-08-25 09:01:48,441 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.47/29.81 kb and 12/33 files
2016-08-25 09:01:48,442 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.37/29.81 kb and 13/33 files
2016-08-25 09:01:48,442 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.27/29.81 kb and 14/33 files
2016-08-25 09:01:48,443 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.17/29.81 kb and 15/33 files
2016-08-25 09:01:48,444 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.18/29.81 kb and 16/33 files
2016-08-25 09:01:48,445 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.08/29.81 kb and 17/33 files
2016-08-25 09:01:48,445 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.98/29.81 kb and 18/33 files
2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.88/29.81 kb and 19/33 files
2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.78/29.81 kb and 20/33 files
2016-08-25 09:01:48,446 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.68/29.81 kb and 21/33 files
2016-08-25 09:01:48,447 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.58/29.81 kb and 22/33 files
2016-08-25 09:01:48,448 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.48/29.81 kb and 23/33 files
2016-08-25 09:01:48,448 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.38/29.81 kb and 24/33 files
2016-08-25 09:01:48,449 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.28/29.81 kb and 25/33 files
2016-08-25 09:01:48,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.18/29.81 kb and 26/33 files
2016-08-25 09:01:48,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.13/29.81 kb and 27/33 files
2016-08-25 09:01:48,451 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.31/29.81 kb and 28/33 files
2016-08-25 09:01:48,452 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.21/29.81 kb and 29/33 files
2016-08-25 09:01:48,453 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.11/29.81 kb and 30/33 files
2016-08-25 09:01:48,454 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.01/29.81 kb and 31/33 files
2016-08-25 09:01:48,455 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.91/29.81 kb and 32/33 files
======================================
2016-08-25 09:01:48,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.81/29.81 kb and 33/33 files
60.5. Found
======================================
======================================
61.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:48,456 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:48,456 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is shutting down
2016-08-25 09:01:48,457 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:48,461 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:48,461 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:48,462 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:48,463 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:48,463 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003d
2016-08-25 09:01:48,464 | INFO | Time-limited test | Session: 0x156bec91725003d closed
2016-08-25 09:01:48,464 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55389 which had sessionid 0x156bec91725003d
2016-08-25 09:01:48,464 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:48,468 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:48,469 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:48,568 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:48,569 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) uptime 3.264 seconds
2016-08-25 09:01:48,569 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:62) is shutdown
Wait for master to start up...
2016-08-25 09:01:48,572 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:48,577 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3ec6fd33
2016-08-25 09:01:48,577 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:48,578 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:48,579 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:48,579 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55413
2016-08-25 09:01:48,579 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55413
2016-08-25 09:01:48,580 | INFO | SyncThread:0 | Established session 0x156bec917250040 with negotiated timeout 10000 for client /127.0.0.1:55413
2016-08-25 09:01:48,580 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250040, negotiated timeout = 10000
2016-08-25 09:01:48,588 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,116 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:49,118 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:49,126 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,129 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:49,131 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:49,341 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55414
2016-08-25 09:01:49,347 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:49,348 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:49,349 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55414
2016-08-25 09:01:49,349 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:49,351 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55414
2016-08-25 09:01:49,351 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:49,582 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000096.log
2016-08-25 09:01:49,586 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:49,587 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000094
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:49,588 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:49,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:49,589 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:49,592 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:49,606 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files
2016-08-25 09:01:49,607 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.09/30.76 kb and 2/34 files
2016-08-25 09:01:49,607 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.99/30.76 kb and 3/34 files
2016-08-25 09:01:49,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.89/30.76 kb and 4/34 files
2016-08-25 09:01:49,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.79/30.76 kb and 5/34 files
2016-08-25 09:01:49,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.80/30.76 kb and 6/34 files
2016-08-25 09:01:49,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.70/30.76 kb and 7/34 files
2016-08-25 09:01:49,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.60/30.76 kb and 8/34 files
2016-08-25 09:01:49,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.50/30.76 kb and 9/34 files
2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.40/30.76 kb and 10/34 files
2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.30/30.76 kb and 11/34 files
2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.20/30.76 kb and 12/34 files
2016-08-25 09:01:49,611 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.10/30.76 kb and 13/34 files
2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.00/30.76 kb and 14/34 files
2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.90/30.76 kb and 15/34 files
2016-08-25 09:01:49,612 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.80/30.76 kb and 16/34 files
2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.69/30.76 kb and 17/34 files
2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.59/30.76 kb and 18/34 files
2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.49/30.76 kb and 19/34 files
2016-08-25 09:01:49,613 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.39/30.76 kb and 20/34 files
2016-08-25 09:01:49,614 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.29/30.76 kb and 21/34 files
2016-08-25 09:01:49,614 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.19/30.76 kb and 22/34 files
2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.09/30.76 kb and 23/34 files
2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.99/30.76 kb and 24/34 files
2016-08-25 09:01:49,615 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.62/30.76 kb and 25/34 files
2016-08-25 09:01:49,616 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.57/30.76 kb and 26/34 files
2016-08-25 09:01:49,616 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.47/30.76 kb and 27/34 files
2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 25.37/30.76 kb and 28/34 files
2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 26.27/30.76 kb and 29/34 files
2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 27.17/30.76 kb and 30/34 files
2016-08-25 09:01:49,617 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.06/30.76 kb and 31/34 files
2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.96/30.76 kb and 32/34 files
2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.86/30.76 kb and 33/34 files
2016-08-25 09:01:49,618 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files
2016-08-25 09:01:49,619 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:49,622 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:49,623 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:49,623 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is starting
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:49,624 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000094
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000096.log
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:49,625 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:49,626 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:49,627 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55419
2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) started
2016-08-25 09:01:49,628 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:49,628 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:49,628 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-2
======================================
61.4. Browse the queue on new master
======================================
======================================
61.5. Found
======================================
======================================
62.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:49,642 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is shutting down
2016-08-25 09:01:49,642 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:49,644 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,645 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:49,645 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003e
2016-08-25 09:01:49,645 | INFO | Time-limited test | Session: 0x156bec91725003e closed
2016-08-25 09:01:49,645 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:49,646 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55397 which had sessionid 0x156bec91725003e
2016-08-25 09:01:49,648 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:49,648 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:49,766 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:49,766 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) uptime 3.458 seconds
2016-08-25 09:01:49,766 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:63) is shutdown
Wait for master to start up...
2016-08-25 09:01:49,768 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:49,771 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@39584da9
2016-08-25 09:01:49,771 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:49,771 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:49,772 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:49,772 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55421
2016-08-25 09:01:49,772 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55421
2016-08-25 09:01:49,773 | INFO | SyncThread:0 | Established session 0x156bec917250041 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55421
2016-08-25 09:01:49,773 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250041, negotiated timeout = 10000
2016-08-25 09:01:49,778 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,889 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:49,889 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:49,889 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:49,889 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:49,891 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:50,122 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:50,125 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:50,130 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:50,383 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55422
2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55422
2016-08-25 09:01:50,388 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:50,389 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55422
2016-08-25 09:01:50,389 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:50,409 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:50,613 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000096.log
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:50,614 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000094
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:50,615 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:50,616 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:50,617 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000096.log
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/MANIFEST-000094
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:50,618 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:50,619 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:50,620 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:50,620 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:50,643 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files
2016-08-25 09:01:50,644 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/30.76 kb and 2/34 files
2016-08-25 09:01:50,644 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/30.76 kb and 1/34 files
2016-08-25 09:01:50,645 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/30.76 kb and 2/34 files
2016-08-25 09:01:50,645 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/30.76 kb and 3/34 files
2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/30.76 kb and 3/34 files
2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/30.76 kb and 4/34 files
2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.64/30.76 kb and 4/34 files
2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/30.76 kb and 5/34 files
2016-08-25 09:01:50,646 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.54/30.76 kb and 5/34 files
2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/30.76 kb and 6/34 files
2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.44/30.76 kb and 6/34 files
2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/30.76 kb and 7/34 files
2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.63/30.76 kb and 7/34 files
2016-08-25 09:01:50,647 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/30.76 kb and 8/34 files
2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.53/30.76 kb and 8/34 files
2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/30.76 kb and 9/34 files
2016-08-25 09:01:50,648 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.54/30.76 kb and 9/34 files
2016-08-25 09:01:50,649 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.44/30.76 kb and 10/34 files
2016-08-25 09:01:50,649 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/30.76 kb and 10/34 files
2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.34/30.76 kb and 11/34 files
2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/30.76 kb and 11/34 files
2016-08-25 09:01:50,650 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.24/30.76 kb and 12/34 files
2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/30.76 kb and 12/34 files
2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.87/30.76 kb and 13/34 files
2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.98/30.76 kb and 13/34 files
2016-08-25 09:01:50,651 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.77/30.76 kb and 14/34 files
2016-08-25 09:01:50,652 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.67/30.76 kb and 15/34 files
2016-08-25 09:01:50,652 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.88/30.76 kb and 14/34 files
2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.57/30.76 kb and 16/34 files
2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.78/30.76 kb and 15/34 files
2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.68/30.76 kb and 16/34 files
2016-08-25 09:01:50,653 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.47/30.76 kb and 17/34 files
2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.37/30.76 kb and 18/34 files
2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.58/30.76 kb and 17/34 files
2016-08-25 09:01:50,654 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.27/30.76 kb and 19/34 files
2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.48/30.76 kb and 18/34 files
2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.16/30.76 kb and 20/34 files
2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.37/30.76 kb and 19/34 files
2016-08-25 09:01:50,655 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.06/30.76 kb and 21/34 files
2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.27/30.76 kb and 20/34 files
2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.96/30.76 kb and 22/34 files
2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.17/30.76 kb and 21/34 files
2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.07/30.76 kb and 22/34 files
2016-08-25 09:01:50,656 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.86/30.76 kb and 23/34 files
2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.97/30.76 kb and 23/34 files
2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.76/30.76 kb and 24/34 files
2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.66/30.76 kb and 25/34 files
2016-08-25 09:01:50,657 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.87/30.76 kb and 24/34 files
2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.56/30.76 kb and 26/34 files
2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.77/30.76 kb and 25/34 files
2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.46/30.76 kb and 27/34 files
2016-08-25 09:01:50,658 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.36/30.76 kb and 28/34 files
2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.40/30.76 kb and 26/34 files
2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.26/30.76 kb and 29/34 files
2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.30/30.76 kb and 27/34 files
2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.16/30.76 kb and 30/34 files
2016-08-25 09:01:50,659 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.20/30.76 kb and 28/34 files
2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.06/30.76 kb and 31/34 files
2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.96/30.76 kb and 32/34 files
2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.21/30.76 kb and 29/34 files
2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.86/30.76 kb and 33/34 files
2016-08-25 09:01:50,660 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.11/30.76 kb and 30/34 files
2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files
2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:50,661 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.01/30.76 kb and 31/34 files
2016-08-25 09:01:50,662 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.91/30.76 kb and 32/34 files
2016-08-25 09:01:50,662 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.81/30.76 kb and 33/34 files
2016-08-25 09:01:50,663 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.76/30.76 kb and 34/34 files
2016-08-25 09:01:50,664 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:50,667 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:50,667 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:50,668 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is starting
2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55427
2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:50,671 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) started
2016-08-25 09:01:50,672 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:50,672 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:50,672 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-1
======================================
62.4. Browse the queue on new master
======================================
======================================
62.5. Found
======================================
======================================
63.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:50,686 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is shutting down
2016-08-25 09:01:50,687 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:50,688 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:50,689 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:01:50,689 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:50,689 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250040
2016-08-25 09:01:50,689 | INFO | Time-limited test | Session: 0x156bec917250040 closed
2016-08-25 09:01:50,689 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:50,689 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55413 which had sessionid 0x156bec917250040
2016-08-25 09:01:50,691 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:50,694 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:50,694 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:50,695 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:50,695 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:50,793 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:50,793 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) uptime 2.224 seconds
2016-08-25 09:01:50,794 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:64) is shutdown
Wait for master to start up...
2016-08-25 09:01:50,795 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:50,798 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4736c6df
2016-08-25 09:01:50,798 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:50,799 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:50,799 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:50,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55429
2016-08-25 09:01:50,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55429
2016-08-25 09:01:50,800 | INFO | SyncThread:0 | Established session 0x156bec917250042 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55429
2016-08-25 09:01:50,800 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250042, negotiated timeout = 10000
2016-08-25 09:01:50,806 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:51,375 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:51,376 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:51,378 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:51,379 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:51,383 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:51,386 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:51,626 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55430
2016-08-25 09:01:51,631 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:51,632 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55430
2016-08-25 09:01:51,632 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:51,633 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:51,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55430
2016-08-25 09:01:51,638 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:51,870 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:51,882 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:51,882 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000099.log
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:51,883 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:51,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:51,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:51,886 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000097
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000099.log
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:51,887 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:51,888 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:51,889 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000097
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:51,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:51,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:51,910 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/31.71 kb and 1/35 files
2016-08-25 09:01:51,911 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/31.71 kb and 2/35 files
2016-08-25 09:01:51,911 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/31.71 kb and 3/35 files
2016-08-25 09:01:51,912 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/31.71 kb and 4/35 files
2016-08-25 09:01:51,912 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/31.71 kb and 5/35 files
2016-08-25 09:01:51,913 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.68/31.71 kb and 6/35 files
2016-08-25 09:01:51,913 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/31.71 kb and 7/35 files
2016-08-25 09:01:51,914 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/31.71 kb and 8/35 files
2016-08-25 09:01:51,915 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/31.71 kb and 9/35 files
2016-08-25 09:01:51,915 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/31.71 kb and 10/35 files
2016-08-25 09:01:51,916 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.18/31.71 kb and 11/35 files
2016-08-25 09:01:51,916 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.08/31.71 kb and 12/35 files
2016-08-25 09:01:51,917 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.97/31.71 kb and 13/35 files
2016-08-25 09:01:51,917 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.87/31.71 kb and 14/35 files
2016-08-25 09:01:51,918 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.77/31.71 kb and 15/35 files
2016-08-25 09:01:51,918 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.67/31.71 kb and 16/35 files
2016-08-25 09:01:51,919 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.57/31.71 kb and 17/35 files
2016-08-25 09:01:51,919 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.52/31.71 kb and 18/35 files
2016-08-25 09:01:51,920 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.42/31.71 kb and 19/35 files
2016-08-25 09:01:51,920 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.32/31.71 kb and 20/35 files
2016-08-25 09:01:51,921 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.22/31.71 kb and 21/35 files
2016-08-25 09:01:51,921 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.12/31.71 kb and 22/35 files
2016-08-25 09:01:51,922 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.01/31.71 kb and 23/35 files
2016-08-25 09:01:51,922 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.91/31.71 kb and 24/35 files
2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.81/31.71 kb and 25/35 files
2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.71/31.71 kb and 26/35 files
2016-08-25 09:01:51,923 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.61/31.71 kb and 27/35 files
2016-08-25 09:01:51,924 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.51/31.71 kb and 28/35 files
2016-08-25 09:01:51,924 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.53/31.71 kb and 29/35 files
2016-08-25 09:01:51,925 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.43/31.71 kb and 30/35 files
2016-08-25 09:01:51,926 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.33/31.71 kb and 31/35 files
2016-08-25 09:01:51,926 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.23/31.71 kb and 32/35 files
2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.13/31.71 kb and 33/35 files
2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/31.71 kb and 1/35 files
2016-08-25 09:01:51,927 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.03/31.71 kb and 34/35 files
2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files
2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/31.71 kb and 2/35 files
2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:51,928 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.10/31.71 kb and 3/35 files
2016-08-25 09:01:51,929 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/31.71 kb and 4/35 files
2016-08-25 09:01:51,929 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.90/31.71 kb and 5/35 files
2016-08-25 09:01:51,930 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.80/31.71 kb and 6/35 files
2016-08-25 09:01:51,930 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.70/31.71 kb and 7/35 files
2016-08-25 09:01:51,931 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.60/31.71 kb and 8/35 files
2016-08-25 09:01:51,931 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.50/31.71 kb and 9/35 files
2016-08-25 09:01:51,932 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.40/31.71 kb and 10/35 files
2016-08-25 09:01:51,932 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.30/31.71 kb and 11/35 files
2016-08-25 09:01:51,933 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.20/31.71 kb and 12/35 files
2016-08-25 09:01:51,934 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.10/31.71 kb and 13/35 files
2016-08-25 09:01:51,934 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.00/31.71 kb and 14/35 files
2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.90/31.71 kb and 15/35 files
2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:51,935 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.80/31.71 kb and 16/35 files
2016-08-25 09:01:51,936 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.70/31.71 kb and 17/35 files
2016-08-25 09:01:51,936 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is starting
2016-08-25 09:01:51,937 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.59/31.71 kb and 18/35 files
2016-08-25 09:01:51,937 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.49/31.71 kb and 19/35 files
2016-08-25 09:01:51,938 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.39/31.71 kb and 20/35 files
2016-08-25 09:01:51,939 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.29/31.71 kb and 21/35 files
2016-08-25 09:01:51,939 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.19/31.71 kb and 22/35 files
2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55435
2016-08-25 09:01:51,940 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.09/31.71 kb and 23/35 files
2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) started
2016-08-25 09:01:51,940 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:51,940 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:51,940 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:51,941 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.77/31.71 kb and 24/35 files
2016-08-25 09:01:51,941 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.67/31.71 kb and 25/35 files
Master started: node-3
======================================
63.4. Browse the queue on new master
======================================
2016-08-25 09:01:51,942 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.56/31.71 kb and 26/35 files
2016-08-25 09:01:51,943 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.46/31.71 kb and 27/35 files
2016-08-25 09:01:51,943 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.36/31.71 kb and 28/35 files
2016-08-25 09:01:51,944 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.26/31.71 kb and 29/35 files
2016-08-25 09:01:51,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.16/31.71 kb and 30/35 files
2016-08-25 09:01:51,945 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.06/31.71 kb and 31/35 files
2016-08-25 09:01:51,946 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.96/31.71 kb and 32/35 files
2016-08-25 09:01:51,947 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.86/31.71 kb and 33/35 files
2016-08-25 09:01:51,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files
2016-08-25 09:01:51,948 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files
2016-08-25 09:01:51,949 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
63.5. Found
======================================
======================================
64.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:51,956 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is shutting down
2016-08-25 09:01:51,956 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:51,958 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:51,959 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:51,960 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:51,960 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725003f
2016-08-25 09:01:51,960 | INFO | Time-limited test | Session: 0x156bec91725003f closed
2016-08-25 09:01:51,961 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:51,961 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55405 which had sessionid 0x156bec91725003f
2016-08-25 09:01:51,965 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:51,965 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:51,965 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:51,965 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:52,079 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:52,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) uptime 4.455 seconds
2016-08-25 09:01:52,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:65) is shutdown
Wait for master to start up...
2016-08-25 09:01:52,081 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:52,083 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@62247009
2016-08-25 09:01:52,083 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:52,084 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:52,085 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:52,085 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55437
2016-08-25 09:01:52,085 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55437
2016-08-25 09:01:52,086 | INFO | SyncThread:0 | Established session 0x156bec917250043 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55437
2016-08-25 09:01:52,086 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250043, negotiated timeout = 10000
2016-08-25 09:01:52,091 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:52,484 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:52,486 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:52,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:52,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:52,489 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:52,492 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:52,753 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55438
2016-08-25 09:01:52,758 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:52,758 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:52,759 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55438
2016-08-25 09:01:52,760 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:52,762 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55438
2016-08-25 09:01:52,762 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:52,772 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:52,975 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000097
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:52,976 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:52,977 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000099.log
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:52,978 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:52,979 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000097
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000099.log
2016-08-25 09:01:52,980 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:52,981 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:52,982 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:52,997 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/31.71 kb and 1/35 files
2016-08-25 09:01:52,997 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/31.71 kb and 2/35 files
2016-08-25 09:01:52,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/31.71 kb and 3/35 files
2016-08-25 09:01:52,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/31.71 kb and 4/35 files
2016-08-25 09:01:52,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/31.71 kb and 5/35 files
2016-08-25 09:01:52,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/31.71 kb and 6/35 files
2016-08-25 09:01:53,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.41/31.71 kb and 7/35 files
2016-08-25 09:01:53,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.09/31.71 kb and 8/35 files
2016-08-25 09:01:53,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.99/31.71 kb and 9/35 files
2016-08-25 09:01:53,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.89/31.71 kb and 10/35 files
2016-08-25 09:01:53,002 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.79/31.71 kb and 11/35 files
2016-08-25 09:01:53,002 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.69/31.71 kb and 12/35 files
2016-08-25 09:01:53,003 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.59/31.71 kb and 13/35 files
2016-08-25 09:01:53,004 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.49/31.71 kb and 14/35 files
2016-08-25 09:01:53,005 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.39/31.71 kb and 15/35 files
2016-08-25 09:01:53,005 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.29/31.71 kb and 16/35 files
2016-08-25 09:01:53,006 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.19/31.71 kb and 17/35 files
2016-08-25 09:01:53,007 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.08/31.71 kb and 18/35 files
2016-08-25 09:01:53,008 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.98/31.71 kb and 19/35 files
2016-08-25 09:01:53,008 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.88/31.71 kb and 20/35 files
2016-08-25 09:01:53,009 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.78/31.71 kb and 21/35 files
2016-08-25 09:01:53,009 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.68/31.71 kb and 22/35 files
2016-08-25 09:01:53,010 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.58/31.71 kb and 23/35 files
2016-08-25 09:01:53,010 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.48/31.71 kb and 24/35 files
2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.38/31.71 kb and 25/35 files
2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.57/31.71 kb and 26/35 files
2016-08-25 09:01:53,011 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.47/31.71 kb and 27/35 files
2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.37/31.71 kb and 28/35 files
2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.26/31.71 kb and 29/35 files
2016-08-25 09:01:53,012 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.17/31.71 kb and 30/35 files
2016-08-25 09:01:53,013 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.06/31.71 kb and 31/35 files
2016-08-25 09:01:53,013 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.01/31.71 kb and 32/35 files
2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.91/31.71 kb and 33/35 files
2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.68/31.71 kb and 1/35 files
2016-08-25 09:01:53,014 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files
2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files
2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.58/31.71 kb and 2/35 files
2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:53,015 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.48/31.71 kb and 3/35 files
2016-08-25 09:01:53,016 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.38/31.71 kb and 4/35 files
2016-08-25 09:01:53,016 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.28/31.71 kb and 5/35 files
2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.23/31.71 kb and 6/35 files
2016-08-25 09:01:53,017 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.13/31.71 kb and 7/35 files
2016-08-25 09:01:53,018 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is starting
2016-08-25 09:01:53,018 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/31.71 kb and 8/35 files
2016-08-25 09:01:53,018 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/31.71 kb and 9/35 files
2016-08-25 09:01:53,019 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.82/31.71 kb and 10/35 files
2016-08-25 09:01:53,021 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.00/31.71 kb and 11/35 files
2016-08-25 09:01:53,021 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.90/31.71 kb and 12/35 files
2016-08-25 09:01:53,022 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.80/31.71 kb and 13/35 files
2016-08-25 09:01:53,023 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.70/31.71 kb and 14/35 files
2016-08-25 09:01:53,023 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55443
2016-08-25 09:01:53,023 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:53,023 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.60/31.71 kb and 15/35 files
2016-08-25 09:01:53,024 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) started
2016-08-25 09:01:53,024 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:53,024 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:53,024 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:53,024 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.62/31.71 kb and 16/35 files
2016-08-25 09:01:53,024 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.51/31.71 kb and 17/35 files
2016-08-25 09:01:53,025 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.41/31.71 kb and 18/35 files
Master started: node-2
======================================
64.4. Browse the queue on new master
======================================
2016-08-25 09:01:53,026 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.31/31.71 kb and 19/35 files
2016-08-25 09:01:53,026 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.21/31.71 kb and 20/35 files
2016-08-25 09:01:53,027 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.11/31.71 kb and 21/35 files
2016-08-25 09:01:53,028 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.01/31.71 kb and 22/35 files
2016-08-25 09:01:53,029 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.92/31.71 kb and 23/35 files
2016-08-25 09:01:53,029 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.81/31.71 kb and 24/35 files
2016-08-25 09:01:53,030 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.71/31.71 kb and 25/35 files
2016-08-25 09:01:53,030 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.61/31.71 kb and 26/35 files
2016-08-25 09:01:53,031 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.51/31.71 kb and 27/35 files
2016-08-25 09:01:53,032 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.41/31.71 kb and 28/35 files
2016-08-25 09:01:53,033 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.31/31.71 kb and 29/35 files
2016-08-25 09:01:53,034 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.21/31.71 kb and 30/35 files
2016-08-25 09:01:53,035 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.11/31.71 kb and 31/35 files
2016-08-25 09:01:53,040 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.01/31.71 kb and 32/35 files
2016-08-25 09:01:53,041 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.91/31.71 kb and 33/35 files
2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.81/31.71 kb and 34/35 files
2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.71/31.71 kb and 35/35 files
2016-08-25 09:01:53,042 | INFO | hawtdispatch-DEFAULT-1 | Attached
======================================
64.5. Found
======================================
======================================
65.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:53,043 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is shutting down
2016-08-25 09:01:53,044 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:53,046 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:53,047 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:53,047 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250041
2016-08-25 09:01:53,047 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:53,047 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:53,047 | INFO | Time-limited test | Session: 0x156bec917250041 closed
2016-08-25 09:01:53,047 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:53,048 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55421 which had sessionid 0x156bec917250041
2016-08-25 09:01:53,055 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:53,056 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:53,150 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:53,151 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) uptime 3.384 seconds
2016-08-25 09:01:53,151 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:66) is shutdown
Wait for master to start up...
2016-08-25 09:01:53,153 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:53,156 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7003f305
2016-08-25 09:01:53,156 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:53,157 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:53,157 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:53,157 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55445
2016-08-25 09:01:53,157 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55445
2016-08-25 09:01:53,158 | INFO | SyncThread:0 | Established session 0x156bec917250044 with negotiated timeout 10000 for client /127.0.0.1:55445
2016-08-25 09:01:53,158 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250044, negotiated timeout = 10000
2016-08-25 09:01:53,166 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:53,772 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:53,776 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:53,796 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:53,796 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:53,799 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:53,803 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:54,029 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55446
2016-08-25 09:01:54,034 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:54,034 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:54,035 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55446
2016-08-25 09:01:54,035 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:54,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55446
2016-08-25 09:01:54,038 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:54,235 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:54,249 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000102.log
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000101.sst
2016-08-25 09:01:54,250 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000100
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:54,251 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000101.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:54,253 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000102.log
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:54,254 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000100
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:54,255 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:54,263 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files
2016-08-25 09:01:54,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files
2016-08-25 09:01:54,264 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files
2016-08-25 09:01:54,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files
2016-08-25 09:01:54,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files
2016-08-25 09:01:54,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files
2016-08-25 09:01:54,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.58/32.66 kb and 7/36 files
2016-08-25 09:01:54,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/32.66 kb and 8/36 files
2016-08-25 09:01:54,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.38/32.66 kb and 9/36 files
2016-08-25 09:01:54,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.28/32.66 kb and 10/36 files
2016-08-25 09:01:54,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.29/32.66 kb and 11/36 files
2016-08-25 09:01:54,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.19/32.66 kb and 12/36 files
2016-08-25 09:01:54,269 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.09/32.66 kb and 13/36 files
2016-08-25 09:01:54,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.99/32.66 kb and 14/36 files
2016-08-25 09:01:54,270 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.89/32.66 kb and 15/36 files
2016-08-25 09:01:54,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.79/32.66 kb and 16/36 files
2016-08-25 09:01:54,271 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.69/32.66 kb and 17/36 files
2016-08-25 09:01:54,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/32.66 kb and 18/36 files
2016-08-25 09:01:54,272 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.49/32.66 kb and 19/36 files
2016-08-25 09:01:54,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.39/32.66 kb and 20/36 files
2016-08-25 09:01:54,273 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.29/32.66 kb and 21/36 files
2016-08-25 09:01:54,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.02/32.66 kb and 22/36 files
2016-08-25 09:01:54,274 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.92/32.66 kb and 23/36 files
2016-08-25 09:01:54,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.82/32.66 kb and 24/36 files
2016-08-25 09:01:54,275 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.71/32.66 kb and 25/36 files
2016-08-25 09:01:54,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.61/32.66 kb and 26/36 files
2016-08-25 09:01:54,276 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.51/32.66 kb and 27/36 files
2016-08-25 09:01:54,277 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.46/32.66 kb and 28/36 files
2016-08-25 09:01:54,277 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.36/32.66 kb and 29/36 files
2016-08-25 09:01:54,278 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.26/32.66 kb and 30/36 files
2016-08-25 09:01:54,278 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files
2016-08-25 09:01:54,279 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files
2016-08-25 09:01:54,279 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files
2016-08-25 09:01:54,280 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files
2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files
2016-08-25 09:01:54,281 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files
2016-08-25 09:01:54,282 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.39/32.66 kb and 6/36 files
2016-08-25 09:01:54,282 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.29/32.66 kb and 7/36 files
2016-08-25 09:01:54,283 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.19/32.66 kb and 8/36 files
2016-08-25 09:01:54,284 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.09/32.66 kb and 9/36 files
2016-08-25 09:01:54,284 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.04/32.66 kb and 10/36 files
2016-08-25 09:01:54,285 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 9.94/32.66 kb and 11/36 files
2016-08-25 09:01:54,286 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 10.84/32.66 kb and 12/36 files
2016-08-25 09:01:54,286 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 11.74/32.66 kb and 13/36 files
2016-08-25 09:01:54,287 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 12.64/32.66 kb and 14/36 files
2016-08-25 09:01:54,287 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 13.54/32.66 kb and 15/36 files
2016-08-25 09:01:54,288 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 14.44/32.66 kb and 16/36 files
2016-08-25 09:01:54,288 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 15.34/32.66 kb and 17/36 files
2016-08-25 09:01:54,289 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 16.24/32.66 kb and 18/36 files
2016-08-25 09:01:54,289 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 17.14/32.66 kb and 19/36 files
2016-08-25 09:01:54,290 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 18.32/32.66 kb and 20/36 files
2016-08-25 09:01:54,291 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 19.22/32.66 kb and 21/36 files
2016-08-25 09:01:54,291 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 20.12/32.66 kb and 22/36 files
2016-08-25 09:01:54,292 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 21.02/32.66 kb and 23/36 files
2016-08-25 09:01:54,293 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 22.75/32.66 kb and 24/36 files
2016-08-25 09:01:54,293 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 23.65/32.66 kb and 25/36 files
2016-08-25 09:01:54,294 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 24.55/32.66 kb and 26/36 files
2016-08-25 09:01:54,295 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 25.45/32.66 kb and 27/36 files
2016-08-25 09:01:54,295 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 26.35/32.66 kb and 28/36 files
2016-08-25 09:01:54,296 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 27.25/32.66 kb and 29/36 files
2016-08-25 09:01:54,296 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.15/32.66 kb and 30/36 files
2016-08-25 09:01:54,297 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files
2016-08-25 09:01:54,297 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files
2016-08-25 09:01:54,298 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files
2016-08-25 09:01:54,299 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files
2016-08-25 09:01:54,299 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files
2016-08-25 09:01:54,300 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files
2016-08-25 09:01:54,300 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:01:54,305 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:54,305 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:54,306 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is starting
2016-08-25 09:01:54,310 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55451
2016-08-25 09:01:54,310 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:54,311 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) started
2016-08-25 09:01:54,311 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:54,311 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:54,311 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-1
======================================
65.4. Browse the queue on new master
======================================
======================================
65.5. Found
======================================
======================================
66.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:54,333 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is shutting down
2016-08-25 09:01:54,334 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:54,338 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:54,339 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:54,339 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:54,339 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:54,339 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:54,340 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:54,340 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250042
2016-08-25 09:01:54,341 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55429 which had sessionid 0x156bec917250042
2016-08-25 09:01:54,341 | INFO | Time-limited test | Session: 0x156bec917250042 closed
2016-08-25 09:01:54,341 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:54,463 | INFO | ActiveMQ BrokerService[node-1] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:54,464 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) uptime 3.669 seconds
2016-08-25 09:01:54,464 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:67) is shutdown
Wait for master to start up...
2016-08-25 09:01:54,466 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:54,469 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@17161429
2016-08-25 09:01:54,469 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:54,470 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:54,470 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:54,471 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55453
2016-08-25 09:01:54,471 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55453
2016-08-25 09:01:54,472 | INFO | SyncThread:0 | Established session 0x156bec917250045 with negotiated timeout 10000 for client /127.0.0.1:55453
2016-08-25 09:01:54,472 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250045, negotiated timeout = 10000
2016-08-25 09:01:54,487 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:54,797 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:54,798 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:54,808 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:54,808 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:54,812 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:54,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:55,036 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55454
2016-08-25 09:01:55,041 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:55,041 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:55,043 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55454
2016-08-25 09:01:55,043 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:55,047 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55454
2016-08-25 09:01:55,047 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:55,058 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:55,224 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:55,224 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000101.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000102.log
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:55,225 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000100
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:55,226 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:55,227 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:55,227 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:55,228 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:55,229 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000101.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:55,230 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000100
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000102.log
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:55,231 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:55,241 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files
2016-08-25 09:01:55,241 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files
2016-08-25 09:01:55,242 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files
2016-08-25 09:01:55,242 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files
2016-08-25 09:01:55,243 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files
2016-08-25 09:01:55,243 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files
2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.30/32.66 kb and 7/36 files
2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.20/32.66 kb and 8/36 files
2016-08-25 09:01:55,244 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.09/32.66 kb and 9/36 files
2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/32.66 kb and 1/36 files
2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/32.66 kb and 2/36 files
2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.00/32.66 kb and 10/36 files
2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/32.66 kb and 3/36 files
2016-08-25 09:01:55,245 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 10.18/32.66 kb and 11/36 files
2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/32.66 kb and 4/36 files
2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.08/32.66 kb and 12/36 files
2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/32.66 kb and 5/36 files
2016-08-25 09:01:55,246 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 11.98/32.66 kb and 13/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/32.66 kb and 6/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 12.88/32.66 kb and 14/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/32.66 kb and 7/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 13.78/32.66 kb and 15/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/32.66 kb and 8/36 files
2016-08-25 09:01:55,247 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 14.68/32.66 kb and 16/36 files
2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/32.66 kb and 9/36 files
2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 15.58/32.66 kb and 17/36 files
2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 16.53/32.66 kb and 18/36 files
2016-08-25 09:01:55,248 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/32.66 kb and 10/36 files
2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 17.42/32.66 kb and 19/36 files
2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/32.66 kb and 11/36 files
2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.32/32.66 kb and 20/36 files
2016-08-25 09:01:55,249 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/32.66 kb and 12/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 18.34/32.66 kb and 21/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/32.66 kb and 13/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/32.66 kb and 14/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 19.24/32.66 kb and 22/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 20.14/32.66 kb and 23/36 files
2016-08-25 09:01:55,250 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/32.66 kb and 15/36 files
2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.39/32.66 kb and 16/36 files
2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.04/32.66 kb and 24/36 files
2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.29/32.66 kb and 17/36 files
2016-08-25 09:01:55,251 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 21.94/32.66 kb and 25/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.19/32.66 kb and 18/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 22.84/32.66 kb and 26/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.09/32.66 kb and 19/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 23.74/32.66 kb and 27/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.11/32.66 kb and 20/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 24.64/32.66 kb and 28/36 files
2016-08-25 09:01:55,252 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.00/32.66 kb and 21/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 26.37/32.66 kb and 29/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.91/32.66 kb and 22/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 27.26/32.66 kb and 30/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.81/32.66 kb and 23/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 28.16/32.66 kb and 31/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.71/32.66 kb and 24/36 files
2016-08-25 09:01:55,253 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files
2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.61/32.66 kb and 25/36 files
2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files
2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.55/32.66 kb and 26/36 files
2016-08-25 09:01:55,254 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.45/32.66 kb and 27/36 files
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.35/32.66 kb and 28/36 files
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:55,255 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.08/32.66 kb and 29/36 files
2016-08-25 09:01:55,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.27/32.66 kb and 30/36 files
2016-08-25 09:01:55,256 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.17/32.66 kb and 31/36 files
2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.06/32.66 kb and 32/36 files
2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:55,257 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.96/32.66 kb and 33/36 files
2016-08-25 09:01:55,258 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.86/32.66 kb and 34/36 files
2016-08-25 09:01:55,258 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is starting
2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.76/32.66 kb and 35/36 files
2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.66/32.66 kb and 36/36 files
2016-08-25 09:01:55,259 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:55,262 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55459
2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) started
2016-08-25 09:01:55,263 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:55,263 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:01:55,263 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
Master started: node-3
======================================
66.4. Browse the queue on new master
======================================
2016-08-25 09:01:55,266 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
66.5. Found
======================================
======================================
67.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:55,279 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is shutting down
2016-08-25 09:01:55,280 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:55,285 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:55,286 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:55,287 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250043
2016-08-25 09:01:55,287 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55437 which had sessionid 0x156bec917250043
2016-08-25 09:01:55,288 | INFO | Time-limited test | Session: 0x156bec917250043 closed
2016-08-25 09:01:55,288 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:55,289 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:55,290 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:55,290 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:55,290 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:55,395 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:55,395 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) uptime 3.315 seconds
2016-08-25 09:01:55,396 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:68) is shutdown
Wait for master to start up...
2016-08-25 09:01:55,398 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:55,400 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@29a8186c
2016-08-25 09:01:55,400 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:55,400 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:55,401 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:55,401 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55461
2016-08-25 09:01:55,401 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55461
2016-08-25 09:01:55,402 | INFO | SyncThread:0 | Established session 0x156bec917250046 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55461
2016-08-25 09:01:55,402 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250046, negotiated timeout = 10000
2016-08-25 09:01:55,407 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:55,956 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:55,966 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:55,971 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:55,971 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:55,976 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:55,980 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:56,205 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55462
2016-08-25 09:01:56,211 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:56,211 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:56,212 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55462
2016-08-25 09:01:56,212 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:56,215 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55462
2016-08-25 09:01:56,215 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:56,425 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:56,435 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:56,435 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000103
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:56,436 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000101.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:56,437 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000105.log
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000104.sst
2016-08-25 09:01:56,438 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000071.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000104.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000065.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000068.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000041.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000044.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000077.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000038.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000080.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000014.sst
2016-08-25 09:01:56,439 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000011.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000095.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000053.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000026.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000056.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000105.log
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000092.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000017.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000101.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000029.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000086.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000089.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000074.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000062.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000103
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000020.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000059.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000008.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000032.sst
2016-08-25 09:01:56,440 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000098.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000047.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000083.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000050.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000023.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000005.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000035.sst
2016-08-25 09:01:56,441 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files
2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files
2016-08-25 09:01:56,455 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files
2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.58/33.61 kb and 3/37 files
2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files
2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.48/33.61 kb and 4/37 files
2016-08-25 09:01:56,456 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files
2016-08-25 09:01:56,457 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files
2016-08-25 09:01:56,457 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.38/33.61 kb and 5/37 files
2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files
2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.28/33.61 kb and 6/37 files
2016-08-25 09:01:56,458 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.40/33.61 kb and 6/37 files
2016-08-25 09:01:56,459 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.30/33.61 kb and 7/37 files
2016-08-25 09:01:56,459 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.18/33.61 kb and 7/37 files
2016-08-25 09:01:56,460 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.20/33.61 kb and 8/37 files
2016-08-25 09:01:56,460 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.08/33.61 kb and 8/37 files
2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.10/33.61 kb and 9/37 files
2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.98/33.61 kb and 9/37 files
2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.00/33.61 kb and 10/37 files
2016-08-25 09:01:56,461 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.93/33.61 kb and 10/37 files
2016-08-25 09:01:56,462 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.89/33.61 kb and 11/37 files
2016-08-25 09:01:56,462 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.83/33.61 kb and 11/37 files
2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.79/33.61 kb and 12/37 files
2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.72/33.61 kb and 12/37 files
2016-08-25 09:01:56,463 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.69/33.61 kb and 13/37 files
2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.62/33.61 kb and 13/37 files
2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.59/33.61 kb and 14/37 files
2016-08-25 09:01:56,464 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.52/33.61 kb and 14/37 files
2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.42/33.61 kb and 15/37 files
2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.49/33.61 kb and 15/37 files
2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.32/33.61 kb and 16/37 files
2016-08-25 09:01:56,465 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.67/33.61 kb and 16/37 files
2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.22/33.61 kb and 17/37 files
2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.57/33.61 kb and 17/37 files
2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.12/33.61 kb and 18/37 files
2016-08-25 09:01:56,466 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.47/33.61 kb and 18/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.02/33.61 kb and 19/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.37/33.61 kb and 19/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.92/33.61 kb and 20/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.27/33.61 kb and 20/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.17/33.61 kb and 21/37 files
2016-08-25 09:01:56,467 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.82/33.61 kb and 21/37 files
2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/33.61 kb and 22/37 files
2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.72/33.61 kb and 22/37 files
2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.97/33.61 kb and 23/37 files
2016-08-25 09:01:56,468 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.62/33.61 kb and 23/37 files
2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.87/33.61 kb and 24/37 files
2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.63/33.61 kb and 24/37 files
2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.65/33.61 kb and 25/37 files
2016-08-25 09:01:56,469 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.53/33.61 kb and 25/37 files
2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.55/33.61 kb and 26/37 files
2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.43/33.61 kb and 26/37 files
2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.45/33.61 kb and 27/37 files
2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.33/33.61 kb and 27/37 files
2016-08-25 09:01:56,470 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.35/33.61 kb and 28/37 files
2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.23/33.61 kb and 28/37 files
2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.25/33.61 kb and 29/37 files
2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.13/33.61 kb and 29/37 files
2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.15/33.61 kb and 30/37 files
2016-08-25 09:01:56,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.03/33.61 kb and 30/37 files
2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.05/33.61 kb and 31/37 files
2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.93/33.61 kb and 31/37 files
2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.95/33.61 kb and 32/37 files
2016-08-25 09:01:56,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.12/33.61 kb and 32/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.85/33.61 kb and 33/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.75/33.61 kb and 34/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.70/33.61 kb and 35/37 files
2016-08-25 09:01:56,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.59/33.61 kb and 36/37 files
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files
2016-08-25 09:01:56,474 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:56,480 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:56,481 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is starting
2016-08-25 09:01:56,483 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55467
2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:56,485 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) started
2016-08-25 09:01:56,486 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:56,486 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:56,486 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-2
======================================
67.4. Browse the queue on new master
======================================
======================================
67.5. Found
======================================
======================================
68.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:56,501 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is shutting down
2016-08-25 09:01:56,502 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:56,504 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:56,504 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:56,504 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250044
2016-08-25 09:01:56,505 | INFO | Time-limited test | Session: 0x156bec917250044 closed
2016-08-25 09:01:56,505 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:56,505 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55445 which had sessionid 0x156bec917250044
2016-08-25 09:01:56,508 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:56,508 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:56,508 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:56,508 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:56,621 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:56,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) uptime 3.471 seconds
2016-08-25 09:01:56,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:69) is shutdown
Wait for master to start up...
2016-08-25 09:01:56,624 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:56,626 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5af33ce2
2016-08-25 09:01:56,627 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:56,627 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:56,628 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:56,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55469
2016-08-25 09:01:56,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55469
2016-08-25 09:01:56,629 | INFO | SyncThread:0 | Established session 0x156bec917250047 with negotiated timeout 10000 for client /127.0.0.1:55469
2016-08-25 09:01:56,629 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250047, negotiated timeout = 10000
2016-08-25 09:01:56,636 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:57,031 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:57,033 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:57,034 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:57,034 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:57,044 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:57,049 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:57,298 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55470
2016-08-25 09:01:57,303 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:57,304 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:57,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55470
2016-08-25 09:01:57,306 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:57,311 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55470
2016-08-25 09:01:57,311 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:57,325 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:57,537 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:57,538 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:57,540 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000101.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000105.log
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000103
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:57,541 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000104.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:57,542 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:57,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000017.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000023.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000026.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000065.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000029.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000105.log
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000092.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000011.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000089.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000083.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000104.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000053.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000103
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000032.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000044.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000071.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000077.sst
2016-08-25 09:01:57,544 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000098.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000014.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000068.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000020.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000005.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000095.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000008.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000047.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000035.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000062.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000080.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000038.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000101.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000074.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000059.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000041.sst
2016-08-25 09:01:57,545 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000050.sst
2016-08-25 09:01:57,546 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000086.sst
2016-08-25 09:01:57,546 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000056.sst
2016-08-25 09:01:57,557 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files
2016-08-25 09:01:57,557 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files
2016-08-25 09:01:57,558 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files
2016-08-25 09:01:57,558 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files
2016-08-25 09:01:57,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files
2016-08-25 09:01:57,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/33.61 kb and 6/37 files
2016-08-25 09:01:57,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/33.61 kb and 7/37 files
2016-08-25 09:01:57,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.20/33.61 kb and 8/37 files
2016-08-25 09:01:57,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.09/33.61 kb and 9/37 files
2016-08-25 09:01:57,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.99/33.61 kb and 10/37 files
2016-08-25 09:01:57,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.89/33.61 kb and 11/37 files
2016-08-25 09:01:57,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 10.79/33.61 kb and 12/37 files
2016-08-25 09:01:57,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 11.69/33.61 kb and 13/37 files
2016-08-25 09:01:57,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.58/33.61 kb and 14/37 files
2016-08-25 09:01:57,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 12.60/33.61 kb and 15/37 files
2016-08-25 09:01:57,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 13.50/33.61 kb and 16/37 files
2016-08-25 09:01:57,565 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 14.40/33.61 kb and 17/37 files
2016-08-25 09:01:57,565 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 15.59/33.61 kb and 18/37 files
2016-08-25 09:01:57,566 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 16.49/33.61 kb and 19/37 files
2016-08-25 09:01:57,566 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 17.39/33.61 kb and 20/37 files
2016-08-25 09:01:57,567 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 18.29/33.61 kb and 21/37 files
2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 19.18/33.61 kb and 22/37 files
2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 20.96/33.61 kb and 23/37 files
2016-08-25 09:01:57,568 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 21.86/33.61 kb and 24/37 files
2016-08-25 09:01:57,569 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 22.76/33.61 kb and 25/37 files
2016-08-25 09:01:57,570 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 23.66/33.61 kb and 26/37 files
2016-08-25 09:01:57,570 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 24.56/33.61 kb and 27/37 files
2016-08-25 09:01:57,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 25.46/33.61 kb and 28/37 files
2016-08-25 09:01:57,571 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 26.36/33.61 kb and 29/37 files
2016-08-25 09:01:57,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 27.31/33.61 kb and 30/37 files
2016-08-25 09:01:57,572 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 28.21/33.61 kb and 31/37 files
2016-08-25 09:01:57,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 29.11/33.61 kb and 32/37 files
2016-08-25 09:01:57,573 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files
2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files
2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files
2016-08-25 09:01:57,574 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files
2016-08-25 09:01:57,575 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files
2016-08-25 09:01:57,575 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:57,578 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:57,579 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is starting
2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55475
2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) started
2016-08-25 09:01:57,583 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:57,584 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:57,584 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:57,585 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/33.61 kb and 1/37 files
2016-08-25 09:01:57,585 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/33.61 kb and 2/37 files
2016-08-25 09:01:57,586 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/33.61 kb and 3/37 files
2016-08-25 09:01:57,586 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/33.61 kb and 4/37 files
2016-08-25 09:01:57,587 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.50/33.61 kb and 5/37 files
2016-08-25 09:01:57,588 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.68/33.61 kb and 6/37 files
Master started: node-1
======================================
68.4. Browse the queue on new master
======================================
2016-08-25 09:01:57,588 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/33.61 kb and 7/37 files
2016-08-25 09:01:57,589 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.48/33.61 kb and 8/37 files
2016-08-25 09:01:57,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.38/33.61 kb and 9/37 files
2016-08-25 09:01:57,590 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.28/33.61 kb and 10/37 files
2016-08-25 09:01:57,591 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 10.18/33.61 kb and 11/37 files
2016-08-25 09:01:57,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 11.08/33.61 kb and 12/37 files
2016-08-25 09:01:57,592 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 12.86/33.61 kb and 13/37 files
2016-08-25 09:01:57,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 13.76/33.61 kb and 14/37 files
2016-08-25 09:01:57,593 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 14.66/33.61 kb and 15/37 files
2016-08-25 09:01:57,594 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 15.56/33.61 kb and 16/37 files
2016-08-25 09:01:57,599 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 16.46/33.61 kb and 17/37 files
2016-08-25 09:01:57,600 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 17.36/33.61 kb and 18/37 files
2016-08-25 09:01:57,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 18.25/33.61 kb and 19/37 files
2016-08-25 09:01:57,601 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 19.15/33.61 kb and 20/37 files
2016-08-25 09:01:57,602 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.05/33.61 kb and 21/37 files
2016-08-25 09:01:57,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 20.07/33.61 kb and 22/37 files
2016-08-25 09:01:57,603 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.02/33.61 kb and 23/37 files
2016-08-25 09:01:57,604 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 21.92/33.61 kb and 24/37 files
2016-08-25 09:01:57,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 22.82/33.61 kb and 25/37 files
2016-08-25 09:01:57,605 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 23.72/33.61 kb and 26/37 files
2016-08-25 09:01:57,606 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 24.62/33.61 kb and 27/37 files
2016-08-25 09:01:57,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 25.52/33.61 kb and 28/37 files
2016-08-25 09:01:57,607 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 26.42/33.61 kb and 29/37 files
2016-08-25 09:01:57,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 27.32/33.61 kb and 30/37 files
2016-08-25 09:01:57,608 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 28.21/33.61 kb and 31/37 files
2016-08-25 09:01:57,609 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 29.11/33.61 kb and 32/37 files
2016-08-25 09:01:57,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.01/33.61 kb and 33/37 files
2016-08-25 09:01:57,611 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 30.91/33.61 kb and 34/37 files
2016-08-25 09:01:57,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 31.81/33.61 kb and 35/37 files
2016-08-25 09:01:57,612 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 32.71/33.61 kb and 36/37 files
2016-08-25 09:01:57,613 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 33.61/33.61 kb and 37/37 files
2016-08-25 09:01:57,613 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:01:57,623 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
68.5. Found
======================================
======================================
69.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:57,633 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is shutting down
2016-08-25 09:01:58,636 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:58,644 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:58,644 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,644 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250045
2016-08-25 09:01:58,645 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:58,645 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:58,645 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:58,645 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:58,648 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55453 which had sessionid 0x156bec917250045
2016-08-25 09:01:58,648 | INFO | Time-limited test | Session: 0x156bec917250045 closed
2016-08-25 09:01:58,648 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:58,661 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,662 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,664 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:58,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:58,696 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55477
2016-08-25 09:01:58,700 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:58,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55477
2016-08-25 09:01:58,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:01:58,711 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:58,713 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:58,713 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000108.log
2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:01:58,714 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000106
2016-08-25 09:01:58,716 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.02/4.17 kb and 1/4 files
2016-08-25 09:01:58,716 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.20/4.17 kb and 2/4 files
2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.14/4.17 kb and 3/4 files
2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files
2016-08-25 09:01:58,717 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:58,719 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:58,720 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is starting
2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55480
2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) started
2016-08-25 09:01:58,724 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:58,725 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:58,725 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:58,753 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:58,753 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) uptime 4.289 seconds
2016-08-25 09:01:58,754 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:70) is shutdown
Wait for master to start up...
Master started: node-3
======================================
69.4. Browse the queue on new master
======================================
2016-08-25 09:01:58,756 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:58,758 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4c6425fa
2016-08-25 09:01:58,758 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:58,759 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:58,759 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:58,759 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55482
2016-08-25 09:01:58,759 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55482
2016-08-25 09:01:58,760 | INFO | SyncThread:0 | Established session 0x156bec917250048 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55482
2016-08-25 09:01:58,760 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250048, negotiated timeout = 10000
======================================
69.5. Found
======================================
======================================
70.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:58,770 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is shutting down
2016-08-25 09:01:58,771 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:58,772 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:58,772 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55477
2016-08-25 09:01:58,772 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:58,775 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,777 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:58,778 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250046
2016-08-25 09:01:58,778 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:58,778 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,778 | INFO | Time-limited test | Session: 0x156bec917250046 closed
2016-08-25 09:01:58,778 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:58,778 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55461 which had sessionid 0x156bec917250046
2016-08-25 09:01:58,781 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:58,898 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:58,899 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) uptime 3.503 seconds
2016-08-25 09:01:58,899 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:71) is shutdown
Wait for master to start up...
2016-08-25 09:01:58,901 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:58,903 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@41b861ab
2016-08-25 09:01:58,903 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:58,904 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:58,905 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:58,905 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55484
2016-08-25 09:01:58,905 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55484
2016-08-25 09:01:58,906 | INFO | SyncThread:0 | Established session 0x156bec917250049 with negotiated timeout 10000 for client /127.0.0.1:55484
2016-08-25 09:01:58,907 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250049, negotiated timeout = 10000
2016-08-25 09:01:58,918 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,158 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:59,158 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,158 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,159 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:59,161 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:59,164 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,181 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55485
2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55485
2016-08-25 09:01:59,184 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:59,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55485
2016-08-25 09:01:59,185 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:59,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,365 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:59,365 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000108.log
2016-08-25 09:01:59,366 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000106
2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/4.17 kb and 1/4 files
2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,370 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.96/4.17 kb and 2/4 files
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.14/4.17 kb and 3/4 files
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000106
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000108.log
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files
2016-08-25 09:01:59,371 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:59,375 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,376 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.03/4.17 kb and 1/4 files
2016-08-25 09:01:59,376 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/4.17 kb and 2/4 files
2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.23/4.17 kb and 3/4 files
2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 4/4 files
2016-08-25 09:01:59,377 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:59,378 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is starting
2016-08-25 09:01:59,382 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55490
2016-08-25 09:01:59,382 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:59,383 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) started
2016-08-25 09:01:59,383 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:59,383 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:59,383 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-2
======================================
70.4. Browse the queue on new master
======================================
2016-08-25 09:01:59,386 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
70.5. Found
======================================
======================================
71.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:59,399 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is shutting down
2016-08-25 09:01:59,399 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:59,399 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | socketQueue interrupted - stopping
2016-08-25 09:01:59,399 | INFO | ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:0 | Could not accept connection during shutdown : java.lang.InterruptedException
2016-08-25 09:01:59,402 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,403 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,403 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,404 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,404 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:59,404 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:59,404 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250047
2016-08-25 09:01:59,405 | INFO | Time-limited test | Session: 0x156bec917250047 closed
2016-08-25 09:01:59,405 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:59,405 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55469 which had sessionid 0x156bec917250047
2016-08-25 09:01:59,438 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:59,440 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,443 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:59,443 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,446 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:59,451 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,480 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55492
2016-08-25 09:01:59,483 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,484 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55492
2016-08-25 09:01:59,485 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:01:59,500 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,502 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000112.log
2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:01:59,503 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000110
2016-08-25 09:01:59,507 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/3.22 kb and 1/5 files
2016-08-25 09:01:59,508 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.20/3.22 kb and 2/5 files
2016-08-25 09:01:59,508 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.14/3.22 kb and 3/5 files
2016-08-25 09:01:59,509 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/3.22 kb and 4/5 files
2016-08-25 09:01:59,509 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:59,510 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files
2016-08-25 09:01:59,510 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) uptime 2.888 seconds
2016-08-25 09:01:59,510 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:01:59,510 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:72) is shutdown
Wait for master to start up...
2016-08-25 09:01:59,512 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:59,513 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,514 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is starting
2016-08-25 09:01:59,516 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@28370f7b
2016-08-25 09:01:59,516 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:59,517 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:59,518 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55495
2016-08-25 09:01:59,518 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55496
2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) started
2016-08-25 09:01:59,518 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:59,518 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55496
2016-08-25 09:01:59,519 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:59,519 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:01:59,519 | INFO | SyncThread:0 | Established session 0x156bec91725004a with negotiated timeout 10000 for client /127.0.0.1:55496
2016-08-25 09:01:59,520 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004a, negotiated timeout = 10000
Master started: node-1
======================================
71.4. Browse the queue on new master
======================================
2016-08-25 09:01:59,529 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,530 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55492
2016-08-25 09:01:59,530 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
======================================
71.5. Found
======================================
======================================
72.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:59,537 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is shutting down
2016-08-25 09:01:59,538 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:59,541 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,542 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:59,542 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250048
2016-08-25 09:01:59,543 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55482 which had sessionid 0x156bec917250048
2016-08-25 09:01:59,543 | INFO | Time-limited test | Session: 0x156bec917250048 closed
2016-08-25 09:01:59,543 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:59,552 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,559 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:01:59,559 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,560 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,561 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,571 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:01:59,571 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,574 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:01:59,578 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,606 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55499
2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55499
2016-08-25 09:01:59,609 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:59,618 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:59,620 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000110
2016-08-25 09:01:59,621 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000112.log
2016-08-25 09:01:59,625 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/3.22 kb and 1/5 files
2016-08-25 09:01:59,627 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.92/3.22 kb and 2/5 files
2016-08-25 09:01:59,628 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.86/3.22 kb and 3/5 files
2016-08-25 09:01:59,628 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.04/3.22 kb and 4/5 files
2016-08-25 09:01:59,629 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files
2016-08-25 09:01:59,629 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:59,631 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:59,635 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is starting
2016-08-25 09:01:59,638 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55502
2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) started
2016-08-25 09:01:59,639 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:59,639 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,639 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,668 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:01:59,669 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) uptime 0.915 seconds
2016-08-25 09:01:59,669 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:73) is shutdown
Wait for master to start up...
Master started: node-3
======================================
72.4. Browse the queue on new master
======================================
2016-08-25 09:01:59,671 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:01:59,673 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@61a8a328
2016-08-25 09:01:59,673 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:59,673 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:59,674 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:59,674 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55504
2016-08-25 09:01:59,674 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55504
2016-08-25 09:01:59,675 | INFO | SyncThread:0 | Established session 0x156bec91725004b with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55504
2016-08-25 09:01:59,675 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004b, negotiated timeout = 10000
======================================
72.5. Found
======================================
======================================
73.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:01:59,681 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is shutting down
2016-08-25 09:01:59,681 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:59,682 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,683 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55499
2016-08-25 09:01:59,683 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:01:59,683 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,685 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,685 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:59,685 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,685 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:01:59,685 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250049
2016-08-25 09:01:59,685 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:01:59,686 | INFO | Time-limited test | Session: 0x156bec917250049 closed
2016-08-25 09:01:59,686 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55484 which had sessionid 0x156bec917250049
2016-08-25 09:01:59,686 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:01:59,687 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:01:59,701 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,704 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:01:59,707 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,739 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55506
2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55506
2016-08-25 09:01:59,742 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:01:59,764 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:01:59,767 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000115.log
2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:01:59,768 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000113
2016-08-25 09:01:59,771 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/4.17 kb and 1/6 files
2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/4.17 kb and 2/6 files
2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.86/4.17 kb and 3/6 files
2016-08-25 09:01:59,772 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.04/4.17 kb and 4/6 files
2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.94/4.17 kb and 5/6 files
2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files
2016-08-25 09:01:59,773 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:59,775 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:59,776 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is starting
2016-08-25 09:01:59,781 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55509
2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) started
2016-08-25 09:01:59,782 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:59,782 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,782 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,790 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:01:59,791 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) uptime 0.892 seconds
2016-08-25 09:01:59,791 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:74) is shutdown
Wait for master to start up...
Master started: node-2
======================================
73.4. Browse the queue on new master
======================================
2016-08-25 09:01:59,794 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:01:59,798 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@79ea3abe
2016-08-25 09:01:59,798 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:59,799 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:59,799 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:01:59,799 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55511
2016-08-25 09:01:59,800 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55511
2016-08-25 09:01:59,801 | INFO | SyncThread:0 | Established session 0x156bec91725004c with negotiated timeout 10000 for client /127.0.0.1:55511
2016-08-25 09:01:59,801 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004c, negotiated timeout = 10000
======================================
73.5. Found
======================================
======================================
74.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:01:59,809 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is shutting down
2016-08-25 09:01:59,810 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:59,811 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:59,812 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,812 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004a
2016-08-25 09:01:59,813 | INFO | Time-limited test | Session: 0x156bec91725004a closed
2016-08-25 09:01:59,813 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:01:59,813 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55496 which had sessionid 0x156bec91725004a
2016-08-25 09:01:59,814 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,816 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55506
2016-08-25 09:01:59,816 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:01:59,817 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,817 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:01:59,817 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Slave stopped
2016-08-25 09:01:59,817 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,819 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,838 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:01:59,838 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,842 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:01:59,844 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,879 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55513
2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Attaching to master: tcp://localhost:55513
2016-08-25 09:01:59,882 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Slave started
2016-08-25 09:01:59,900 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,902 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000113
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:01:59,903 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000115.log
2016-08-25 09:01:59,906 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/4.17 kb and 1/6 files
2016-08-25 09:01:59,906 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.17/4.17 kb and 2/6 files
2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.07/4.17 kb and 3/6 files
2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.09/4.17 kb and 4/6 files
2016-08-25 09:01:59,907 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.99/4.17 kb and 5/6 files
2016-08-25 09:01:59,908 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files
2016-08-25 09:01:59,908 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:01:59,910 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,910 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is starting
2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55516
2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:01:59,914 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) started
2016-08-25 09:01:59,915 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:01:59,915 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,915 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30781 mb of usable space. - resetting to maximum available disk space: 30781 mb
2016-08-25 09:01:59,948 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:01:59,949 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) uptime 0.439 seconds
2016-08-25 09:01:59,949 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:75) is shutdown
Wait for master to start up...
Master started: node-1
======================================
74.4. Browse the queue on new master
======================================
2016-08-25 09:01:59,951 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:01:59,955 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@301ed798
2016-08-25 09:01:59,955 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:01:59,956 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:01:59,957 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:01:59,957 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55518
2016-08-25 09:01:59,957 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55518
2016-08-25 09:01:59,958 | INFO | SyncThread:0 | Established session 0x156bec91725004d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55518
2016-08-25 09:01:59,958 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004d, negotiated timeout = 10000
======================================
74.5. Found
======================================
======================================
75.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:01:59,966 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is shutting down
2016-08-25 09:01:59,967 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:01:59,969 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:01:59,970 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004b
2016-08-25 09:01:59,970 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:01:59,970 | INFO | Time-limited test | Session: 0x156bec91725004b closed
2016-08-25 09:01:59,971 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:01:59,971 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55504 which had sessionid 0x156bec91725004b
2016-08-25 09:01:59,971 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,971 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:01:59,972 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:01:59,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55513
2016-08-25 09:01:59,974 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:01:59,975 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped
2016-08-25 09:01:59,975 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:01:59,977 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,003 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:00,003 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,007 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:00,009 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,049 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Master started: tcp://localhost:55520
2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Attaching to master: tcp://localhost:55520
2016-08-25 09:02:00,052 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave started
2016-08-25 09:02:00,079 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:00,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) uptime 0.410 seconds
2016-08-25 09:02:00,079 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:76) is shutdown
Wait for master to start up...
2016-08-25 09:02:00,081 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:00,083 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,083 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@318be84d
2016-08-25 09:02:00,084 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:00,085 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:00,086 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:00,086 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55523
2016-08-25 09:02:00,086 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55523
2016-08-25 09:02:00,087 | INFO | SyncThread:0 | Established session 0x156bec91725004e with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55523
2016-08-25 09:02:00,087 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,087 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725004e, negotiated timeout = 10000
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000118.log
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000116
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:00,089 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:00,095 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/5.12 kb and 1/7 files
2016-08-25 09:02:00,096 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/5.12 kb and 2/7 files
2016-08-25 09:02:00,097 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.74/5.12 kb and 3/7 files
2016-08-25 09:02:00,097 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.92/5.12 kb and 4/7 files
2016-08-25 09:02:00,097 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,098 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.21/5.12 kb and 5/7 files
2016-08-25 09:02:00,098 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.22/5.12 kb and 6/7 files
2016-08-25 09:02:00,099 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files
2016-08-25 09:02:00,099 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:00,099 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55520
2016-08-25 09:02:00,099 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:00,104 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,106 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is starting
2016-08-25 09:02:00,112 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55525
2016-08-25 09:02:00,112 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:00,113 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) started
2016-08-25 09:02:00,113 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:00,113 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,113 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-3
======================================
75.4. Browse the queue on new master
======================================
======================================
75.5. Found
======================================
======================================
76.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:00,129 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is shutting down
2016-08-25 09:02:00,130 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:00,134 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,137 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:00,138 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004c
2016-08-25 09:02:00,139 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55511 which had sessionid 0x156bec91725004c
2016-08-25 09:02:00,139 | INFO | Time-limited test | Session: 0x156bec91725004c closed
2016-08-25 09:02:00,139 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:00,140 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,140 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,164 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:00,164 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,167 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped
2016-08-25 09:02:00,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,192 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:00,195 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,230 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55527
2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55527
2016-08-25 09:02:00,234 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:00,253 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:00,254 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) uptime 0.463 seconds
2016-08-25 09:02:00,254 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:77) is shutdown
Wait for master to start up...
2016-08-25 09:02:00,255 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,256 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:00,258 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@567172fc
2016-08-25 09:02:00,258 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:00,258 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,259 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:00,259 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000118.log
2016-08-25 09:02:00,259 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55530
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:00,259 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:00,259 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55530
2016-08-25 09:02:00,260 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000116
2016-08-25 09:02:00,260 | INFO | SyncThread:0 | Established session 0x156bec91725004f with negotiated timeout 10000 for client /127.0.0.1:55530
2016-08-25 09:02:00,260 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725004f, negotiated timeout = 10000
2016-08-25 09:02:00,265 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files
2016-08-25 09:02:00,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.96/5.12 kb and 2/7 files
2016-08-25 09:02:00,266 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.86/5.12 kb and 3/7 files
2016-08-25 09:02:00,267 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,267 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/5.12 kb and 4/7 files
2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.94/5.12 kb and 5/7 files
2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.84/5.12 kb and 6/7 files
2016-08-25 09:02:00,268 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55527
2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files
2016-08-25 09:02:00,268 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:00,268 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:00,271 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,272 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is starting
2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55532
2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) started
2016-08-25 09:02:00,276 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:00,276 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,277 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-2
======================================
76.4. Browse the queue on new master
======================================
2016-08-25 09:02:00,292 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
76.5. Found
======================================
======================================
77.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:00,295 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is shutting down
2016-08-25 09:02:00,295 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:00,298 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,299 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,299 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:00,299 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,299 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004d
2016-08-25 09:02:00,300 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,300 | INFO | Time-limited test | Session: 0x156bec91725004d closed
2016-08-25 09:02:00,301 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55518 which had sessionid 0x156bec91725004d
2016-08-25 09:02:00,301 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000118.log
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000116
2016-08-25 09:02:00,302 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:00,302 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,303 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,303 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,313 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:00,314 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,375 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:00,375 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,378 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:00,381 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,407 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:00,408 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) uptime 0.458 seconds
2016-08-25 09:02:00,408 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:78) is shutdown
Wait for master to start up...
2016-08-25 09:02:00,410 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:00,412 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@2c6bedc9
2016-08-25 09:02:00,412 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:00,413 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:00,413 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:00,413 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55535
2016-08-25 09:02:00,413 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55535
2016-08-25 09:02:00,414 | INFO | SyncThread:0 | Established session 0x156bec917250050 with negotiated timeout 10000 for client /127.0.0.1:55535
2016-08-25 09:02:00,414 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250050, negotiated timeout = 10000
2016-08-25 09:02:00,429 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55536
2016-08-25 09:02:00,432 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,432 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,433 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55536
2016-08-25 09:02:00,433 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:00,434 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55536
2016-08-25 09:02:00,434 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:00,449 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,464 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,465 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000119
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000121.log
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:00,466 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:00,467 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000119
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000121.log
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:00,468 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:00,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.33/6.07 kb and 1/8 files
2016-08-25 09:02:00,470 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.23/6.07 kb and 2/8 files
2016-08-25 09:02:00,471 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.42/6.07 kb and 3/8 files
2016-08-25 09:02:00,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.32/6.07 kb and 4/8 files
2016-08-25 09:02:00,472 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.22/6.07 kb and 5/8 files
2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.23/6.07 kb and 6/8 files
2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.13/6.07 kb and 7/8 files
2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.94/6.07 kb and 1/8 files
2016-08-25 09:02:00,473 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:00,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.28/6.07 kb and 2/8 files
2016-08-25 09:02:00,474 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.29/6.07 kb and 3/8 files
2016-08-25 09:02:00,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.19/6.07 kb and 4/8 files
2016-08-25 09:02:00,475 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.38/6.07 kb and 5/8 files
2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.28/6.07 kb and 6/8 files
2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files
2016-08-25 09:02:00,476 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,477 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:00,477 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:00,477 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is starting
2016-08-25 09:02:00,480 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55541
2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) started
2016-08-25 09:02:00,482 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:00,482 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,482 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-1
======================================
77.4. Browse the queue on new master
======================================
======================================
77.5. Found
======================================
======================================
78.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:00,499 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is shutting down
2016-08-25 09:02:00,500 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:00,503 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,504 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:00,504 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004e
2016-08-25 09:02:00,505 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55523 which had sessionid 0x156bec91725004e
2016-08-25 09:02:00,505 | INFO | Time-limited test | Session: 0x156bec91725004e closed
2016-08-25 09:02:00,505 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:00,509 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,509 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,509 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,509 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:00,578 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,580 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,583 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:00,586 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,621 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:00,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) uptime 0.543 seconds
2016-08-25 09:02:00,622 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:79) is shutdown
Wait for master to start up...
2016-08-25 09:02:00,624 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:00,626 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1a984a63
2016-08-25 09:02:00,626 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:00,627 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:00,627 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:00,627 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55543
2016-08-25 09:02:00,628 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55543
2016-08-25 09:02:00,628 | INFO | SyncThread:0 | Established session 0x156bec917250051 with negotiated timeout 10000 for client /127.0.0.1:55543
2016-08-25 09:02:00,628 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250051, negotiated timeout = 10000
2016-08-25 09:02:00,636 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55544
2016-08-25 09:02:00,640 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,640 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55544
2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55544
2016-08-25 09:02:00,641 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:00,674 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,683 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000119
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000121.log
2016-08-25 09:02:00,684 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:00,688 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/6.07 kb and 1/8 files
2016-08-25 09:02:00,688 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.92/6.07 kb and 2/8 files
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000121.log
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/6.07 kb and 3/8 files
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000119
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:00,689 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:00,690 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.71/6.07 kb and 4/8 files
2016-08-25 09:02:00,690 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.66/6.07 kb and 5/8 files
2016-08-25 09:02:00,691 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.99/6.07 kb and 6/8 files
2016-08-25 09:02:00,691 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files
2016-08-25 09:02:00,692 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:00,692 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:00,694 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files
2016-08-25 09:02:00,694 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/6.07 kb and 2/8 files
2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.98/6.07 kb and 3/8 files
2016-08-25 09:02:00,695 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/6.07 kb and 4/8 files
2016-08-25 09:02:00,696 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is starting
2016-08-25 09:02:00,696 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.82/6.07 kb and 5/8 files
2016-08-25 09:02:00,696 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.16/6.07 kb and 6/8 files
2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.06/6.07 kb and 7/8 files
2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:00,697 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55549
2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) started
2016-08-25 09:02:00,700 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:00,700 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,700 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,701 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
Master started: node-3
======================================
78.4. Browse the queue on new master
======================================
======================================
78.5. Found
======================================
======================================
79.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:00,717 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is shutting down
2016-08-25 09:02:00,718 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:00,720 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,721 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:00,721 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725004f
2016-08-25 09:02:00,721 | INFO | Time-limited test | Session: 0x156bec91725004f closed
2016-08-25 09:02:00,721 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:00,722 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55530 which had sessionid 0x156bec91725004f
2016-08-25 09:02:00,724 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,724 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:00,724 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:00,724 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,826 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:00,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) uptime 0.572 seconds
2016-08-25 09:02:00,826 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:80) is shutdown
Wait for master to start up...
2016-08-25 09:02:00,828 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:00,832 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5652fdf4
2016-08-25 09:02:00,832 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:00,833 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:00,833 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,833 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:00,834 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55551
2016-08-25 09:02:00,834 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:00,834 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55551
2016-08-25 09:02:00,835 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:00,836 | INFO | SyncThread:0 | Established session 0x156bec917250052 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55551
2016-08-25 09:02:00,836 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250052, negotiated timeout = 10000
2016-08-25 09:02:00,837 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:00,840 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:00,847 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,901 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55552
2016-08-25 09:02:00,905 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,905 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55552
2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55552
2016-08-25 09:02:00,907 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:00,953 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,953 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:00,957 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000124.log
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000122
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000124.log
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000122
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:00,958 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:00,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files
2016-08-25 09:02:00,963 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/7.02 kb and 2/9 files
2016-08-25 09:02:00,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/7.02 kb and 3/9 files
2016-08-25 09:02:00,964 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/7.02 kb and 4/9 files
2016-08-25 09:02:00,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.78/7.02 kb and 5/9 files
2016-08-25 09:02:00,965 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.80/7.02 kb and 6/9 files
2016-08-25 09:02:00,966 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.18/7.02 kb and 7/9 files
2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.08/7.02 kb and 8/9 files
2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:00,967 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:00,968 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/7.02 kb and 1/9 files
2016-08-25 09:02:00,968 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.02 kb and 2/9 files
2016-08-25 09:02:00,969 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/7.02 kb and 3/9 files
2016-08-25 09:02:00,969 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/7.02 kb and 4/9 files
2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.31/7.02 kb and 5/9 files
2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:00,970 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.21/7.02 kb and 6/9 files
2016-08-25 09:02:00,971 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.11/7.02 kb and 7/9 files
2016-08-25 09:02:00,971 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is starting
2016-08-25 09:02:00,971 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.01/7.02 kb and 8/9 files
2016-08-25 09:02:00,972 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:00,972 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:00,976 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55557
2016-08-25 09:02:00,976 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:00,977 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:00,977 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) started
2016-08-25 09:02:00,977 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:00,977 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:00,977 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
Master started: node-2
======================================
79.4. Browse the queue on new master
======================================
======================================
79.5. Found
======================================
======================================
80.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:00,998 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is shutting down
2016-08-25 09:02:00,998 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:01,006 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,009 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:01,010 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250050
2016-08-25 09:02:01,011 | INFO | Time-limited test | Session: 0x156bec917250050 closed
2016-08-25 09:02:01,012 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:01,012 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55535 which had sessionid 0x156bec917250050
2016-08-25 09:02:01,013 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,014 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:01,014 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,014 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:01,102 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:01,104 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,104 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:01,104 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,108 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:01,111 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,127 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:01,127 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) uptime 0.719 seconds
2016-08-25 09:02:01,127 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:81) is shutdown
Wait for master to start up...
2016-08-25 09:02:01,129 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:01,131 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@33fca8e4
2016-08-25 09:02:01,131 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:01,132 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:01,132 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:01,132 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55559
2016-08-25 09:02:01,133 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55559
2016-08-25 09:02:01,133 | INFO | SyncThread:0 | Established session 0x156bec917250053 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55559
2016-08-25 09:02:01,133 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250053, negotiated timeout = 10000
2016-08-25 09:02:01,164 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55560
2016-08-25 09:02:01,168 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,169 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55560
2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55560
2016-08-25 09:02:01,170 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:01,181 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,184 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000122
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:01,185 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:01,186 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000124.log
2016-08-25 09:02:01,190 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/7.02 kb and 1/9 files
2016-08-25 09:02:01,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.96/7.02 kb and 2/9 files
2016-08-25 09:02:01,191 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.34/7.02 kb and 3/9 files
2016-08-25 09:02:01,192 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.24/7.02 kb and 4/9 files
2016-08-25 09:02:01,193 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.14/7.02 kb and 5/9 files
2016-08-25 09:02:01,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.04/7.02 kb and 6/9 files
2016-08-25 09:02:01,195 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.94/7.02 kb and 7/9 files
2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.84/7.02 kb and 8/9 files
2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:01,196 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:01,199 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,200 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is starting
2016-08-25 09:02:01,204 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55564
2016-08-25 09:02:01,204 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:01,205 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) started
2016-08-25 09:02:01,205 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:01,205 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
2016-08-25 09:02:01,205 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
Master started: node-1
======================================
80.4. Browse the queue on new master
======================================
======================================
80.5. Found
======================================
======================================
81.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:01,217 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is shutting down
2016-08-25 09:02:01,217 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:01,220 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,221 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,221 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,221 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:01,221 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250051
2016-08-25 09:02:01,222 | INFO | Time-limited test | Session: 0x156bec917250051 closed
2016-08-25 09:02:01,222 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55543 which had sessionid 0x156bec917250051
2016-08-25 09:02:01,222 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:01,234 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,274 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:01,275 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,278 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,327 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:01,328 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) uptime 0.706 seconds
2016-08-25 09:02:01,328 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:82) is shutdown
Wait for master to start up...
2016-08-25 09:02:01,330 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:01,332 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3f91bd18
2016-08-25 09:02:01,332 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:01,333 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:01,333 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:01,333 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55566
2016-08-25 09:02:01,334 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55566
2016-08-25 09:02:01,335 | INFO | SyncThread:0 | Established session 0x156bec917250054 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55566
2016-08-25 09:02:01,335 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250054, negotiated timeout = 10000
2016-08-25 09:02:01,342 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,349 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:01,353 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:01,356 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,411 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55567
2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55567
2016-08-25 09:02:01,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:01,417 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55567
2016-08-25 09:02:01,417 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:01,454 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:01,493 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:01,494 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:01,494 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000127.log
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000125
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:01,495 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:01,500 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/7.98 kb and 1/10 files
2016-08-25 09:02:01,500 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/7.98 kb and 2/10 files
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.10/7.98 kb and 3/10 files
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000125
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:01,501 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000127.log
2016-08-25 09:02:01,502 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.00/7.98 kb and 4/10 files
2016-08-25 09:02:01,502 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.90/7.98 kb and 5/10 files
2016-08-25 09:02:01,503 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.80/7.98 kb and 6/10 files
2016-08-25 09:02:01,503 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.70/7.98 kb and 7/10 files
2016-08-25 09:02:01,504 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.13/7.98 kb and 8/10 files
2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files
2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files
2016-08-25 09:02:01,505 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:01,507 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files
2016-08-25 09:02:01,507 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.98 kb and 2/10 files
2016-08-25 09:02:01,508 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/7.98 kb and 3/10 files
2016-08-25 09:02:01,508 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:01,509 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.64/7.98 kb and 4/10 files
2016-08-25 09:02:01,509 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is starting
2016-08-25 09:02:01,509 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/7.98 kb and 5/10 files
2016-08-25 09:02:01,510 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.55/7.98 kb and 6/10 files
2016-08-25 09:02:01,510 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.99/7.98 kb and 7/10 files
2016-08-25 09:02:01,511 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.89/7.98 kb and 8/10 files
2016-08-25 09:02:01,511 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.79/7.98 kb and 9/10 files
2016-08-25 09:02:01,512 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files
2016-08-25 09:02:01,512 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:01,513 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55572
2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) started
2016-08-25 09:02:01,514 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:01,514 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
2016-08-25 09:02:01,514 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
Master started: node-2
======================================
81.4. Browse the queue on new master
======================================
2016-08-25 09:02:01,516 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
81.5. Found
======================================
======================================
82.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:01,531 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is shutting down
2016-08-25 09:02:01,532 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:01,537 | INFO | Starting broker node: node-3-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:02:01,537 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,543 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:01,543 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250053
2016-08-25 09:02:01,544 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,544 | INFO | Time-limited test | Session: 0x156bec917250053 closed
2016-08-25 09:02:01,544 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:01,544 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55559 which had sessionid 0x156bec917250053
2016-08-25 09:02:01,545 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,546 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:01,546 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,547 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:01,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:01,631 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,633 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:01,633 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,635 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:01,638 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,661 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:01,662 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) uptime 0.534 seconds
2016-08-25 09:02:01,662 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:83) is shutdown
Wait for master to start up...
2016-08-25 09:02:01,665 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:01,666 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@51698ab9
2016-08-25 09:02:01,666 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:01,667 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:01,668 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:01,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55574
2016-08-25 09:02:01,668 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55574
2016-08-25 09:02:01,669 | INFO | SyncThread:0 | Established session 0x156bec917250055 with negotiated timeout 10000 for client /127.0.0.1:55574
2016-08-25 09:02:01,669 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250055, negotiated timeout = 10000
2016-08-25 09:02:01,707 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55575
2016-08-25 09:02:01,714 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,714 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:01,715 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55575
2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55575
2016-08-25 09:02:01,716 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:01,729 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,763 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:01,763 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000127.log
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:01,764 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000125
2016-08-25 09:02:01,765 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst
2016-08-25 09:02:01,765 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:01,769 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:01,769 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/7.98 kb and 2/10 files
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000126.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/7.98 kb and 3/10 files
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000125
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000127.log
2016-08-25 09:02:01,770 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:01,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/7.98 kb and 4/10 files
2016-08-25 09:02:01,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.61/7.98 kb and 5/10 files
2016-08-25 09:02:01,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.80/7.98 kb and 6/10 files
2016-08-25 09:02:01,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.70/7.98 kb and 7/10 files
2016-08-25 09:02:01,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.13/7.98 kb and 8/10 files
2016-08-25 09:02:01,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files
2016-08-25 09:02:01,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files
2016-08-25 09:02:01,774 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:01,775 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/7.98 kb and 1/10 files
2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/7.98 kb and 2/10 files
2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,776 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/7.98 kb and 3/10 files
2016-08-25 09:02:01,777 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.71/7.98 kb and 4/10 files
2016-08-25 09:02:01,777 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is starting
2016-08-25 09:02:01,777 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.61/7.98 kb and 5/10 files
2016-08-25 09:02:01,778 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.51/7.98 kb and 6/10 files
2016-08-25 09:02:01,778 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.95/7.98 kb and 7/10 files
2016-08-25 09:02:01,779 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.85/7.98 kb and 8/10 files
2016-08-25 09:02:01,779 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.03/7.98 kb and 9/10 files
2016-08-25 09:02:01,780 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.98/7.98 kb and 10/10 files
2016-08-25 09:02:01,780 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:01,781 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55580
2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) started
2016-08-25 09:02:01,782 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:01,782 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
2016-08-25 09:02:01,782 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30790 mb of usable space. - resetting to maximum available disk space: 30790 mb
Master started: node-3
======================================
82.4. Browse the queue on new master
======================================
2016-08-25 09:02:01,784 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
82.5. Found
======================================
======================================
83.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:01,799 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is shutting down
2016-08-25 09:02:01,800 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:01,805 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,806 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,806 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:01,806 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:01,806 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:01,806 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250052
2016-08-25 09:02:01,806 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:01,807 | INFO | Time-limited test | Session: 0x156bec917250052 closed
2016-08-25 09:02:01,807 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:01,807 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55551 which had sessionid 0x156bec917250052
2016-08-25 09:02:01,913 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:01,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) uptime 1.086 seconds
2016-08-25 09:02:01,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:84) is shutdown
Wait for master to start up...
2016-08-25 09:02:01,916 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:01,918 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3018e40b
2016-08-25 09:02:01,918 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:01,919 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:01,920 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:01,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55582
2016-08-25 09:02:01,920 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55582
2016-08-25 09:02:01,921 | INFO | SyncThread:0 | Established session 0x156bec917250056 with negotiated timeout 10000 for client /127.0.0.1:55582
2016-08-25 09:02:01,921 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250056, negotiated timeout = 10000
2016-08-25 09:02:01,927 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,965 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:01,966 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:01,967 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,968 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:01,971 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:01,975 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,036 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55583
2016-08-25 09:02:02,041 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,041 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,042 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55583
2016-08-25 09:02:02,042 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:02,043 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55583
2016-08-25 09:02:02,043 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:02,100 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:02,103 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:02,104 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000129.sst
2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000130.log
2016-08-25 09:02:02,105 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000128
2016-08-25 09:02:02,108 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000130.log
2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000129.sst
2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:02,110 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000128
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:02,111 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:02,112 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/8.93 kb and 1/11 files
2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/8.93 kb and 2/11 files
2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.74/8.93 kb and 3/11 files
2016-08-25 09:02:02,113 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.64/8.93 kb and 4/11 files
2016-08-25 09:02:02,114 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/8.93 kb and 5/11 files
2016-08-25 09:02:02,114 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.44/8.93 kb and 6/11 files
2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.34/8.93 kb and 7/11 files
2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.35/8.93 kb and 8/11 files
2016-08-25 09:02:02,115 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.25/8.93 kb and 9/11 files
2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.44/8.93 kb and 10/11 files
2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files
2016-08-25 09:02:02,116 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:02,117 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.18/8.93 kb and 1/11 files
2016-08-25 09:02:02,118 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.08/8.93 kb and 2/11 files
2016-08-25 09:02:02,118 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/8.93 kb and 3/11 files
2016-08-25 09:02:02,119 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.88/8.93 kb and 4/11 files
2016-08-25 09:02:02,119 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:02,120 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.78/8.93 kb and 5/11 files
2016-08-25 09:02:02,120 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.72/8.93 kb and 6/11 files
2016-08-25 09:02:02,121 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is starting
2016-08-25 09:02:02,121 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.62/8.93 kb and 7/11 files
2016-08-25 09:02:02,122 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 7.52/8.93 kb and 8/11 files
2016-08-25 09:02:02,122 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.01/8.93 kb and 9/11 files
2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.03/8.93 kb and 10/11 files
2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files
2016-08-25 09:02:02,123 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55588
2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) started
2016-08-25 09:02:02,125 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:02,125 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:02,126 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
83.4. Browse the queue on new master
======================================
2016-08-25 09:02:02,128 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
83.5. Found
======================================
======================================
84.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:02,147 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is shutting down
2016-08-25 09:02:02,148 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:02,151 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,153 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:02,154 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250054
2016-08-25 09:02:02,155 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,155 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:02,155 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,155 | INFO | Time-limited test | Session: 0x156bec917250054 closed
2016-08-25 09:02:02,155 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:02,155 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:02,155 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55566 which had sessionid 0x156bec917250054
2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:02,266 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,268 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,271 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:02,275 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,277 | INFO | ActiveMQ BrokerService[node-1] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:02,277 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) uptime 0.949 seconds
2016-08-25 09:02:02,277 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:85) is shutdown
Wait for master to start up...
2016-08-25 09:02:02,279 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:02,280 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@c30944a
2016-08-25 09:02:02,280 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:02,281 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:02,281 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:02,281 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55590
2016-08-25 09:02:02,281 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55590
2016-08-25 09:02:02,282 | INFO | SyncThread:0 | Established session 0x156bec917250057 with negotiated timeout 10000 for client /127.0.0.1:55590
2016-08-25 09:02:02,282 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250057, negotiated timeout = 10000
2016-08-25 09:02:02,334 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55591
2016-08-25 09:02:02,337 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,337 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55591
2016-08-25 09:02:02,338 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:02,339 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55591
2016-08-25 09:02:02,339 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:02,352 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,393 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:02,394 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000129.sst
2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst
2016-08-25 09:02:02,395 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000130.log
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000128
2016-08-25 09:02:02,396 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:02,397 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000128
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000129.sst
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000130.log
2016-08-25 09:02:02,398 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000126.sst
2016-08-25 09:02:02,401 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.93 kb and 1/11 files
2016-08-25 09:02:02,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/8.93 kb and 2/11 files
2016-08-25 09:02:02,402 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/8.93 kb and 3/11 files
2016-08-25 09:02:02,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/8.93 kb and 4/11 files
2016-08-25 09:02:02,403 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/8.93 kb and 5/11 files
2016-08-25 09:02:02,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/8.93 kb and 6/11 files
2016-08-25 09:02:02,404 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.30/8.93 kb and 7/11 files
2016-08-25 09:02:02,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.48/8.93 kb and 8/11 files
2016-08-25 09:02:02,405 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.50/8.93 kb and 9/11 files
2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.99/8.93 kb and 10/11 files
2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.93 kb and 1/11 files
2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files
2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.80/8.93 kb and 2/11 files
2016-08-25 09:02:02,406 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:02,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.70/8.93 kb and 3/11 files
2016-08-25 09:02:02,407 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.60/8.93 kb and 4/11 files
2016-08-25 09:02:02,408 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/8.93 kb and 5/11 files
2016-08-25 09:02:02,409 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,409 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.44/8.93 kb and 6/11 files
2016-08-25 09:02:02,410 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.93/8.93 kb and 7/11 files
2016-08-25 09:02:02,410 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is starting
2016-08-25 09:02:02,411 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/8.93 kb and 8/11 files
2016-08-25 09:02:02,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.84/8.93 kb and 9/11 files
2016-08-25 09:02:02,412 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.03/8.93 kb and 10/11 files
2016-08-25 09:02:02,413 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.93/8.93 kb and 11/11 files
2016-08-25 09:02:02,413 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55596
2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) started
2016-08-25 09:02:02,414 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:02,414 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:02,415 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
84.4. Browse the queue on new master
======================================
2016-08-25 09:02:02,418 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
84.5. Found
======================================
======================================
85.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:02,432 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is shutting down
2016-08-25 09:02:02,433 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:02,436 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,437 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,437 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,437 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:02,437 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,437 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:02,438 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250055
2016-08-25 09:02:02,439 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55574 which had sessionid 0x156bec917250055
2016-08-25 09:02:02,439 | INFO | Time-limited test | Session: 0x156bec917250055 closed
2016-08-25 09:02:02,439 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:02,546 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:02,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) uptime 0.884 seconds
2016-08-25 09:02:02,546 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:86) is shutdown
Wait for master to start up...
2016-08-25 09:02:02,548 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:02,556 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@645abf3c
2016-08-25 09:02:02,556 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:02,557 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:02,558 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:02,558 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55598
2016-08-25 09:02:02,558 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55598
2016-08-25 09:02:02,559 | INFO | SyncThread:0 | Established session 0x156bec917250058 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55598
2016-08-25 09:02:02,559 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250058, negotiated timeout = 10000
2016-08-25 09:02:02,568 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,623 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:02,625 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,632 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:02,632 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,636 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:02,639 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,702 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55599
2016-08-25 09:02:02,705 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,705 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55599
2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:02,707 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55599
2016-08-25 09:02:02,708 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:02,769 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,769 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:02,774 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,775 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000129.sst
2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000132.sst
2016-08-25 09:02:02,776 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000131
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000133.log
2016-08-25 09:02:02,777 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:02,778 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000123.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000129.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000120.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000126.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000117.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000132.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000131
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000111.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000133.log
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000114.sst
2016-08-25 09:02:02,779 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000109.sst
2016-08-25 09:02:02,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.94/9.88 kb and 1/12 files
2016-08-25 09:02:02,783 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/9.88 kb and 2/12 files
2016-08-25 09:02:02,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.74/9.88 kb and 3/12 files
2016-08-25 09:02:02,784 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.64/9.88 kb and 4/12 files
2016-08-25 09:02:02,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.54/9.88 kb and 5/12 files
2016-08-25 09:02:02,785 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.55/9.88 kb and 6/12 files
2016-08-25 09:02:02,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.45/9.88 kb and 7/12 files
2016-08-25 09:02:02,786 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.00/9.88 kb and 8/12 files
2016-08-25 09:02:02,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.89/9.88 kb and 9/12 files
2016-08-25 09:02:02,787 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.79/9.88 kb and 10/12 files
2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.98/9.88 kb and 11/12 files
2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files
2016-08-25 09:02:02,788 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:02,789 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.88 kb and 1/12 files
2016-08-25 09:02:02,789 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.80/9.88 kb and 2/12 files
2016-08-25 09:02:02,791 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.70/9.88 kb and 3/12 files
2016-08-25 09:02:02,791 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.60/9.88 kb and 4/12 files
2016-08-25 09:02:02,792 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.50/9.88 kb and 5/12 files
2016-08-25 09:02:02,792 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.40/9.88 kb and 6/12 files
2016-08-25 09:02:02,793 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,793 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.94/9.88 kb and 7/12 files
2016-08-25 09:02:02,794 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.95/9.88 kb and 8/12 files
2016-08-25 09:02:02,794 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is starting
2016-08-25 09:02:02,794 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.85/9.88 kb and 9/12 files
2016-08-25 09:02:02,796 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.04/9.88 kb and 10/12 files
2016-08-25 09:02:02,797 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.94/9.88 kb and 11/12 files
2016-08-25 09:02:02,798 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files
2016-08-25 09:02:02,798 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:02,798 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55604
2016-08-25 09:02:02,798 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:02,799 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) started
2016-08-25 09:02:02,799 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:02,799 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:02,799 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
85.4. Browse the queue on new master
======================================
2016-08-25 09:02:02,803 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
85.5. Found
======================================
======================================
86.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:02,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is shutting down
2016-08-25 09:02:02,819 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:02,823 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,825 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:02,825 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250056
2016-08-25 09:02:02,826 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55582 which had sessionid 0x156bec917250056
2016-08-25 09:02:02,826 | INFO | Time-limited test | Session: 0x156bec917250056 closed
2016-08-25 09:02:02,826 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:02,829 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,829 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:02,829 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:02,829 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:02,940 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:02,941 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) uptime 1.027 seconds
2016-08-25 09:02:02,941 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:87) is shutdown
Wait for master to start up...
2016-08-25 09:02:02,943 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:02,947 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@322c2ddd
2016-08-25 09:02:02,947 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:02,948 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:02,949 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:02,949 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55606
2016-08-25 09:02:02,949 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55606
2016-08-25 09:02:02,950 | INFO | SyncThread:0 | Established session 0x156bec917250059 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55606
2016-08-25 09:02:02,950 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250059, negotiated timeout = 10000
2016-08-25 09:02:02,960 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,969 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:02,969 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,972 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:02,979 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:02,982 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:02,985 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,054 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55607
2016-08-25 09:02:03,061 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,061 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55607
2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55607
2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:03,062 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:03,073 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,076 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,077 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000131
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000120.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000133.log
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000109.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000114.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000117.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000132.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000111.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000123.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000126.sst
2016-08-25 09:02:03,078 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000129.sst
2016-08-25 09:02:03,086 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.54/9.88 kb and 1/12 files
2016-08-25 09:02:03,087 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.44/9.88 kb and 2/12 files
2016-08-25 09:02:03,087 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.63/9.88 kb and 3/12 files
2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.57/9.88 kb and 4/12 files
2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.47/9.88 kb and 5/12 files
2016-08-25 09:02:03,088 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.48/9.88 kb and 6/12 files
2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.38/9.88 kb and 7/12 files
2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.28/9.88 kb and 8/12 files
2016-08-25 09:02:03,089 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.18/9.88 kb and 9/12 files
2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.08/9.88 kb and 10/12 files
2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.98/9.88 kb and 11/12 files
2016-08-25 09:02:03,090 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.88/9.88 kb and 12/12 files
2016-08-25 09:02:03,091 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:03,093 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,094 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is starting
2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55611
2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) started
2016-08-25 09:02:03,098 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,098 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:03,098 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
86.4. Browse the queue on new master
======================================
======================================
86.5. Found
======================================
======================================
87.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:03,114 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is shutting down
2016-08-25 09:02:03,114 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:03,117 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,118 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,118 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,119 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:03,119 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250057
2016-08-25 09:02:03,120 | INFO | Time-limited test | Session: 0x156bec917250057 closed
2016-08-25 09:02:03,120 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55590 which had sessionid 0x156bec917250057
2016-08-25 09:02:03,120 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:03,138 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,198 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:03,198 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,200 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,228 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:03,228 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) uptime 0.951 seconds
2016-08-25 09:02:03,228 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:88) is shutdown
Wait for master to start up...
2016-08-25 09:02:03,230 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:03,232 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1d3e3a05
2016-08-25 09:02:03,232 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:03,233 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:03,233 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:03,234 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55613
2016-08-25 09:02:03,234 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55613
2016-08-25 09:02:03,234 | INFO | SyncThread:0 | Established session 0x156bec91725005a with negotiated timeout 10000 for client /127.0.0.1:55613
2016-08-25 09:02:03,235 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005a, negotiated timeout = 10000
2016-08-25 09:02:03,241 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,245 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:03,249 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:03,253 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,276 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55614
2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55614
2016-08-25 09:02:03,280 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:03,281 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55614
2016-08-25 09:02:03,281 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:03,347 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,347 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000136.log
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000136.log
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000134
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000134
2016-08-25 09:02:03,350 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:03,352 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/2.86 kb and 1/4 files
2016-08-25 09:02:03,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/2.86 kb and 1/4 files
2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/2.86 kb and 2/4 files
2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/2.86 kb and 2/4 files
2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.85/2.86 kb and 3/4 files
2016-08-25 09:02:03,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.14/2.86 kb and 3/4 files
2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files
2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files
2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:03,354 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:03,358 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,358 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,359 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is starting
2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55619
2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,363 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) started
2016-08-25 09:02:03,364 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,364 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:03,364 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
87.4. Browse the queue on new master
======================================
======================================
87.5. Found
======================================
======================================
88.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:03,383 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is shutting down
2016-08-25 09:02:03,384 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:03,388 | INFO | Starting broker node: node-2-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:02:03,388 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,390 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,390 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,391 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,391 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,393 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:03,394 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:03,394 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,394 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:03,394 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250059
2016-08-25 09:02:03,395 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,395 | INFO | Time-limited test | Session: 0x156bec917250059 closed
2016-08-25 09:02:03,395 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55606 which had sessionid 0x156bec917250059
2016-08-25 09:02:03,395 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:03,397 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,401 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:03,404 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,425 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55621
2016-08-25 09:02:03,430 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,431 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55621
2016-08-25 09:02:03,432 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:03,444 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:03,446 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000134
2016-08-25 09:02:03,447 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000136.log
2016-08-25 09:02:03,447 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:03,449 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/2.86 kb and 1/4 files
2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.74/2.86 kb and 2/4 files
2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.92/2.86 kb and 3/4 files
2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.86/2.86 kb and 4/4 files
2016-08-25 09:02:03,450 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:03,453 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,454 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is starting
2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55624
2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) started
2016-08-25 09:02:03,457 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,457 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,457 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,514 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:03,514 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) uptime 0.573 seconds
2016-08-25 09:02:03,514 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:89) is shutdown
Wait for master to start up...
Master started: node-2
======================================
88.4. Browse the queue on new master
======================================
2016-08-25 09:02:03,516 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:03,518 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3ab65057
2016-08-25 09:02:03,518 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:03,518 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:03,519 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:03,519 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55626
2016-08-25 09:02:03,519 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55626
2016-08-25 09:02:03,519 | INFO | SyncThread:0 | Established session 0x156bec91725005b with negotiated timeout 10000 for client /127.0.0.1:55626
2016-08-25 09:02:03,520 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005b, negotiated timeout = 10000
======================================
88.5. Found
======================================
======================================
89.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:03,527 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is shutting down
2016-08-25 09:02:03,527 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:03,528 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,530 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55621
2016-08-25 09:02:03,530 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:03,530 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,531 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,531 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,531 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:03,531 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,534 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:03,534 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,534 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250058
2016-08-25 09:02:03,534 | INFO | Time-limited test | Session: 0x156bec917250058 closed
2016-08-25 09:02:03,535 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:03,535 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55598 which had sessionid 0x156bec917250058
2016-08-25 09:02:03,547 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:03,547 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,550 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:03,553 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,581 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55628
2016-08-25 09:02:03,584 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,585 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55628
2016-08-25 09:02:03,585 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:03,602 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,604 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000138
2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000140.log
2016-08-25 09:02:03,605 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:03,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/3.22 kb and 1/5 files
2016-08-25 09:02:03,608 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/3.22 kb and 2/5 files
2016-08-25 09:02:03,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.02/3.22 kb and 3/5 files
2016-08-25 09:02:03,609 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.20/3.22 kb and 4/5 files
2016-08-25 09:02:03,610 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files
2016-08-25 09:02:03,610 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:03,612 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,612 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is starting
2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55631
2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) started
2016-08-25 09:02:03,615 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,615 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,616 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,636 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:03,637 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) uptime 1.091 seconds
2016-08-25 09:02:03,637 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:90) is shutdown
Wait for master to start up...
Master started: node-1
======================================
89.4. Browse the queue on new master
======================================
2016-08-25 09:02:03,639 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:03,642 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@47a034e7
2016-08-25 09:02:03,642 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:03,643 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:03,644 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:03,644 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55633
2016-08-25 09:02:03,645 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55633
2016-08-25 09:02:03,645 | INFO | SyncThread:0 | Established session 0x156bec91725005c with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55633
2016-08-25 09:02:03,645 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725005c, negotiated timeout = 10000
======================================
89.5. Found
======================================
======================================
90.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:03,652 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is shutting down
2016-08-25 09:02:03,652 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:03,656 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,661 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55628
2016-08-25 09:02:03,661 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:03,662 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:03,662 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,663 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,663 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:03,663 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005a
2016-08-25 09:02:03,664 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,664 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,664 | INFO | Time-limited test | Session: 0x156bec91725005a closed
2016-08-25 09:02:03,664 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:03,665 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55613 which had sessionid 0x156bec91725005a
2016-08-25 09:02:03,666 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:03,666 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,669 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:03,673 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,700 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55635
2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55635
2016-08-25 09:02:03,703 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:03,714 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,717 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,718 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:03,718 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000140.log
2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000138
2016-08-25 09:02:03,719 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:03,721 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/3.22 kb and 1/5 files
2016-08-25 09:02:03,721 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.12/3.22 kb and 2/5 files
2016-08-25 09:02:03,722 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.14/3.22 kb and 3/5 files
2016-08-25 09:02:03,722 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.32/3.22 kb and 4/5 files
2016-08-25 09:02:03,723 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.22/3.22 kb and 5/5 files
2016-08-25 09:02:03,723 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:03,725 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,727 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is starting
2016-08-25 09:02:03,731 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55638
2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) started
2016-08-25 09:02:03,732 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,732 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,732 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:03,792 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:03,792 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) uptime 0.563 seconds
2016-08-25 09:02:03,793 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:91) is shutdown
Wait for master to start up...
Master started: node-3
======================================
90.4. Browse the queue on new master
======================================
2016-08-25 09:02:03,794 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:03,796 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@7371ece
2016-08-25 09:02:03,796 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:03,797 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:03,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:03,798 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55640
2016-08-25 09:02:03,798 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55640
2016-08-25 09:02:03,798 | INFO | SyncThread:0 | Established session 0x156bec91725005d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55640
2016-08-25 09:02:03,798 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725005d, negotiated timeout = 10000
2016-08-25 09:02:03,805 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
======================================
90.5. Found
======================================
======================================
91.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:03,806 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is shutting down
2016-08-25 09:02:03,806 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:03,806 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55635
2016-08-25 09:02:03,807 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:03,811 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,816 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:03,816 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005b
2016-08-25 09:02:03,817 | INFO | Time-limited test | Session: 0x156bec91725005b closed
2016-08-25 09:02:03,817 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55626 which had sessionid 0x156bec91725005b
2016-08-25 09:02:03,817 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:03,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:03,837 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,839 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:03,840 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,859 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:03,860 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:03,860 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:03,863 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:03,866 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,910 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Master started: tcp://localhost:55642
2016-08-25 09:02:03,913 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,914 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55642
2016-08-25 09:02:03,914 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:03,920 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:03,921 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) uptime 0.406 seconds
2016-08-25 09:02:03,921 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:92) is shutdown
Wait for master to start up...
2016-08-25 09:02:03,923 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:03,925 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@6488a31b
2016-08-25 09:02:03,926 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:03,926 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:03,927 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:03,927 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55644
2016-08-25 09:02:03,927 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55644
2016-08-25 09:02:03,928 | INFO | SyncThread:0 | Established session 0x156bec91725005e with negotiated timeout 10000 for client /127.0.0.1:55644
2016-08-25 09:02:03,928 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005e, negotiated timeout = 10000
2016-08-25 09:02:03,941 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:03,942 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55642
2016-08-25 09:02:03,942 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:03,947 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,961 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,962 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000143.log
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000141
2016-08-25 09:02:03,963 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:03,967 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000141
2016-08-25 09:02:03,968 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000143.log
2016-08-25 09:02:03,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/4.17 kb and 1/6 files
2016-08-25 09:02:03,969 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/4.17 kb and 2/6 files
2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.02/4.17 kb and 3/6 files
2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.04/4.17 kb and 4/6 files
2016-08-25 09:02:03,970 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.27/4.17 kb and 5/6 files
2016-08-25 09:02:03,971 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files
2016-08-25 09:02:03,971 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:03,972 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/4.17 kb and 1/6 files
2016-08-25 09:02:03,972 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.91/4.17 kb and 2/6 files
2016-08-25 09:02:03,973 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.81/4.17 kb and 3/6 files
2016-08-25 09:02:03,973 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.75/4.17 kb and 4/6 files
2016-08-25 09:02:03,974 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.98/4.17 kb and 5/6 files
2016-08-25 09:02:03,974 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:03,975 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files
2016-08-25 09:02:03,975 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:03,976 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is starting
2016-08-25 09:02:03,979 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:03,981 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55648
2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) started
2016-08-25 09:02:03,982 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:03,982 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:03,982 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
91.4. Browse the queue on new master
======================================
======================================
91.5. Found
======================================
======================================
92.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:04,002 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is shutting down
2016-08-25 09:02:04,003 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:04,009 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,014 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,015 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,015 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,016 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,020 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:04,021 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005c
2016-08-25 09:02:04,021 | INFO | Time-limited test | Session: 0x156bec91725005c closed
2016-08-25 09:02:04,021 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:04,021 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55633 which had sessionid 0x156bec91725005c
2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:04,050 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,052 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,053 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:04,055 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,100 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55650
2016-08-25 09:02:04,103 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,104 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55650
2016-08-25 09:02:04,104 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:04,131 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:04,134 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000141
2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000143.log
2016-08-25 09:02:04,135 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:04,140 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/4.17 kb and 1/6 files
2016-08-25 09:02:04,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/4.17 kb and 2/6 files
2016-08-25 09:02:04,141 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.85/4.17 kb and 3/6 files
2016-08-25 09:02:04,142 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.09/4.17 kb and 4/6 files
2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.27/4.17 kb and 5/6 files
2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.17/4.17 kb and 6/6 files
2016-08-25 09:02:04,143 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:04,146 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,147 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is starting
2016-08-25 09:02:04,150 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55653
2016-08-25 09:02:04,150 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:04,151 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) started
2016-08-25 09:02:04,151 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:04,151 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:02:04,151 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30787 mb of usable space. - resetting to maximum available disk space: 30787 mb
2016-08-25 09:02:04,160 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:04,161 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) uptime 0.523 seconds
2016-08-25 09:02:04,161 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:93) is shutdown
Wait for master to start up...
Master started: node-1
======================================
92.4. Browse the queue on new master
======================================
2016-08-25 09:02:04,163 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:04,166 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@609f293a
2016-08-25 09:02:04,166 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:04,166 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:04,167 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:04,167 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55655
2016-08-25 09:02:04,167 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55655
2016-08-25 09:02:04,168 | INFO | SyncThread:0 | Established session 0x156bec91725005f with negotiated timeout 10000 for client /127.0.0.1:55655
2016-08-25 09:02:04,168 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725005f, negotiated timeout = 10000
======================================
92.5. Found
======================================
======================================
93.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:04,177 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is shutting down
2016-08-25 09:02:04,177 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:04,179 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,180 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:04,180 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005d
2016-08-25 09:02:04,181 | INFO | Time-limited test | Session: 0x156bec91725005d closed
2016-08-25 09:02:04,181 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:04,181 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55640 which had sessionid 0x156bec91725005d
2016-08-25 09:02:04,181 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,181 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,187 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55650
2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:04,189 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave stopped
2016-08-25 09:02:04,191 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:04,237 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,239 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:04,243 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,283 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55657
2016-08-25 09:02:04,284 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:04,285 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) uptime 0.492 seconds
2016-08-25 09:02:04,285 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:94) is shutdown
Wait for master to start up...
2016-08-25 09:02:04,287 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:04,290 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@20bb0006
2016-08-25 09:02:04,290 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:04,290 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,290 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Attaching to master: tcp://localhost:55657
2016-08-25 09:02:04,291 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Slave started
2016-08-25 09:02:04,291 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:04,291 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:04,292 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55659
2016-08-25 09:02:04,292 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55659
2016-08-25 09:02:04,293 | INFO | SyncThread:0 | Established session 0x156bec917250060 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55659
2016-08-25 09:02:04,293 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250060, negotiated timeout = 10000
2016-08-25 09:02:04,302 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,303 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55657
2016-08-25 09:02:04,303 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:04,333 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,337 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000146.log
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000144
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:04,338 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:04,344 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files
2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.20/5.12 kb and 2/7 files
2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:04,345 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.49/5.12 kb and 3/7 files
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000146.log
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.38/5.12 kb and 4/7 files
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000144
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:04,346 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:04,347 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.28/5.12 kb and 5/7 files
2016-08-25 09:02:04,347 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.18/5.12 kb and 6/7 files
2016-08-25 09:02:04,348 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files
2016-08-25 09:02:04,348 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/5.12 kb and 1/7 files
2016-08-25 09:02:04,351 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/5.12 kb and 2/7 files
2016-08-25 09:02:04,352 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is starting
2016-08-25 09:02:04,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.10/5.12 kb and 3/7 files
2016-08-25 09:02:04,352 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/5.12 kb and 4/7 files
2016-08-25 09:02:04,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.28/5.12 kb and 5/7 files
2016-08-25 09:02:04,353 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.18/5.12 kb and 6/7 files
2016-08-25 09:02:04,354 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files
2016-08-25 09:02:04,354 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:04,355 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55663
2016-08-25 09:02:04,355 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:04,356 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) started
2016-08-25 09:02:04,356 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:04,356 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:04,356 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:04,357 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
Master started: node-3
======================================
93.4. Browse the queue on new master
======================================
======================================
93.5. Found
======================================
======================================
94.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:04,371 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is shutting down
2016-08-25 09:02:04,371 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:04,374 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,376 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:04,377 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,377 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,377 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,377 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,377 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005e
2016-08-25 09:02:04,377 | INFO | Time-limited test | Session: 0x156bec91725005e closed
2016-08-25 09:02:04,377 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:04,377 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55644 which had sessionid 0x156bec91725005e
2016-08-25 09:02:04,440 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:04,443 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,444 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:04,444 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,449 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:04,455 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,489 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:04,489 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) uptime 0.567 seconds
2016-08-25 09:02:04,489 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:95) is shutdown
Wait for master to start up...
2016-08-25 09:02:04,491 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:04,493 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@61c28cfb
2016-08-25 09:02:04,493 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:04,494 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:04,494 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:04,494 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55665
2016-08-25 09:02:04,494 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55665
2016-08-25 09:02:04,495 | INFO | SyncThread:0 | Established session 0x156bec917250061 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55665
2016-08-25 09:02:04,495 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250061, negotiated timeout = 10000
2016-08-25 09:02:04,498 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Master started: tcp://localhost:55666
2016-08-25 09:02:04,505 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,505 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55666
2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55666
2016-08-25 09:02:04,507 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:04,526 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:04,531 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000144
2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:04,532 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000146.log
2016-08-25 09:02:04,537 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/5.12 kb and 1/7 files
2016-08-25 09:02:04,537 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.79/5.12 kb and 2/7 files
2016-08-25 09:02:04,538 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.08/5.12 kb and 3/7 files
2016-08-25 09:02:04,538 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.02/5.12 kb and 4/7 files
2016-08-25 09:02:04,539 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.92/5.12 kb and 5/7 files
2016-08-25 09:02:04,539 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.93/5.12 kb and 6/7 files
2016-08-25 09:02:04,540 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.12/5.12 kb and 7/7 files
2016-08-25 09:02:04,540 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:04,542 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,543 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is starting
2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55670
2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) started
2016-08-25 09:02:04,547 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:04,548 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:04,548 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
94.4. Browse the queue on new master
======================================
2016-08-25 09:02:04,558 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
94.5. Found
======================================
======================================
95.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:04,563 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is shutting down
2016-08-25 09:02:04,563 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:04,565 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,566 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:04,566 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725005f
2016-08-25 09:02:04,567 | INFO | Time-limited test | Session: 0x156bec91725005f closed
2016-08-25 09:02:04,567 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55655 which had sessionid 0x156bec91725005f
2016-08-25 09:02:04,567 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:04,567 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000144
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:04,568 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000146.log
2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,571 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,571 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,572 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,609 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:04,609 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,611 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,666 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:04,669 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:04,671 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:04,672 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) uptime 0.511 seconds
2016-08-25 09:02:04,672 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:96) is shutdown
Wait for master to start up...
2016-08-25 09:02:04,673 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,674 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:04,676 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@62399a23
2016-08-25 09:02:04,676 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:04,677 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:04,677 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:04,677 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55673
2016-08-25 09:02:04,677 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55673
2016-08-25 09:02:04,678 | INFO | SyncThread:0 | Established session 0x156bec917250062 with negotiated timeout 10000 for client /127.0.0.1:55673
2016-08-25 09:02:04,678 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250062, negotiated timeout = 10000
2016-08-25 09:02:04,725 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55674
2016-08-25 09:02:04,728 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,728 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,729 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55674
2016-08-25 09:02:04,729 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:04,730 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55674
2016-08-25 09:02:04,730 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:04,763 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,764 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,766 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,766 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000149.log
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000147
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000147
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000149.log
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:04,767 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:04,770 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/6.07 kb and 1/8 files
2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files
2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/6.07 kb and 2/8 files
2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.24/6.07 kb and 2/8 files
2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.25/6.07 kb and 3/8 files
2016-08-25 09:02:04,771 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/6.07 kb and 3/8 files
2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.15/6.07 kb and 4/8 files
2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.92/6.07 kb and 4/8 files
2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 3.09/6.07 kb and 5/8 files
2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.26/6.07 kb and 5/8 files
2016-08-25 09:02:04,772 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 4.27/6.07 kb and 6/8 files
2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.15/6.07 kb and 6/8 files
2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files
2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.05/6.07 kb and 7/8 files
2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:04,773 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:04,774 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:04,774 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:04,776 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,776 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,777 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is starting
2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55679
2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) started
2016-08-25 09:02:04,781 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:04,781 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:04,781 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
95.4. Browse the queue on new master
======================================
======================================
95.5. Found
======================================
======================================
96.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:04,796 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is shutting down
2016-08-25 09:02:04,796 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:04,800 | INFO | Starting broker node: node-1-EventThread | Previously elected master is not online, staring new election
2016-08-25 09:02:04,800 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,801 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,802 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,802 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:04,803 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:04,803 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:04,803 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250061
2016-08-25 09:02:04,804 | INFO | Time-limited test | Session: 0x156bec917250061 closed
2016-08-25 09:02:04,804 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:04,804 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55665 which had sessionid 0x156bec917250061
2016-08-25 09:02:04,805 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:04,892 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,895 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:04,898 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:04,901 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,913 | INFO | ActiveMQ BrokerService[node-3] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:04,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) uptime 0.423 seconds
2016-08-25 09:02:04,913 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:97) is shutdown
Wait for master to start up...
2016-08-25 09:02:04,914 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:04,916 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3cf21ac1
2016-08-25 09:02:04,916 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:04,917 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:04,917 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:04,917 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55681
2016-08-25 09:02:04,917 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55681
2016-08-25 09:02:04,918 | INFO | SyncThread:0 | Established session 0x156bec917250063 with negotiated timeout 10000 for client /127.0.0.1:55681
2016-08-25 09:02:04,918 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250063, negotiated timeout = 10000
2016-08-25 09:02:04,950 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55682
2016-08-25 09:02:04,954 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,954 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:04,955 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55682
2016-08-25 09:02:04,955 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:04,956 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55682
2016-08-25 09:02:04,956 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:04,967 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:04,993 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000149.log
2016-08-25 09:02:04,994 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000147
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000149.log
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000147
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:04,996 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:04,997 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:04,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/6.07 kb and 1/8 files
2016-08-25 09:02:04,998 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.84/6.07 kb and 2/8 files
2016-08-25 09:02:04,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.73/6.07 kb and 3/8 files
2016-08-25 09:02:04,999 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.63/6.07 kb and 4/8 files
2016-08-25 09:02:05,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.53/6.07 kb and 5/8 files
2016-08-25 09:02:05,000 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.54/6.07 kb and 6/8 files
2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/6.07 kb and 7/8 files
2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.18/6.07 kb and 1/8 files
2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:05,001 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:05,002 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.20/6.07 kb and 2/8 files
2016-08-25 09:02:05,002 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.54/6.07 kb and 3/8 files
2016-08-25 09:02:05,003 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.48/6.07 kb and 4/8 files
2016-08-25 09:02:05,003 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.38/6.07 kb and 5/8 files
2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.27/6.07 kb and 6/8 files
2016-08-25 09:02:05,004 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.17/6.07 kb and 7/8 files
2016-08-25 09:02:05,005 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is starting
2016-08-25 09:02:05,005 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.07/6.07 kb and 8/8 files
2016-08-25 09:02:05,005 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:05,008 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55687
2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) started
2016-08-25 09:02:05,009 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:05,009 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:05,009 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
96.4. Browse the queue on new master
======================================
======================================
96.5. Found
======================================
======================================
97.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:05,025 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is shutting down
2016-08-25 09:02:05,025 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:05,028 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,029 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:05,029 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250060
2016-08-25 09:02:05,029 | INFO | Time-limited test | Session: 0x156bec917250060 closed
2016-08-25 09:02:05,029 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:05,029 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55659 which had sessionid 0x156bec917250060
2016-08-25 09:02:05,033 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,033 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,033 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,033 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,136 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:05,136 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) uptime 0.851 seconds
2016-08-25 09:02:05,137 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:98) is shutdown
Wait for master to start up...
2016-08-25 09:02:05,139 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,139 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:05,141 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,141 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@3407a533
2016-08-25 09:02:05,141 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:05,142 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:05,143 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:05,143 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55689
2016-08-25 09:02:05,143 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55689
2016-08-25 09:02:05,143 | INFO | SyncThread:0 | Established session 0x156bec917250064 with negotiated timeout 10000 for client /127.0.0.1:55689
2016-08-25 09:02:05,143 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250064, negotiated timeout = 10000
2016-08-25 09:02:05,144 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:05,147 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,201 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55690
2016-08-25 09:02:05,205 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,205 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55690
2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55690
2016-08-25 09:02:05,206 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:05,240 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,278 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,280 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:05,281 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000150
2016-08-25 09:02:05,282 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000152.log
2016-08-25 09:02:05,283 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,283 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000150
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000152.log
2016-08-25 09:02:05,284 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:05,287 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files
2016-08-25 09:02:05,287 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files
2016-08-25 09:02:05,288 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.84/7.02 kb and 2/9 files
2016-08-25 09:02:05,288 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/7.02 kb and 2/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.69/7.02 kb and 3/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.85/7.02 kb and 3/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.59/7.02 kb and 4/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.75/7.02 kb and 4/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.98/7.02 kb and 5/9 files
2016-08-25 09:02:05,289 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.65/7.02 kb and 5/9 files
2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.88/7.02 kb and 6/9 files
2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.54/7.02 kb and 6/9 files
2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.82/7.02 kb and 7/9 files
2016-08-25 09:02:05,290 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.44/7.02 kb and 7/9 files
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.00/7.02 kb and 8/9 files
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.83/7.02 kb and 8/9 files
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:05,291 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:05,294 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,294 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,295 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is starting
2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55695
2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) started
2016-08-25 09:02:05,299 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:05,299 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:05,299 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
97.4. Browse the queue on new master
======================================
======================================
97.5. Found
======================================
======================================
98.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:05,314 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is shutting down
2016-08-25 09:02:05,314 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:05,317 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,318 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,318 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,318 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,318 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,318 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:05,319 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250062
2016-08-25 09:02:05,319 | INFO | Time-limited test | Session: 0x156bec917250062 closed
2016-08-25 09:02:05,319 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55673 which had sessionid 0x156bec917250062
2016-08-25 09:02:05,319 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,415 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:05,417 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,420 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:05,424 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,436 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:05,437 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) uptime 0.765 seconds
2016-08-25 09:02:05,437 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:99) is shutdown
Wait for master to start up...
2016-08-25 09:02:05,440 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:05,442 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@227ec5dd
2016-08-25 09:02:05,442 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:05,443 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:05,443 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:05,443 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55697
2016-08-25 09:02:05,444 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55697
2016-08-25 09:02:05,446 | INFO | SyncThread:0 | Established session 0x156bec917250065 with negotiated timeout 10000 for client /127.0.0.1:55697
2016-08-25 09:02:05,446 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250065, negotiated timeout = 10000
2016-08-25 09:02:05,487 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Master started: tcp://localhost:55698
2016-08-25 09:02:05,492 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,492 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55698
2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:05,494 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55698
2016-08-25 09:02:05,495 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:05,510 | INFO | hawtdispatch-DEFAULT-3 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,540 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,542 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,542 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000150
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000152.log
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000150
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000152.log
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:05,543 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:05,544 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:05,547 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/7.02 kb and 1/9 files
2016-08-25 09:02:05,547 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/7.02 kb and 1/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.79/7.02 kb and 2/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.91/7.02 kb and 2/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.73/7.02 kb and 3/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.30/7.02 kb and 3/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.75/7.02 kb and 4/9 files
2016-08-25 09:02:05,548 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.20/7.02 kb and 4/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.10/7.02 kb and 5/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.93/7.02 kb and 5/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.00/7.02 kb and 6/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.32/7.02 kb and 6/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.18/7.02 kb and 7/9 files
2016-08-25 09:02:05,549 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.22/7.02 kb and 7/9 files
2016-08-25 09:02:05,550 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.08/7.02 kb and 8/9 files
2016-08-25 09:02:05,550 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.12/7.02 kb and 8/9 files
2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.02/7.02 kb and 9/9 files
2016-08-25 09:02:05,551 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:05,554 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,554 | INFO | hawtdispatch-DEFAULT-3 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,555 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is starting
2016-08-25 09:02:05,558 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55703
2016-08-25 09:02:05,558 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:05,559 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) started
2016-08-25 09:02:05,559 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:05,559 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:05,559 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
98.4. Browse the queue on new master
======================================
======================================
98.5. Found
======================================
======================================
99.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:05,574 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is shutting down
2016-08-25 09:02:05,575 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:05,577 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,578 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,578 | WARN | hawtdispatch-DEFAULT-3 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,578 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:05,578 | INFO | hawtdispatch-DEFAULT-3 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,579 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:05,579 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250063
2016-08-25 09:02:05,580 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55681 which had sessionid 0x156bec917250063
2016-08-25 09:02:05,580 | INFO | Time-limited test | Session: 0x156bec917250063 closed
2016-08-25 09:02:05,580 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:05,688 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:05,689 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) uptime 0.776 seconds
2016-08-25 09:02:05,689 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:100) is shutdown
Wait for master to start up...
2016-08-25 09:02:05,692 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:05,695 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@769bca67
2016-08-25 09:02:05,695 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:05,696 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:05,697 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:05,697 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55705
2016-08-25 09:02:05,698 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55705
2016-08-25 09:02:05,698 | INFO | SyncThread:0 | Established session 0x156bec917250066 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55705
2016-08-25 09:02:05,699 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250066, negotiated timeout = 10000
2016-08-25 09:02:05,708 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,720 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:05,721 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,726 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:05,726 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,729 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:05,733 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,797 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55706
2016-08-25 09:02:05,816 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:05,818 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55706
2016-08-25 09:02:05,818 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:05,819 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55706
2016-08-25 09:02:05,819 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:05,859 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,864 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,869 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:05,870 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000155.log
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000153
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:05,871 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:05,872 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:05,872 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000153
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000155.log
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:05,873 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:05,877 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/7.97 kb and 1/10 files
2016-08-25 09:02:05,877 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.84/7.97 kb and 2/10 files
2016-08-25 09:02:05,878 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.73/7.97 kb and 3/10 files
2016-08-25 09:02:05,879 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.63/7.97 kb and 4/10 files
2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.65/7.97 kb and 5/10 files
2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.44/7.97 kb and 1/10 files
2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/7.97 kb and 6/10 files
2016-08-25 09:02:05,880 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.63/7.97 kb and 2/10 files
2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.73/7.97 kb and 7/10 files
2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.52/7.97 kb and 3/10 files
2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.17/7.97 kb and 8/10 files
2016-08-25 09:02:05,881 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.42/7.97 kb and 4/10 files
2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files
2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files
2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files
2016-08-25 09:02:05,883 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:05,882 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files
2016-08-25 09:02:05,883 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.16/7.97 kb and 7/10 files
2016-08-25 09:02:05,884 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.05/7.97 kb and 8/10 files
2016-08-25 09:02:05,884 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files
2016-08-25 09:02:05,885 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files
2016-08-25 09:02:05,885 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:05,886 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:05,887 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is starting
2016-08-25 09:02:05,888 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,891 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55711
2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) started
2016-08-25 09:02:05,892 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:05,892 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:05,892 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
99.4. Browse the queue on new master
======================================
======================================
99.5. Found
======================================
======================================
100.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:05,910 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is shutting down
2016-08-25 09:02:05,911 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:05,914 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:05,914 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:05,914 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250064
2016-08-25 09:02:05,915 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,915 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55689 which had sessionid 0x156bec917250064
2016-08-25 09:02:05,915 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:05,915 | INFO | Time-limited test | Session: 0x156bec917250064 closed
2016-08-25 09:02:05,915 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:05,915 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:05,915 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,021 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:06,022 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:06,022 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,024 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,026 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:06,030 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,031 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:06,032 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) uptime 0.895 seconds
2016-08-25 09:02:06,032 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:101) is shutdown
Wait for master to start up...
2016-08-25 09:02:06,034 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:06,036 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@74ddb0f9
2016-08-25 09:02:06,036 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:06,037 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:06,038 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:06,038 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55713
2016-08-25 09:02:06,038 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55713
2016-08-25 09:02:06,039 | INFO | SyncThread:0 | Established session 0x156bec917250067 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55713
2016-08-25 09:02:06,039 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250067, negotiated timeout = 10000
2016-08-25 09:02:06,095 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55714
2016-08-25 09:02:06,100 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,100 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,101 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55714
2016-08-25 09:02:06,101 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:06,102 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55714
2016-08-25 09:02:06,102 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:06,116 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:06,154 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,155 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000155.log
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000153
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:06,156 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:06,157 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000153
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000155.log
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:06,158 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:06,162 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.18/7.97 kb and 1/10 files
2016-08-25 09:02:06,163 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.12/7.97 kb and 2/10 files
2016-08-25 09:02:06,164 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.02/7.97 kb and 3/10 files
2016-08-25 09:02:06,165 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.92/7.97 kb and 4/10 files
2016-08-25 09:02:06,165 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files
2016-08-25 09:02:06,166 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files
2016-08-25 09:02:06,167 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.16/7.97 kb and 7/10 files
2016-08-25 09:02:06,167 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.05/7.97 kb and 8/10 files
2016-08-25 09:02:06,168 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.44/7.97 kb and 1/10 files
2016-08-25 09:02:06,168 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files
2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.63/7.97 kb and 2/10 files
2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files
2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:06,169 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.52/7.97 kb and 3/10 files
2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.46/7.97 kb and 4/10 files
2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.36/7.97 kb and 5/10 files
2016-08-25 09:02:06,170 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.26/7.97 kb and 6/10 files
2016-08-25 09:02:06,171 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.27/7.97 kb and 7/10 files
2016-08-25 09:02:06,171 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:06,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.17/7.97 kb and 8/10 files
2016-08-25 09:02:06,172 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.07/7.97 kb and 9/10 files
2016-08-25 09:02:06,172 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is starting
2016-08-25 09:02:06,173 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.97/7.97 kb and 10/10 files
2016-08-25 09:02:06,173 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:06,176 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55719
2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) started
2016-08-25 09:02:06,177 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:06,177 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:06,177 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-2
======================================
100.4. Browse the queue on new master
======================================
======================================
100.5. Found
======================================
======================================
101.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:06,193 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is shutting down
2016-08-25 09:02:06,194 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:06,197 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,199 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,199 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,199 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:06,199 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,200 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:06,200 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250065
2016-08-25 09:02:06,201 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55697 which had sessionid 0x156bec917250065
2016-08-25 09:02:06,201 | INFO | Time-limited test | Session: 0x156bec917250065 closed
2016-08-25 09:02:06,201 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:06,308 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:06,308 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) uptime 0.871 seconds
2016-08-25 09:02:06,308 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:102) is shutdown
Wait for master to start up...
2016-08-25 09:02:06,310 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:06,313 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5180a0e5
2016-08-25 09:02:06,313 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:06,314 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:06,315 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:06,315 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55721
2016-08-25 09:02:06,315 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55721
2016-08-25 09:02:06,316 | INFO | SyncThread:0 | Established session 0x156bec917250068 with negotiated timeout 10000 for client /127.0.0.1:55721
2016-08-25 09:02:06,316 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec917250068, negotiated timeout = 10000
2016-08-25 09:02:06,336 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,397 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:06,400 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,402 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:06,402 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,406 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:06,411 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,481 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55722
2016-08-25 09:02:06,485 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,485 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,486 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55722
2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55722
2016-08-25 09:02:06,487 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:06,548 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000157.sst
2016-08-25 09:02:06,553 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000158.log
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000156
2016-08-25 09:02:06,554 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:06,559 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/8.92 kb and 1/11 files
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/8.92 kb and 2/11 files
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/8.92 kb and 3/11 files
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000157.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:06,560 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/8.92 kb and 4/11 files
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000156
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000158.log
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.77/8.92 kb and 5/11 files
2016-08-25 09:02:06,561 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.67/8.92 kb and 6/11 files
2016-08-25 09:02:06,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.57/8.92 kb and 7/11 files
2016-08-25 09:02:06,562 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.47/8.92 kb and 8/11 files
2016-08-25 09:02:06,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.41/8.92 kb and 9/11 files
2016-08-25 09:02:06,563 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.90/8.92 kb and 10/11 files
2016-08-25 09:02:06,564 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files
2016-08-25 09:02:06,564 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/8.92 kb and 1/11 files
2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.84/8.92 kb and 2/11 files
2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.73/8.92 kb and 3/11 files
2016-08-25 09:02:06,566 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:06,567 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.63/8.92 kb and 4/11 files
2016-08-25 09:02:06,567 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is starting
2016-08-25 09:02:06,567 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.53/8.92 kb and 5/11 files
2016-08-25 09:02:06,568 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.43/8.92 kb and 6/11 files
2016-08-25 09:02:06,568 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.32/8.92 kb and 7/11 files
2016-08-25 09:02:06,569 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.82/8.92 kb and 8/11 files
2016-08-25 09:02:06,569 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.83/8.92 kb and 9/11 files
2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.73/8.92 kb and 10/11 files
2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files
2016-08-25 09:02:06,570 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:06,571 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55727
2016-08-25 09:02:06,572 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:06,573 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) started
2016-08-25 09:02:06,573 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:06,573 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:06,573 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
101.4. Browse the queue on new master
======================================
2016-08-25 09:02:06,577 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
======================================
101.5. Found
======================================
======================================
102.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:06,596 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is shutting down
2016-08-25 09:02:06,597 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:06,599 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,599 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:06,599 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250066
2016-08-25 09:02:06,600 | INFO | Time-limited test | Session: 0x156bec917250066 closed
2016-08-25 09:02:06,600 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55705 which had sessionid 0x156bec917250066
2016-08-25 09:02:06,600 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:06,604 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,604 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,604 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:06,604 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:06,718 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:06,718 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) uptime 1.028 seconds
2016-08-25 09:02:06,719 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:103) is shutdown
Wait for master to start up...
2016-08-25 09:02:06,720 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:06,722 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1c18c548
2016-08-25 09:02:06,723 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:06,723 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:06,723 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:06,724 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:06,724 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55729
2016-08-25 09:02:06,724 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55729
2016-08-25 09:02:06,725 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:06,725 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,725 | INFO | SyncThread:0 | Established session 0x156bec917250069 with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55729
2016-08-25 09:02:06,725 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec917250069, negotiated timeout = 10000
2016-08-25 09:02:06,726 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,728 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:06,730 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,811 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Master started: tcp://localhost:55730
2016-08-25 09:02:06,815 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,815 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55730
2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55730
2016-08-25 09:02:06,817 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:06,835 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,882 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:06,883 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000158.log
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:06,884 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000157.sst
2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000156
2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:06,885 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:06,889 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:06,890 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000158.log
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000157.sst
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/MANIFEST-000156
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:06,891 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:06,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.02/8.92 kb and 1/11 files
2016-08-25 09:02:06,897 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.91/8.92 kb and 2/11 files
2016-08-25 09:02:06,898 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.10/8.92 kb and 3/11 files
2016-08-25 09:02:06,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.00/8.92 kb and 4/11 files
2016-08-25 09:02:06,899 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.93/8.92 kb and 5/11 files
2016-08-25 09:02:06,900 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.83/8.92 kb and 6/11 files
2016-08-25 09:02:06,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.33/8.92 kb and 7/11 files
2016-08-25 09:02:06,901 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.22/8.92 kb and 8/11 files
2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.12/8.92 kb and 9/11 files
2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.02/8.92 kb and 10/11 files
2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files
2016-08-25 09:02:06,902 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:06,903 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.02/8.92 kb and 1/11 files
2016-08-25 09:02:06,904 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.91/8.92 kb and 2/11 files
2016-08-25 09:02:06,904 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.81/8.92 kb and 3/11 files
2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.00/8.92 kb and 4/11 files
2016-08-25 09:02:06,905 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.89/8.92 kb and 5/11 files
2016-08-25 09:02:06,906 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is starting
2016-08-25 09:02:06,906 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.79/8.92 kb and 6/11 files
2016-08-25 09:02:06,906 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.69/8.92 kb and 7/11 files
2016-08-25 09:02:06,907 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.58/8.92 kb and 8/11 files
2016-08-25 09:02:06,908 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.08/8.92 kb and 9/11 files
2016-08-25 09:02:06,908 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.02/8.92 kb and 10/11 files
2016-08-25 09:02:06,909 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.92/8.92 kb and 11/11 files
2016-08-25 09:02:06,909 | INFO | hawtdispatch-DEFAULT-1 | Attached
2016-08-25 09:02:06,909 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55735
2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) started
2016-08-25 09:02:06,910 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:06,910 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:06,910 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
102.4. Browse the queue on new master
======================================
2016-08-25 09:02:06,913 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
======================================
102.5. Found
======================================
======================================
103.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:06,924 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is shutting down
2016-08-25 09:02:06,925 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:06,927 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:06,927 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,927 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:06,927 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:06,927 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250067
2016-08-25 09:02:06,928 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:06,928 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:06,928 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55713 which had sessionid 0x156bec917250067
2016-08-25 09:02:06,928 | INFO | Time-limited test | Session: 0x156bec917250067 closed
2016-08-25 09:02:06,928 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:07,036 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:07,036 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) uptime 1.004 seconds
2016-08-25 09:02:07,036 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:104) is shutdown
Wait for master to start up...
2016-08-25 09:02:07,039 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:07,041 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@4c0906ae
2016-08-25 09:02:07,041 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:07,042 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:07,043 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:07,043 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55737
2016-08-25 09:02:07,043 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55737
2016-08-25 09:02:07,044 | INFO | SyncThread:0 | Established session 0x156bec91725006a with negotiated timeout 10000 for client /127.0.0.1:55737
2016-08-25 09:02:07,045 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006a, negotiated timeout = 10000
2016-08-25 09:02:07,052 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,125 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:07,129 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,134 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:07,134 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,137 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:07,141 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,234 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55738
2016-08-25 09:02:07,238 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,238 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,239 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55738
2016-08-25 09:02:07,239 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:07,240 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55738
2016-08-25 09:02:07,240 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:07,304 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,315 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,315 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000157.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/MANIFEST-000159
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000161.log
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:07,316 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 0000000000000000.index/000160.sst
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 1.79/9.86 kb and 2/12 files
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000145.sst
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.69/9.86 kb and 3/12 files
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000151.sst
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/MANIFEST-000159
2016-08-25 09:02:07,322 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000139.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000148.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000142.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.59/9.86 kb and 4/12 files
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000154.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000157.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000137.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000160.sst
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-1 | Slave requested: 0000000000000000.index/000161.log
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.49/9.86 kb and 5/12 files
2016-08-25 09:02:07,323 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.03/9.86 kb and 6/12 files
2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.93/9.86 kb and 7/12 files
2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.87/9.86 kb and 8/12 files
2016-08-25 09:02:07,324 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.05/9.86 kb and 9/12 files
2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.95/9.86 kb and 10/12 files
2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files
2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files
2016-08-25 09:02:07,325 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:07,328 | INFO | hawtdispatch-DEFAULT-4 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,329 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is starting
2016-08-25 09:02:07,329 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files
2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 1.79/9.86 kb and 2/12 files
2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 2.34/9.86 kb and 3/12 files
2016-08-25 09:02:07,330 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 3.24/9.86 kb and 4/12 files
2016-08-25 09:02:07,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.13/9.86 kb and 5/12 files
2016-08-25 09:02:07,331 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 4.15/9.86 kb and 6/12 files
2016-08-25 09:02:07,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.05/9.86 kb and 7/12 files
2016-08-25 09:02:07,332 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 5.95/9.86 kb and 8/12 files
2016-08-25 09:02:07,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 6.84/9.86 kb and 9/12 files
2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55743
2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) started
2016-08-25 09:02:07,333 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:07,333 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 7.78/9.86 kb and 10/12 files
2016-08-25 09:02:07,333 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:07,333 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 8.68/9.86 kb and 11/12 files
2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files
2016-08-25 09:02:07,334 | INFO | hawtdispatch-DEFAULT-1 | Attached
Master started: node-2
======================================
103.4. Browse the queue on new master
======================================
2016-08-25 09:02:07,339 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
103.5. Found
======================================
======================================
104.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:07,352 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is shutting down
2016-08-25 09:02:07,354 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:07,357 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,358 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:07,359 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:07,359 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:07,359 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,360 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:07,360 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250068
2016-08-25 09:02:07,361 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55721 which had sessionid 0x156bec917250068
2016-08-25 09:02:07,361 | INFO | Time-limited test | Session: 0x156bec917250068 closed
2016-08-25 09:02:07,361 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:07,475 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:07,476 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) uptime 1.167 seconds
2016-08-25 09:02:07,476 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:105) is shutdown
Wait for master to start up...
2016-08-25 09:02:07,478 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:07,480 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1d25b88f
2016-08-25 09:02:07,480 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:07,481 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:07,481 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:07,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55745
2016-08-25 09:02:07,481 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55745
2016-08-25 09:02:07,482 | INFO | SyncThread:0 | Established session 0x156bec91725006b with negotiated timeout 10000 for client /127.0.0.1:55745
2016-08-25 09:02:07,482 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006b, negotiated timeout = 10000
2016-08-25 09:02:07,487 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,499 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave stopped
2016-08-25 09:02:07,501 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,512 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:07,513 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,515 | INFO | Starting broker node: node-3-EventThread | Promoted to master
2016-08-25 09:02:07,518 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,595 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Master started: tcp://localhost:55746
2016-08-25 09:02:07,600 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,600 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Attaching to master: tcp://localhost:55746
2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Attaching to master: tcp://localhost:55746
2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Slave started
2016-08-25 09:02:07,601 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave started
2016-08-25 09:02:07,616 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:07,667 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,668 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000157.sst
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000159
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000161.log
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:07,669 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000160.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000145.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000161.log
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000154.sst
2016-08-25 09:02:07,670 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000139.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/MANIFEST-000159
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000157.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000151.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000137.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000148.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000142.sst
2016-08-25 09:02:07,671 | INFO | hawtdispatch-DEFAULT-3 | Slave requested: 000000000000056d.index/000160.sst
2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files
2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.91/9.86 kb and 2/12 files
2016-08-25 09:02:07,674 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.46/9.86 kb and 3/12 files
2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.36/9.86 kb and 4/12 files
2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 0.90/9.86 kb and 1/12 files
2016-08-25 09:02:07,675 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 3.29/9.86 kb and 5/12 files
2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.08/9.86 kb and 2/12 files
2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 4.48/9.86 kb and 6/12 files
2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 5.38/9.86 kb and 7/12 files
2016-08-25 09:02:07,676 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 2.98/9.86 kb and 3/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 3.88/9.86 kb and 4/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 6.27/9.86 kb and 8/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 7.17/9.86 kb and 9/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 4.42/9.86 kb and 5/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 5.32/9.86 kb and 6/12 files
2016-08-25 09:02:07,677 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.07/9.86 kb and 10/12 files
2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files
2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 6.22/9.86 kb and 7/12 files
2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 7.16/9.86 kb and 8/12 files
2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files
2016-08-25 09:02:07,678 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:07,679 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.05/9.86 kb and 9/12 files
2016-08-25 09:02:07,679 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.07/9.86 kb and 10/12 files
2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 8.97/9.86 kb and 11/12 files
2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attaching... Downloaded 9.86/9.86 kb and 12/12 files
2016-08-25 09:02:07,680 | INFO | hawtdispatch-DEFAULT-3 | Attached
2016-08-25 09:02:07,681 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:07,681 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is starting
2016-08-25 09:02:07,684 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55751
2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) started
2016-08-25 09:02:07,685 | INFO | Starting broker node: node-3 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:07,685 | WARN | Starting broker node: node-3 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:07,685 | WARN | Starting broker node: node-3 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-3
======================================
104.4. Browse the queue on new master
======================================
======================================
104.5. Found
======================================
======================================
105.3. Restart master node
======================================
Stopping node-3
2016-08-25 09:02:07,705 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is shutting down
2016-08-25 09:02:07,706 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:07,709 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,711 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:07,711 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec917250069
2016-08-25 09:02:07,711 | INFO | Time-limited test | Session: 0x156bec917250069 closed
2016-08-25 09:02:07,711 | INFO | Starting broker node: node-3-EventThread | EventThread shut down
2016-08-25 09:02:07,711 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /0:0:0:0:0:0:0:1:55729 which had sessionid 0x156bec917250069
2016-08-25 09:02:07,715 | WARN | hawtdispatch-DEFAULT-2 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:07,715 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:07,716 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:07,716 | INFO | hawtdispatch-DEFAULT-2 | Slave has disconnected: e2fef573-6987-4c23-89fc-e23581268d5d
2016-08-25 09:02:07,817 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3]
2016-08-25 09:02:07,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) uptime 1.099 seconds
2016-08-25 09:02:07,818 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-3, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:106) is shutdown
Wait for master to start up...
2016-08-25 09:02:07,820 | INFO | Starting broker node: node-3 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-3, localhost:54916//default]
2016-08-25 09:02:07,822 | INFO | Starting broker node: node-3 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@63de5fd6
2016-08-25 09:02:07,822 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:07,823 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:07,823 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:07,823 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55753
2016-08-25 09:02:07,823 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55753
2016-08-25 09:02:07,824 | INFO | SyncThread:0 | Established session 0x156bec91725006c with negotiated timeout 10000 for client /127.0.0.1:55753
2016-08-25 09:02:07,824 | INFO | Starting broker node: node-3-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006c, negotiated timeout = 10000
2016-08-25 09:02:07,830 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave stopped
2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave stopped
2016-08-25 09:02:07,875 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,877 | INFO | Starting broker node: node-1-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:07,879 | INFO | Starting broker node: node-1-EventThread | Promoted to master
2016-08-25 09:02:07,883 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,910 | INFO | ActiveMQ BrokerService[node-1] Task-2 | Master started: tcp://localhost:55754
2016-08-25 09:02:07,915 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,916 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:07,916 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Attaching to master: tcp://localhost:55754
2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Slave started
2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Attaching to master: tcp://localhost:55754
2016-08-25 09:02:07,917 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave started
2016-08-25 09:02:07,932 | INFO | hawtdispatch-DEFAULT-4 | Slave has connected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:07,991 | INFO | hawtdispatch-DEFAULT-2 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:07,992 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000165.sst
2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/MANIFEST-000162
2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:07,993 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 0000000000000000.index/000164.log
2016-08-25 09:02:07,994 | INFO | hawtdispatch-DEFAULT-4 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/CURRENT
2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/MANIFEST-000162
2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000165.sst
2016-08-25 09:02:07,995 | INFO | hawtdispatch-DEFAULT-4 | Slave requested: 0000000000000000.index/000164.log
2016-08-25 09:02:07,996 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 0.94/2.87 kb and 1/4 files
2016-08-25 09:02:07,996 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.67/2.87 kb and 2/4 files
2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.69/2.87 kb and 3/4 files
2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files
2016-08-25 09:02:07,997 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:07,999 | INFO | hawtdispatch-DEFAULT-2 | Slave has now caught up: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:08,000 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is starting
2016-08-25 09:02:08,001 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.02/2.87 kb and 1/4 files
2016-08-25 09:02:08,002 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 0.75/2.87 kb and 2/4 files
2016-08-25 09:02:08,003 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 1.69/2.87 kb and 3/4 files
2016-08-25 09:02:08,004 | INFO | hawtdispatch-DEFAULT-4 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files
2016-08-25 09:02:08,004 | INFO | hawtdispatch-DEFAULT-4 | Attached
2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55759
2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) started
2016-08-25 09:02:08,005 | INFO | Starting broker node: node-1 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:08,005 | WARN | Starting broker node: node-1 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1 only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
2016-08-25 09:02:08,006 | WARN | Starting broker node: node-1 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30789 mb of usable space. - resetting to maximum available disk space: 30789 mb
Master started: node-1
======================================
105.4. Browse the queue on new master
======================================
2016-08-25 09:02:08,008 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
======================================
105.5. Found
======================================
======================================
106.3. Restart master node
======================================
Stopping node-1
2016-08-25 09:02:08,021 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is shutting down
2016-08-25 09:02:08,021 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:08,024 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,025 | WARN | hawtdispatch-DEFAULT-4 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:08,025 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:08,025 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:08,025 | INFO | hawtdispatch-DEFAULT-4 | Slave has disconnected: ac72117a-1229-44e6-8fb2-8e8f1bf10dc5
2016-08-25 09:02:08,025 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:08,025 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725006a
2016-08-25 09:02:08,026 | INFO | Time-limited test | Session: 0x156bec91725006a closed
2016-08-25 09:02:08,026 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55737 which had sessionid 0x156bec91725006a
2016-08-25 09:02:08,026 | INFO | Starting broker node: node-1-EventThread | EventThread shut down
2016-08-25 09:02:08,041 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Slave stopped
2016-08-25 09:02:08,041 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,042 | INFO | Starting broker node: node-2-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,048 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave stopped
2016-08-25 09:02:08,052 | INFO | Starting broker node: node-2-EventThread | Promoted to master
2016-08-25 09:02:08,054 | INFO | ActiveMQ BrokerService[node-2] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:08,078 | INFO | ActiveMQ BrokerService[node-2] Task-2 | Master started: tcp://localhost:55761
2016-08-25 09:02:08,081 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Using the pure java LevelDB implementation.
2016-08-25 09:02:08,083 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Attaching to master: tcp://localhost:55761
2016-08-25 09:02:08,083 | INFO | ActiveMQ BrokerService[node-3] Task-2 | Slave started
2016-08-25 09:02:08,097 | INFO | hawtdispatch-DEFAULT-1 | Slave has connected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:08,100 | INFO | hawtdispatch-DEFAULT-2 | Slave skipping download of: log/0000000000000000.log
2016-08-25 09:02:08,100 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000164.log
2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/MANIFEST-000162
2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/CURRENT
2016-08-25 09:02:08,101 | INFO | hawtdispatch-DEFAULT-2 | Slave requested: 000000000000056d.index/000165.sst
2016-08-25 09:02:08,104 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.18/2.87 kb and 1/4 files
2016-08-25 09:02:08,104 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.92/2.87 kb and 2/4 files
2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 1.93/2.87 kb and 3/4 files
2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attaching... Downloaded 2.87/2.87 kb and 4/4 files
2016-08-25 09:02:08,105 | INFO | hawtdispatch-DEFAULT-2 | Attached
2016-08-25 09:02:08,107 | INFO | hawtdispatch-DEFAULT-1 | Slave has now caught up: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:08,108 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is starting
2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Listening for connections at: tcp://Samuel-Mas-MacBook-Pro.local:55764
2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Connector tcp://0.0.0.0:0 started
2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) started
2016-08-25 09:02:08,112 | INFO | Starting broker node: node-2 | For help or more information please see: http://activemq.apache.org
2016-08-25 09:02:08,112 | WARN | Starting broker node: node-2 | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2 only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:08,112 | WARN | Starting broker node: node-2 | Temporary Store limit is 51200 mb (current store usage is 100 mb). The data directory: /Users/samuel.ma/git/lab/activemq/activemq-leveldb-store only has 30785 mb of usable space. - resetting to maximum available disk space: 30785 mb
2016-08-25 09:02:08,145 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1]
2016-08-25 09:02:08,146 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) uptime 1.109 seconds
2016-08-25 09:02:08,146 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-1, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:107) is shutdown
Wait for master to start up...
Master started: node-2
======================================
106.4. Browse the queue on new master
======================================
2016-08-25 09:02:08,148 | INFO | Starting broker node: node-1 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-1, localhost:54916//default]
2016-08-25 09:02:08,149 | INFO | Starting broker node: node-1 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@1288fe70
2016-08-25 09:02:08,149 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:08,150 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Opening socket connection to server localhost/0:0:0:0:0:0:0:1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:08,151 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Socket connection established to localhost/0:0:0:0:0:0:0:1:54916, initiating session
2016-08-25 09:02:08,151 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /0:0:0:0:0:0:0:1:55766
2016-08-25 09:02:08,151 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /0:0:0:0:0:0:0:1:55766
2016-08-25 09:02:08,152 | INFO | SyncThread:0 | Established session 0x156bec91725006d with negotiated timeout 10000 for client /0:0:0:0:0:0:0:1:55766
2016-08-25 09:02:08,152 | INFO | Starting broker node: node-1-SendThread(localhost:54916) | Session establishment complete on server localhost/0:0:0:0:0:0:0:1:54916, sessionid = 0x156bec91725006d, negotiated timeout = 10000
======================================
106.5. Found
======================================
======================================
107.3. Restart master node
======================================
Stopping node-2
2016-08-25 09:02:08,158 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is shutting down
2016-08-25 09:02:08,159 | INFO | Time-limited test | Connector tcp://0.0.0.0:0 stopped
2016-08-25 09:02:08,159 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Using the pure java LevelDB implementation.
2016-08-25 09:02:08,160 | INFO | ZooKeeper state change dispatcher thread | StateChangeDispatcher terminated.
2016-08-25 09:02:08,161 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Attaching to master: tcp://localhost:55761
2016-08-25 09:02:08,161 | INFO | ProcessThread(sid:0 cport:-1): | Processed session termination for sessionid: 0x156bec91725006b
2016-08-25 09:02:08,161 | INFO | ActiveMQ BrokerService[node-1] Task-1 | Slave started
2016-08-25 09:02:08,161 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,161 | INFO | Time-limited test | Session: 0x156bec91725006b closed
2016-08-25 09:02:08,161 | INFO | Starting broker node: node-2-EventThread | EventThread shut down
2016-08-25 09:02:08,161 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Closed socket connection for client /127.0.0.1:55745 which had sessionid 0x156bec91725006b
2016-08-25 09:02:08,162 | WARN | hawtdispatch-DEFAULT-1 | Unexpected session error: java.io.IOException: Connection reset by peer
2016-08-25 09:02:08,162 | INFO | hawtdispatch-DEFAULT-1 | Slave has disconnected: 2b5bec4f-a92b-46f6-ada8-477620658956
2016-08-25 09:02:08,175 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Slave stopped
2016-08-25 09:02:08,175 | INFO | ActiveMQ BrokerService[node-3] Task-1 | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,177 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
2016-08-25 09:02:08,267 | INFO | ActiveMQ BrokerService[node-2] Task-3 | Stopped LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2]
2016-08-25 09:02:08,268 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) uptime 0.792 seconds
2016-08-25 09:02:08,268 | INFO | Time-limited test | Apache ActiveMQ 5.15.0-SNAPSHOT (node-2, ID:Samuel-Mas-MacBook-Pro.local-54926-1472079667961-0:108) is shutdown
Wait for master to start up...
2016-08-25 09:02:08,270 | INFO | Starting broker node: node-2 | Using Persistence Adapter: Replicated LevelDB[/Users/samuel.ma/git/lab/activemq/activemq-leveldb-store/target/activemq-data/leveldb-elections/node-2, localhost:54916//default]
2016-08-25 09:02:08,272 | INFO | Starting broker node: node-2 | Initiating client connection, connectString=localhost:54916 sessionTimeout=15000 watcher=org.apache.activemq.leveldb.replicated.groups.ZKClient@5f8c78e
2016-08-25 09:02:08,272 | INFO | ZooKeeper state change dispatcher thread | Starting StateChangeDispatcher
2016-08-25 09:02:08,272 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Opening socket connection to server localhost/127.0.0.1:54916. Will not attempt to authenticate using SASL (unknown error)
2016-08-25 09:02:08,273 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Socket connection established to localhost/127.0.0.1:54916, initiating session
2016-08-25 09:02:08,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Accepted socket connection from /127.0.0.1:55767
2016-08-25 09:02:08,273 | INFO | NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0 | Client attempting to establish new session at /127.0.0.1:55767
2016-08-25 09:02:08,273 | INFO | SyncThread:0 | Established session 0x156bec91725006e with negotiated timeout 10000 for client /127.0.0.1:55767
2016-08-25 09:02:08,273 | INFO | Starting broker node: node-2-SendThread(localhost:54916) | Session establishment complete on server localhost/127.0.0.1:54916, sessionid = 0x156bec91725006e, negotiated timeout = 10000
2016-08-25 09:02:08,277 | INFO | Starting broker node: node-3-EventThread | Not enough cluster members have reported their update positions yet.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)