You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/06/03 01:28:38 UTC

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3315 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3315/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1

Error Message:
CollectionStateWatcher not notified of stateformat=1 collection creation

Stack Trace:
java.lang.AssertionError: CollectionStateWatcher not notified of stateformat=1 collection creation
	at __randomizedtesting.SeedInfo.seed([3011B606D36DAB04:57D6AAE827D58FA5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1(TestCollectionStateWatchers.java:267)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12831 lines...]
   [junit4] Suite: org.apache.solr.common.cloud.TestCollectionStateWatchers
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/init-core-data-001
   [junit4]   2> 0    INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.e.j.u.log Logging initialized @८४१२ms
   [junit4]   2> 23   INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 122  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 127  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 128  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 339  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64346
   [junit4]   2> 481  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 682  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 820  INFO  (zkCallback-3-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78186224 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 821  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 827  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 830  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@27863802{/solr,null,AVAILABLE}
   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12bd2247{/solr,null,AVAILABLE}
   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@207554f2{/solr,null,AVAILABLE}
   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15b198aa{/solr,null,AVAILABLE}
   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@2f73642b{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५२}
   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.Server Started @१०१२५ms
   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64352}
   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@54722d49{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५०}
   [junit4]   2> 1704 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.Server Started @१०१२८ms
   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@3904fe13{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५१}
   [junit4]   2> 1706 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.Server Started @१०१३०ms
   [junit4]   2> 1706 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64351}
   [junit4]   2> 1701 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@28f92961{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५३}
   [junit4]   2> 1707 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.Server Started @१०१३१ms
   [junit4]   2> 1707 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64353}
   [junit4]   2> 1704 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64350}
   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4'
   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3'
   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2'
   [junit4]   2> 1847 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1'
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1864 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1865 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1875 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1876 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1875 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1876 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1878 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1878 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1879 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1880 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1888 INFO  (zkCallback-5-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7b4f1bf2 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1888 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1889 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1889 INFO  (zkCallback-4-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69f79b73 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1890 INFO  (zkCallback-7-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1113c498 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1891 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1892 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1892 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1893 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1894 INFO  (zkCallback-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@8d55a67 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1895 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1895 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1897 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1899 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1900 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1902 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/.
   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/.
   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/.
   [junit4]   2> 2287 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/.
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 662789511
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 1398327457
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2]
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 1188624207
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 808316540
   [junit4]   2> 2456 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1]
   [junit4]   2> 2456 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4]
   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3]
   [junit4]   2> 2461 WARN  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/lib
   [junit4]   2> 2462 WARN  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/lib
   [junit4]   2> 2463 WARN  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/lib
   [junit4]   2> 2463 WARN  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/lib
   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 3222 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3222 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3227 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3228 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3248 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3250 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3250 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3248 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3260 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
   [junit4]   2> 3271 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3378 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3380 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3381 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3385 INFO  (zkCallback-16-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@44d5df17 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3386 INFO  (zkCallback-18-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e16f304 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3386 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@aeef7b1 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3387 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@151b95df name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3389 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3388 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3390 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3390 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3388 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3392 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@3ebdca53
   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@99734e9
   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@3906f974
   [junit4]   2> 3417 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@633c9475
   [junit4]   2> 3423 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3424 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3425 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3426 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ab852f9 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3426 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3427 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e673c57 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3427 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46f80990 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3427 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3427 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3446 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@147f4b52 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3447 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3447 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3461 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3462 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3462 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3475 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3476 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3476 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3485 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3486 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3486 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3538 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 3539 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 3541 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 3542 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 3543 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3552 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3552 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3553 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3555 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3555 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3557 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3558 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3559 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3559 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3567 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3569 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3581 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3659 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 3661 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 3663 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 3663 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 3666 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3676 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3677 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000
   [junit4]   2> 3678 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671498-127.0.0.1:64353_solr-n_0000000001
   [junit4]   2> 3696 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64352_solr
   [junit4]   2> 3699 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3700 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 3701 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002
   [junit4]   2> 3706 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 3706 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer (id=96004804580671497-127.0.0.1:64352_solr-n_0000000000) starting
   [junit4]   2> 3707 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64353_solr
   [junit4]   2> 3710 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64353_solr
   [junit4]   2> 3717 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3717 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3717 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3718 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3718 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671498-127.0.0.1:64353_solr-n_0000000001 to know if I could be the leader
   [junit4]   2> 3731 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3731 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3733 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3734 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3734 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3736 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671500-127.0.0.1:64350_solr-n_0000000003
   [junit4]   2> 3736 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64351_solr as DOWN
   [junit4]   2> 3737 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 3739 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002 to know if I could be the leader
   [junit4]   2> 3740 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64350_solr as DOWN
   [junit4]   2> 3848 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3956 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 3956 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 4242 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 4250 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 4252 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64352_solr as DOWN
   [junit4]   2> 4265 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 4284 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:64350_solr"} current state version: 0
   [junit4]   2> 4308 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64350_solr
   [junit4]   2> 4319 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:64351_solr"} current state version: 0
   [junit4]   2> 4319 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64351_solr
   [junit4]   2> 4331 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:64352_solr"} current state version: 0
   [junit4]   2> 4332 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64352_solr
   [junit4]   2> 4366 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/.
   [junit4]   2> 4377 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 4388 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
   [junit4]   2> 4389 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 4851 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64351_solr
   [junit4]   2> 4851 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64350_solr
   [junit4]   2> 4852 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64351_solr
   [junit4]   2> 4852 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64350_solr
   [junit4]   2> 4857 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4858 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4859 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4859 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4861 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 4861 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 4862 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 4863 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
   [junit4]   2> 4892 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 4892 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 4894 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/.
   [junit4]   2> 4895 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 4897 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
   [junit4]   2> 4897 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 4903 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 4903 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 4904 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/.
   [junit4]   2> 4905 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 4907 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
   [junit4]   2> 4907 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 5270 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64352_solr
   [junit4]   2> 5271 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64352_solr
   [junit4]   2> 5274 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5274 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5276 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5276 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 5276 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 5277 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 5278 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 5278 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 5295 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 5295 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 5299 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/.
   [junit4]   2> 5300 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 5302 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
   [junit4]   2> 5302 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 5303 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 5303 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5307 INFO  (zkCallback-28-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7fd55bc6 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5308 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5308 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5364 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:64346/solr
   [junit4]   2> 5383 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 5384 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5387 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@715b610a name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5387 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5388 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5388 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 5390 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 5392 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
   [junit4]   2> 5422 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-replication1.xml
   [junit4]   2> 5435 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-sql.xml
   [junit4]   2> 5442 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema.xml
   [junit4]   2> 5447 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-managed-schema.xml
   [junit4]   2> 5451 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-slave1.xml
   [junit4]   2> 5456 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-sql.xml
   [junit4]   2> 5461 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig.xml
   [junit4]   2> 5526 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatchesWorkForStateFormat1
   [junit4]   2> 5526 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 5532 INFO  (zkCallback-32-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4bac7f49 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5532 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5532 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5533 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5564 INFO  (backgroundWatchers-1-thread-1) [    ] o.a.s.c.c.ZkStateReader Deleting data for [stateformat1]
   [junit4]   2> 7538 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=30dab8af-6efb-49d3-94d8-654355a86598&replicationFactor=1&collection.configName=config&name=stateformat1&action=CREATE&numShards=1&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7556 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=30dab8af-6efb-49d3-94d8-654355a86598&replicationFactor=1&collection.configName=config&name=stateformat1&action=CREATE&numShards=1&stateFormat=1&wt=javabin&version=2} status=0 QTime=336
   [junit4]   2> 7585 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running/mn-30dab8af-6efb-49d3-94d8-654355a86598
   [junit4]   2> 7590 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"stateformat1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"config",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"1",
   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 7596 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"stateformat1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"config",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"1",
   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 7599 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection stateformat1
   [junit4]   2> 7615 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/stateformat1 
   [junit4]   2> 7617 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/stateformat1
   [junit4]   2> 7625 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 7625 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 7625 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 7627 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 7628 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 7643 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"stateformat1",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"1",
   [junit4]   2>   "collection.configName":"config",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"1",
   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 7643 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: stateformat1
   [junit4]   2> 7682 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7682 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7682 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7682 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7683 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7755 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection stateformat1, shardNames [shard1] , replicationFactor : 1
   [junit4]   2> 7756 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core stateformat1_shard1_replica1 as part of shard shard1 of collection stateformat1 on 127.0.0.1:64350_solr
   [junit4]   2> 7803 INFO  (qtp2049292927-40) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={async=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&qt=/admin/cores&collection.configName=config&name=stateformat1_shard1_replica1&action=CREATE&numShards=1&collection=stateformat1&shard=shard1&wt=javabin&version=2} status=0 QTime=11
   [junit4]   2> 7814 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation core create command async=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&qt=/admin/cores&collection.configName=config&name=stateformat1_shard1_replica1&action=CREATE&numShards=1&collection=stateformat1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 7830 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=stateformat1_shard1_replica1, collection.configName=config, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=stateformat1, numShards=1, dataDir=data/, shard=shard1}
   [junit4]   2> 7831 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 7833 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Registering watch for collection stateformat1
   [junit4]   2> 7836 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader Deleting data for [stateformat1]
   [junit4]   2> 7836 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 7839 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"stateformat1_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:64350/solr",
   [junit4]   2>   "node_name":"127.0.0.1:64350_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"stateformat1",
   [junit4]   2>   "operation":"state"} current state version: 1
   [junit4]   2> 7851 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"stateformat1_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:64350/solr",
   [junit4]   2>   "node_name":"127.0.0.1:64350_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"stateformat1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 7858 INFO  (qtp2049292927-42) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 7859 INFO  (qtp2049292927-42) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 7906 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 7927 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=21
   [junit4]   2> 7989 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7989 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7990 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7990 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 7992 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
   [junit4]   2> 8837 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController waiting to find shard id in clusterstate for stateformat1_shard1_replica1
   [junit4]   2> 8837 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Check for collection zkNode:stateformat1
   [junit4]   2> 8838 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 8838 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/stateformat1]
   [junit4]   2> 8840 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/stateformat1] [configName]=[config] specified config exists in ZooKeeper
   [junit4]   2> 8847 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/stateformat1_shard1_replica1'
   [junit4]   2> 8848 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 8848 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 8855 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 8893 INFO  (qtp2049292927-47) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 8893 INFO  (qtp2049292927-47) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 8928 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 8931 INFO  (qtp1159626161-38) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 8939 INFO  (qtp1159626161-38) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=7
   [junit4]   2> 8969 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 9063 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 9213 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema [stateformat1_shard1_replica1] Schema name=test
   [junit4]   2> 9898 INFO  (qtp2049292927-50) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 9898 INFO  (qtp2049292927-50) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 9942 INFO  (qtp1159626161-41) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 9944 INFO  (qtp1159626161-41) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 10903 INFO  (qtp2049292927-54) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 10903 INFO  (qtp2049292927-54) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 10946 INFO  (qtp1159626161-46) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 10948 INFO  (qtp1159626161-46) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 11761 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 11762 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 11881 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 11882 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 11906 INFO  (qtp2049292927-55) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 11907 INFO  (qtp2049292927-55) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 11952 INFO  (qtp1159626161-52) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 11953 INFO  (qtp1159626161-52) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 12089 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/config', cwd=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
   [junit4]   2> 12096 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'stateformat1_shard1_replica1' using configuration from collection stateformat1
   [junit4]   2> 12099 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 12223 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore [[stateformat1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/stateformat1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/]
   [junit4]   2> 12235 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore JMX monitoring not detected for core: stateformat1_shard1_replica1
   [junit4]   2> 12667 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data
   [junit4]   2> 12678 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index/
   [junit4]   2> 12678 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore [stateformat1_shard1_replica1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 12679 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index
   [junit4]   2> 12911 INFO  (qtp2049292927-29) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
   [junit4]   2> 12911 INFO  (qtp2049292927-29) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 12957 INFO  (qtp1159626161-53) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 12961 INFO  (qtp1159626161-53) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=4
   [junit4]   2> 13304 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6ea1ebcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1864dac4),segFN=segments_1,generation=1}
   [junit4]   2> 13311 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 13312 INFO  (OldIndexDirectoryCleanupThreadForCore-stateformat1_shard1_replica1) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core stateformat1_shard1_replica1 in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/
   [junit4]   2> 13314 WARN  (OldIndexDirectoryCleanupThreadForCore-stateformat1_shard1_replica1) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.DirectoryFactory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 13923 INFO  (qtp2049292927-40) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreA

[...truncated too long message...]

0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/./testcollection_shard1_replica1/data
   [junit4]   2> 51330 INFO  (zkCallback-47-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delayed/state.json] for collection [delayed] has occurred - updating... (live nodes size: [3])
   [junit4]   2> 51330 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 51330 INFO  (zkCallback-41-thread-2-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 51330 WARN  (zkCallback-41-thread-2-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 51330 INFO  (zkCallback-47-thread-2-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
   [junit4]   2> 51331 WARN  (zkCallback-47-thread-2-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 51331 WARN  (zkCallback-47-thread-3-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector 
   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:105)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 51332 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
   [junit4]   2> 51332 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.Overseer Overseer (id=96004804580671497-127.0.0.1:64352_solr-n_0000000000) closing
   [junit4]   2> 51333 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4d07dd56{/solr,null,UNAVAILABLE}
   [junit4]   2> 51333 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:64352_solr
   [junit4]   2> 51337 ERROR (jetty-launcher-2-thread-1-EventThread) [n:127.0.0.1:64352_solr    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/81944345@1cc72bd7 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@59c5a62b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 300]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 51338 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@207554f2{/solr,null,UNAVAILABLE}
   [junit4]   2> 52835 WARN  (zkCallback-47-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections/delayed/state.json]
   [junit4]   2> 52836 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@91b95c4{/solr,null,UNAVAILABLE}
   [junit4]   2> 52840 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64346 64346
   [junit4]   2> 52860 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64346 64346
   [junit4]   2> 54537 WARN  (Thread-1) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	6	/solr/security.json
   [junit4]   2> 	6	/solr/configs/config
   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	16	/solr/collections/falsepredicate/state.json
   [junit4]   2> 	12	/solr/collections/waitforstate/state.json
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	7	/solr/collections/testcollection/state.json
   [junit4]   2> 	3	/solr/overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000
   [junit4]   2> 	2	/solr/collections/delayed/state.json
   [junit4]   2> 	2	/solr/collections/currentstate/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002
   [junit4]   2> 	2	/solr/collections/tobedeleted/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	7	/solr/collections
   [junit4]   2> 	5	/solr/live_nodes
   [junit4]   2> 
   [junit4]   2> 54537 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=hi-IN, timezone=EST5EDT
   [junit4]   2> NOTE: Mac OS X 10.11.4 x86_64/Oracle Corporation 1.8.0_72 (64-bit)/cpus=3,threads=1,free=45878088,total=105119744
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionAdminRequestRequiredParamsTest, TestCollectionStateWatchers]
   [junit4] Completed [7/83 (1!)] on J0 in 56.69s, 9 tests, 1 failure <<< FAILURES!

[...truncated 238 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:740: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:684: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:59: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:246: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:531: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1427: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:984: There were test failures: 83 suites, 641 tests, 1 failure, 2 ignored [seed: 3011B606D36DAB04]

Total time: 85 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3315 - Failure!

Posted by Alan Woodward <al...@flax.co.uk>.
Turns out 10 seconds is too low a timeout to start a core on this machine… I'll up the test timeouts.

Alan Woodward
www.flax.co.uk


On 3 Jun 2016, at 02:28, Policeman Jenkins Server wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3315/
> Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
> 
> 1 tests failed.
> FAILED:  org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1
> 
> Error Message:
> CollectionStateWatcher not notified of stateformat=1 collection creation
> 
> Stack Trace:
> java.lang.AssertionError: CollectionStateWatcher not notified of stateformat=1 collection creation
> 	at __randomizedtesting.SeedInfo.seed([3011B606D36DAB04:57D6AAE827D58FA5]:0)
> 	at org.junit.Assert.fail(Assert.java:93)
> 	at org.junit.Assert.assertTrue(Assert.java:43)
> 	at org.apache.solr.common.cloud.TestCollectionStateWatchers.testWatchesWorkForStateFormat1(TestCollectionStateWatchers.java:267)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
> 	at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 
> Build Log:
> [...truncated 12831 lines...]
>   [junit4] Suite: org.apache.solr.common.cloud.TestCollectionStateWatchers
>   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/init-core-data-001
>   [junit4]   2> 0    INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.e.j.u.log Logging initialized @८४१२ms
>   [junit4]   2> 23   INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
>   [junit4]   2> 122  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>   [junit4]   2> 127  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
>   [junit4]   2> 128  INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer Starting server
>   [junit4]   2> 339  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:64346
>   [junit4]   2> 481  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 682  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 820  INFO  (zkCallback-3-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@78186224 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 821  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 827  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 830  INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
>   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>   [junit4]   2> 1551 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
>   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@27863802{/solr,null,AVAILABLE}
>   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12bd2247{/solr,null,AVAILABLE}
>   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@207554f2{/solr,null,AVAILABLE}
>   [junit4]   2> 1661 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15b198aa{/solr,null,AVAILABLE}
>   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@2f73642b{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५२}
>   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.Server Started @१०१२५ms
>   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64352}
>   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@54722d49{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५०}
>   [junit4]   2> 1704 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.Server Started @१०१२८ms
>   [junit4]   2> 1700 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.ServerConnector Started ServerConnector@3904fe13{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५१}
>   [junit4]   2> 1706 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.Server Started @१०१३०ms
>   [junit4]   2> 1706 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64351}
>   [junit4]   2> 1701 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.ServerConnector Started ServerConnector@28f92961{HTTP/1.1,[http/1.1]}{127.0.0.1:६४३५३}
>   [junit4]   2> 1707 INFO  (jetty-launcher-2-thread-4) [    ] o.e.j.s.Server Started @१०१३१ms
>   [junit4]   2> 1707 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64353}
>   [junit4]   2> 1704 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=64350}
>   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
>   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
>   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
>   [junit4]   2> 1757 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4554617c
>   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4'
>   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3'
>   [junit4]   2> 1846 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2'
>   [junit4]   2> 1847 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1'
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1864 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 1863 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>   [junit4]   2> 1865 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>   [junit4]   2> 1875 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 1876 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 1875 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 1876 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 1878 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1878 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1879 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1880 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 1888 INFO  (zkCallback-5-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7b4f1bf2 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1888 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 1889 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 1889 INFO  (zkCallback-4-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69f79b73 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1890 INFO  (zkCallback-7-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1113c498 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1891 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 1892 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 1892 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 1893 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 1894 INFO  (zkCallback-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@8d55a67 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 1895 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>   [junit4]   2> 1895 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 1897 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 1899 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>   [junit4]   2> 1900 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>   [junit4]   2> 1902 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/.
>   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/.
>   [junit4]   2> 2286 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/.
>   [junit4]   2> 2287 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/.
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer New CoreContainer 662789511
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 1398327457
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2]
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer New CoreContainer 1188624207
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 808316540
>   [junit4]   2> 2456 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1]
>   [junit4]   2> 2456 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4]
>   [junit4]   2> 2455 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3]
>   [junit4]   2> 2461 WARN  (jetty-launcher-2-thread-3) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/lib
>   [junit4]   2> 2462 WARN  (jetty-launcher-2-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/lib
>   [junit4]   2> 2463 WARN  (jetty-launcher-2-thread-4) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/lib
>   [junit4]   2> 2463 WARN  (jetty-launcher-2-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/lib to classpath: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/lib
>   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
>   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
>   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
>   [junit4]   2> 2526 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
>   [junit4]   2> 3222 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
>   [junit4]   2> 3222 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
>   [junit4]   2> 3227 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
>   [junit4]   2> 3228 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
>   [junit4]   2> 3248 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 3250 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 3250 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 3248 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
>   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 3260 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 3259 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
>   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
>   [junit4]   2> 3271 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
>   [junit4]   2> 3269 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:64346/solr
>   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
>   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.ZkController zkHost includes chroot
>   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
>   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 3375 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.ZkController zkHost includes chroot
>   [junit4]   2> 3376 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 3378 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3380 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3381 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3385 INFO  (zkCallback-16-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@44d5df17 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3386 INFO  (zkCallback-18-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2e16f304 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3386 INFO  (zkCallback-17-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@aeef7b1 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3387 INFO  (zkCallback-19-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@151b95df name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-3) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 3387 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3389 INFO  (jetty-launcher-2-thread-4) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 3388 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3390 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3390 INFO  (jetty-launcher-2-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 3388 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3392 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@3ebdca53
>   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@99734e9
>   [junit4]   2> 3410 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@3906f974
>   [junit4]   2> 3417 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/538620521@633c9475
>   [junit4]   2> 3423 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3424 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3425 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3426 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ab852f9 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3426 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3427 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e673c57 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3427 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46f80990 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3427 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3427 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3446 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@147f4b52 name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 3447 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 3447 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 3461 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>   [junit4]   2> 3462 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>   [junit4]   2> 3462 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
>   [junit4]   2> 3475 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>   [junit4]   2> 3476 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>   [junit4]   2> 3476 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
>   [junit4]   2> 3485 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
>   [junit4]   2> 3486 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
>   [junit4]   2> 3486 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
>   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
>   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
>   [junit4]   2> 3501 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
>   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
>   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
>   [junit4]   2> 3507 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
>   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>   [junit4]   2> 3535 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
>   [junit4]   2> 3538 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
>   [junit4]   2> 3539 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
>   [junit4]   2> 3541 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
>   [junit4]   2> 3542 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
>   [junit4]   2> 3543 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
>   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
>   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
>   [junit4]   2> 3545 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
>   [junit4]   2> 3552 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>   [junit4]   2> 3552 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>   [junit4]   2> 3553 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
>   [junit4]   2> 3555 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>   [junit4]   2> 3555 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>   [junit4]   2> 3557 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>   [junit4]   2> 3558 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>   [junit4]   2> 3559 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>   [junit4]   2> 3559 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>   [junit4]   2> 3567 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>   [junit4]   2> 3569 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>   [junit4]   2> 3579 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>   [junit4]   2> 3581 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
>   [junit4]   2> 3659 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>   [junit4]   2> 3661 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
>   [junit4]   2> 3663 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
>   [junit4]   2> 3663 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
>   [junit4]   2> 3666 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>   [junit4]   2> 3676 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>   [junit4]   2> 3677 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000
>   [junit4]   2> 3678 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671498-127.0.0.1:64353_solr-n_0000000001
>   [junit4]   2> 3696 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:64352_solr
>   [junit4]   2> 3699 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>   [junit4]   2> 3700 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
>   [junit4]   2> 3701 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002
>   [junit4]   2> 3706 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000 to know if I could be the leader
>   [junit4]   2> 3706 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer (id=96004804580671497-127.0.0.1:64352_solr-n_0000000000) starting
>   [junit4]   2> 3707 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64353_solr
>   [junit4]   2> 3710 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64353_solr
>   [junit4]   2> 3717 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>   [junit4]   2> 3717 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>   [junit4]   2> 3717 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>   [junit4]   2> 3718 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>   [junit4]   2> 3718 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671498-127.0.0.1:64353_solr-n_0000000001 to know if I could be the leader
>   [junit4]   2> 3731 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>   [junit4]   2> 3731 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
>   [junit4]   2> 3733 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>   [junit4]   2> 3734 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>   [junit4]   2> 3734 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
>   [junit4]   2> 3736 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96004804580671500-127.0.0.1:64350_solr-n_0000000003
>   [junit4]   2> 3736 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64351_solr as DOWN
>   [junit4]   2> 3737 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
>   [junit4]   2> 3739 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002 to know if I could be the leader
>   [junit4]   2> 3740 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64350_solr as DOWN
>   [junit4]   2> 3848 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
>   [junit4]   2> 3956 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>   [junit4]   2> 3956 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>   [junit4]   2> 4242 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
>   [junit4]   2> 4250 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Starting to work on the main queue
>   [junit4]   2> 4252 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:64352_solr as DOWN
>   [junit4]   2> 4265 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
>   [junit4]   2> 4284 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
>   [junit4]   2>   "operation":"downnode",
>   [junit4]   2>   "node_name":"127.0.0.1:64350_solr"} current state version: 0
>   [junit4]   2> 4308 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64350_solr
>   [junit4]   2> 4319 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
>   [junit4]   2>   "operation":"downnode",
>   [junit4]   2>   "node_name":"127.0.0.1:64351_solr"} current state version: 0
>   [junit4]   2> 4319 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64351_solr
>   [junit4]   2> 4331 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 3, message = {
>   [junit4]   2>   "operation":"downnode",
>   [junit4]   2>   "node_name":"127.0.0.1:64352_solr"} current state version: 0
>   [junit4]   2> 4332 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:64352_solr
>   [junit4]   2> 4366 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/.
>   [junit4]   2> 4377 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>   [junit4]   2> 4388 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
>   [junit4]   2> 4389 INFO  (jetty-launcher-2-thread-4) [n:127.0.0.1:64353_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>   [junit4]   2> 4851 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64351_solr
>   [junit4]   2> 4851 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64350_solr
>   [junit4]   2> 4852 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64351_solr
>   [junit4]   2> 4852 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64350_solr
>   [junit4]   2> 4857 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
>   [junit4]   2> 4858 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
>   [junit4]   2> 4859 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
>   [junit4]   2> 4859 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
>   [junit4]   2> 4861 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
>   [junit4]   2> 4861 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
>   [junit4]   2> 4862 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
>   [junit4]   2> 4863 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
>   [junit4]   2> 4892 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>   [junit4]   2> 4892 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>   [junit4]   2> 4894 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/.
>   [junit4]   2> 4895 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>   [junit4]   2> 4897 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
>   [junit4]   2> 4897 INFO  (jetty-launcher-2-thread-2) [n:127.0.0.1:64350_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>   [junit4]   2> 4903 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>   [junit4]   2> 4903 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>   [junit4]   2> 4904 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node3/.
>   [junit4]   2> 4905 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>   [junit4]   2> 4907 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
>   [junit4]   2> 4907 INFO  (jetty-launcher-2-thread-3) [n:127.0.0.1:64351_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>   [junit4]   2> 5270 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64352_solr
>   [junit4]   2> 5271 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:64352_solr
>   [junit4]   2> 5274 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 5274 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 5276 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 5276 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 5276 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>   [junit4]   2> 5277 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>   [junit4]   2> 5278 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>   [junit4]   2> 5278 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
>   [junit4]   2> 5295 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
>   [junit4]   2> 5295 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
>   [junit4]   2> 5299 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node1/.
>   [junit4]   2> 5300 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
>   [junit4]   2> 5302 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
>   [junit4]   2> 5302 INFO  (jetty-launcher-2-thread-1) [n:127.0.0.1:64352_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
>   [junit4]   2> 5303 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 5303 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 5307 INFO  (zkCallback-28-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7fd55bc6 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 5308 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 5308 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 5364 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:64346/solr
>   [junit4]   2> 5383 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 5384 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 5387 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@715b610a name:ZooKeeperConnection Watcher:127.0.0.1:64346/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 5387 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 5388 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 5388 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
>   [junit4]   2> 5390 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
>   [junit4]   2> 5392 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
>   [junit4]   2> 5422 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-replication1.xml
>   [junit4]   2> 5435 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema-sql.xml
>   [junit4]   2> 5442 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/schema.xml
>   [junit4]   2> 5447 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-managed-schema.xml
>   [junit4]   2> 5451 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-slave1.xml
>   [junit4]   2> 5456 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig-sql.xml
>   [junit4]   2> 5461 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/config/solrconfig.xml
>   [junit4]   2> 5526 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatchesWorkForStateFormat1
>   [junit4]   2> 5526 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
>   [junit4]   2> 5532 INFO  (zkCallback-32-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4bac7f49 name:ZooKeeperConnection Watcher:127.0.0.1:64346 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
>   [junit4]   2> 5532 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
>   [junit4]   2> 5532 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
>   [junit4]   2> 5533 INFO  (TEST-TestCollectionStateWatchers.testWatchesWorkForStateFormat1-seed#[3011B606D36DAB04]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
>   [junit4]   2> 5564 INFO  (backgroundWatchers-1-thread-1) [    ] o.a.s.c.c.ZkStateReader Deleting data for [stateformat1]
>   [junit4]   2> 7538 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=30dab8af-6efb-49d3-94d8-654355a86598&replicationFactor=1&collection.configName=config&name=stateformat1&action=CREATE&numShards=1&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 7556 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=30dab8af-6efb-49d3-94d8-654355a86598&replicationFactor=1&collection.configName=config&name=stateformat1&action=CREATE&numShards=1&stateFormat=1&wt=javabin&version=2} status=0 QTime=336
>   [junit4]   2> 7585 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running/mn-30dab8af-6efb-49d3-94d8-654355a86598
>   [junit4]   2> 7590 INFO  (OverseerCollectionConfigSetProcessor-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
>   [junit4]   2>   "name":"stateformat1",
>   [junit4]   2>   "fromApi":"true",
>   [junit4]   2>   "replicationFactor":"1",
>   [junit4]   2>   "collection.configName":"config",
>   [junit4]   2>   "numShards":"1",
>   [junit4]   2>   "stateFormat":"1",
>   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
>   [junit4]   2>   "operation":"create"}
>   [junit4]   2> 7596 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
>   [junit4]   2>   "name":"stateformat1",
>   [junit4]   2>   "fromApi":"true",
>   [junit4]   2>   "replicationFactor":"1",
>   [junit4]   2>   "collection.configName":"config",
>   [junit4]   2>   "numShards":"1",
>   [junit4]   2>   "stateFormat":"1",
>   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
>   [junit4]   2>   "operation":"create"}
>   [junit4]   2> 7599 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection stateformat1
>   [junit4]   2> 7615 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/stateformat1 
>   [junit4]   2> 7617 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/stateformat1
>   [junit4]   2> 7625 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>   [junit4]   2> 7625 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>   [junit4]   2> 7625 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>   [junit4]   2> 7627 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>   [junit4]   2> 7628 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
>   [junit4]   2> 7643 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>   [junit4]   2>   "name":"stateformat1",
>   [junit4]   2>   "fromApi":"true",
>   [junit4]   2>   "replicationFactor":"1",
>   [junit4]   2>   "collection.configName":"config",
>   [junit4]   2>   "numShards":"1",
>   [junit4]   2>   "stateFormat":"1",
>   [junit4]   2>   "async":"30dab8af-6efb-49d3-94d8-654355a86598",
>   [junit4]   2>   "operation":"create"} current state version: 0
>   [junit4]   2> 7643 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: stateformat1
>   [junit4]   2> 7682 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7682 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7682 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7682 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7683 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7755 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection stateformat1, shardNames [shard1] , replicationFactor : 1
>   [junit4]   2> 7756 INFO  (OverseerThreadFactory-17-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core stateformat1_shard1_replica1 as part of shard shard1 of collection stateformat1 on 127.0.0.1:64350_solr
>   [junit4]   2> 7803 INFO  (qtp2049292927-40) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={async=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&qt=/admin/cores&collection.configName=config&name=stateformat1_shard1_replica1&action=CREATE&numShards=1&collection=stateformat1&shard=shard1&wt=javabin&version=2} status=0 QTime=11
>   [junit4]   2> 7814 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation core create command async=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&qt=/admin/cores&collection.configName=config&name=stateformat1_shard1_replica1&action=CREATE&numShards=1&collection=stateformat1&shard=shard1&wt=javabin&version=2
>   [junit4]   2> 7830 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=stateformat1_shard1_replica1, collection.configName=config, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=stateformat1, numShards=1, dataDir=data/, shard=shard1}
>   [junit4]   2> 7831 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController publishing state=down
>   [junit4]   2> 7833 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Registering watch for collection stateformat1
>   [junit4]   2> 7836 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader Deleting data for [stateformat1]
>   [junit4]   2> 7836 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController look for our core node name
>   [junit4]   2> 7839 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
>   [junit4]   2>   "core":"stateformat1_shard1_replica1",
>   [junit4]   2>   "roles":null,
>   [junit4]   2>   "base_url":"http://127.0.0.1:64350/solr",
>   [junit4]   2>   "node_name":"127.0.0.1:64350_solr",
>   [junit4]   2>   "numShards":"1",
>   [junit4]   2>   "state":"down",
>   [junit4]   2>   "shard":"shard1",
>   [junit4]   2>   "collection":"stateformat1",
>   [junit4]   2>   "operation":"state"} current state version: 1
>   [junit4]   2> 7851 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
>   [junit4]   2>   "core":"stateformat1_shard1_replica1",
>   [junit4]   2>   "roles":null,
>   [junit4]   2>   "base_url":"http://127.0.0.1:64350/solr",
>   [junit4]   2>   "node_name":"127.0.0.1:64350_solr",
>   [junit4]   2>   "numShards":"1",
>   [junit4]   2>   "state":"down",
>   [junit4]   2>   "shard":"shard1",
>   [junit4]   2>   "collection":"stateformat1",
>   [junit4]   2>   "operation":"state"}
>   [junit4]   2> 7858 INFO  (qtp2049292927-42) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 7859 INFO  (qtp2049292927-42) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
>   [junit4]   2> 7906 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 7927 INFO  (qtp1159626161-36) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=21
>   [junit4]   2> 7989 INFO  (zkCallback-22-thread-1-processing-n:127.0.0.1:64351_solr) [n:127.0.0.1:64351_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7989 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7990 INFO  (zkCallback-23-thread-1-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7990 INFO  (zkCallback-20-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 7992 INFO  (zkCallback-21-thread-1-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [4])
>   [junit4]   2> 8837 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController waiting to find shard id in clusterstate for stateformat1_shard1_replica1
>   [junit4]   2> 8837 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Check for collection zkNode:stateformat1
>   [junit4]   2> 8838 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ZkController Collection zkNode exists
>   [junit4]   2> 8838 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/stateformat1]
>   [junit4]   2> 8840 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/stateformat1] [configName]=[config] specified config exists in ZooKeeper
>   [junit4]   2> 8847 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/stateformat1_shard1_replica1'
>   [junit4]   2> 8848 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
>   [junit4]   2> 8848 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>   [junit4]   2> 8855 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
>   [junit4]   2> 8893 INFO  (qtp2049292927-47) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 8893 INFO  (qtp2049292927-47) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
>   [junit4]   2> 8928 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig current version of requestparams : -1
>   [junit4]   2> 8931 INFO  (qtp1159626161-38) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 8939 INFO  (qtp1159626161-38) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=7
>   [junit4]   2> 8969 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
>   [junit4]   2> 9063 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
>   [junit4]   2> 9213 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema [stateformat1_shard1_replica1] Schema name=test
>   [junit4]   2> 9898 INFO  (qtp2049292927-50) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 9898 INFO  (qtp2049292927-50) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
>   [junit4]   2> 9942 INFO  (qtp1159626161-41) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 9944 INFO  (qtp1159626161-41) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
>   [junit4]   2> 10903 INFO  (qtp2049292927-54) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 10903 INFO  (qtp2049292927-54) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
>   [junit4]   2> 10946 INFO  (qtp1159626161-46) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 10948 INFO  (qtp1159626161-46) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
>   [junit4]   2> 11761 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema default search field in schema is text
>   [junit4]   2> 11762 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema unique key field: id
>   [junit4]   2> 11881 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
>   [junit4]   2> 11882 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
>   [junit4]   2> 11906 INFO  (qtp2049292927-55) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 11907 INFO  (qtp2049292927-55) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
>   [junit4]   2> 11952 INFO  (qtp1159626161-52) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 11953 INFO  (qtp1159626161-52) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
>   [junit4]   2> 12089 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/config', cwd=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0
>   [junit4]   2> 12096 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1  x:stateformat1_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'stateformat1_shard1_replica1' using configuration from collection stateformat1
>   [junit4]   2> 12099 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
>   [junit4]   2> 12223 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore [[stateformat1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/stateformat1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/]
>   [junit4]   2> 12235 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore JMX monitoring not detected for core: stateformat1_shard1_replica1
>   [junit4]   2> 12667 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data
>   [junit4]   2> 12678 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index/
>   [junit4]   2> 12678 WARN  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore [stateformat1_shard1_replica1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index' doesn't exist. Creating new index...
>   [junit4]   2> 12679 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/index
>   [junit4]   2> 12911 INFO  (qtp2049292927-29) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : 30dab8af-6efb-49d3-94d8-654355a865982375203249381381
>   [junit4]   2> 12911 INFO  (qtp2049292927-29) [n:127.0.0.1:64350_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=30dab8af-6efb-49d3-94d8-654355a865982375203249381381&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
>   [junit4]   2> 12957 INFO  (qtp1159626161-53) [n:127.0.0.1:64352_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>   [junit4]   2> 12961 INFO  (qtp1159626161-53) [n:127.0.0.1:64352_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=30dab8af-6efb-49d3-94d8-654355a86598&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=4
>   [junit4]   2> 13304 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
>   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6ea1ebcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1864dac4),segFN=segments_1,generation=1}
>   [junit4]   2> 13311 INFO  (parallelCoreAdminExecutor-20-thread-1-processing-n:127.0.0.1:64350_solr 30dab8af-6efb-49d3-94d8-654355a865982375203249381381 CREATE) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
>   [junit4]   2> 13312 INFO  (OldIndexDirectoryCleanupThreadForCore-stateformat1_shard1_replica1) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core stateformat1_shard1_replica1 in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/
>   [junit4]   2> 13314 WARN  (OldIndexDirectoryCleanupThreadForCore-stateformat1_shard1_replica1) [n:127.0.0.1:64350_solr c:stateformat1 s:shard1 r:core_node1 x:stateformat1_shard1_replica1] o.a.s.c.DirectoryFactory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node2/./stateformat1_shard1_replica1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
>   [junit4]   2> 13923 INFO  (qtp2049292927-40) [n:127.0.0.1:64350_solr    ] o.a.s.h.a.CoreA
> 
> [...truncated too long message...]
> 
> 0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001/tempDir-001/node4/./testcollection_shard1_replica1/data
>   [junit4]   2> 51330 INFO  (zkCallback-47-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delayed/state.json] for collection [delayed] has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 51330 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 51330 INFO  (zkCallback-41-thread-2-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 51330 WARN  (zkCallback-41-thread-2-processing-n:127.0.0.1:64350_solr) [n:127.0.0.1:64350_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
>   [junit4]   2> 51330 INFO  (zkCallback-47-thread-2-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [3])
>   [junit4]   2> 51331 WARN  (zkCallback-47-thread-2-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
>   [junit4]   2> 51331 WARN  (zkCallback-47-thread-3-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.LeaderElector 
>   [junit4]   2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)
>   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:105)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
>   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
>   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
>   [junit4]   2> 51332 INFO  (zkCallback-21-thread-2-processing-n:127.0.0.1:64352_solr) [n:127.0.0.1:64352_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
>   [junit4]   2> 51332 INFO  (jetty-launcher-2-thread-1) [    ] o.a.s.c.Overseer Overseer (id=96004804580671497-127.0.0.1:64352_solr-n_0000000000) closing
>   [junit4]   2> 51333 INFO  (jetty-launcher-2-thread-3) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4d07dd56{/solr,null,UNAVAILABLE}
>   [junit4]   2> 51333 INFO  (OverseerStateUpdate-96004804580671497-127.0.0.1:64352_solr-n_0000000000) [n:127.0.0.1:64352_solr    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:64352_solr
>   [junit4]   2> 51337 ERROR (jetty-launcher-2-thread-1-EventThread) [n:127.0.0.1:64352_solr    ] o.a.z.ClientCnxn Error while calling watcher 
>   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/81944345@1cc72bd7 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@59c5a62b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 300]
>   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
>   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
>   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
>   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
>   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
>   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>   [junit4]   2> 51338 INFO  (jetty-launcher-2-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@207554f2{/solr,null,UNAVAILABLE}
>   [junit4]   2> 52835 WARN  (zkCallback-47-thread-1-processing-n:127.0.0.1:64353_solr) [n:127.0.0.1:64353_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections/delayed/state.json]
>   [junit4]   2> 52836 INFO  (jetty-launcher-2-thread-2) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@91b95c4{/solr,null,UNAVAILABLE}
>   [junit4]   2> 52840 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64346 64346
>   [junit4]   2> 52860 INFO  (Thread-1) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:64346 64346
>   [junit4]   2> 54537 WARN  (Thread-1) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
>   [junit4]   2> Maximum concurrent create/delete watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	7	/solr/aliases.json
>   [junit4]   2> 	7	/solr/clusterprops.json
>   [junit4]   2> 	6	/solr/security.json
>   [junit4]   2> 	6	/solr/configs/config
>   [junit4]   2> 	3	/solr/collections/stateformat1/state.json
>   [junit4]   2> 
>   [junit4]   2> Maximum concurrent data watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	16	/solr/collections/falsepredicate/state.json
>   [junit4]   2> 	12	/solr/collections/waitforstate/state.json
>   [junit4]   2> 	7	/solr/clusterstate.json
>   [junit4]   2> 	7	/solr/collections/testcollection/state.json
>   [junit4]   2> 	3	/solr/overseer_elect/election/96004804580671497-127.0.0.1:64352_solr-n_0000000000
>   [junit4]   2> 	2	/solr/collections/delayed/state.json
>   [junit4]   2> 	2	/solr/collections/currentstate/state.json
>   [junit4]   2> 	2	/solr/overseer_elect/election/96004804580671499-127.0.0.1:64351_solr-n_0000000002
>   [junit4]   2> 	2	/solr/collections/tobedeleted/state.json
>   [junit4]   2> 
>   [junit4]   2> Maximum concurrent children watches above limit:
>   [junit4]   2> 
>   [junit4]   2> 	7	/solr/collections
>   [junit4]   2> 	5	/solr/live_nodes
>   [junit4]   2> 
>   [junit4]   2> 54537 INFO  (SUITE-TestCollectionStateWatchers-seed#[3011B606D36DAB04]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
>   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-solrj/test/J0/temp/solr.common.cloud.TestCollectionStateWatchers_3011B606D36DAB04-001
>   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62), sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=hi-IN, timezone=EST5EDT
>   [junit4]   2> NOTE: Mac OS X 10.11.4 x86_64/Oracle Corporation 1.8.0_72 (64-bit)/cpus=3,threads=1,free=45878088,total=105119744
>   [junit4]   2> NOTE: All tests run in this JVM: [CollectionAdminRequestRequiredParamsTest, TestCollectionStateWatchers]
>   [junit4] Completed [7/83 (1!)] on J0 in 56.69s, 9 tests, 1 failure <<< FAILURES!
> 
> [...truncated 238 lines...]
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:740: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:684: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:59: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:246: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:531: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1427: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:984: There were test failures: 83 suites, 641 tests, 1 failure, 2 ignored [seed: 3011B606D36DAB04]
> 
> Total time: 85 minutes 37 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> [WARNINGS] Skipping publisher since build result is FAILURE
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org