You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/01/09 00:50:40 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11546 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11546/
Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 7976 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:463: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:376: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/build.xml:65: impossible to resolve dependencies:
	resolve failed - see output for details

Total time: 32 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_25) - Build # 11549 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11549/
Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.testDistribSearch

Error Message:
Error from server at https://127.0.0.1:44277: CLUSTERSTATUS the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44277: CLUSTERSTATUS the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([348F48E0FC0297BF:B569C6F88B5DF783]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
	at org.apache.solr.cloud.MultiThreadedOCPTest.testLongAndShortRunningParallelApiCalls(MultiThreadedOCPTest.java:249)
	at org.apache.solr.cloud.MultiThreadedOCPTest.doTest(MultiThreadedOCPTest.java:75)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9155 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/init-core-data-001
   [junit4]   2> 516270 T3005 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 516270 T3005 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 516281 T3005 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 516282 T3005 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 516282 T3006 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 516382 T3005 oasc.ZkTestServer.run start zk server on port:56919
   [junit4]   2> 516383 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 516384 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516387 T3013 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0b6224 name:ZooKeeperConnection Watcher:127.0.0.1:56919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516387 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516388 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 516388 T3005 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 516391 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 516392 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516393 T3016 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@541074b0 name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516393 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516393 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 516394 T3005 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 516395 T3005 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 516396 T3005 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 516397 T3005 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 516398 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 516399 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 516401 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 516401 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 516403 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 516403 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 516404 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 516405 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 516406 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 516407 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 516408 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 516408 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 516410 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 516410 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 516412 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 516412 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 516414 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 516414 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 516415 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 516416 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 516417 T3005 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 516418 T3005 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 516420 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 516427 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516427 T3019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7647ebd5 name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516428 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516428 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 516837 T3005 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 516853 T3005 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@953de04{/,null,AVAILABLE}
   [junit4]   2> 516867 T3005 oejs.AbstractConnector.doStart Started ServerConnector@2f221533{SSL-http/1.1}{127.0.0.1:43170}
   [junit4]   2> 516867 T3005 oejs.Server.doStart Started @518329ms
   [junit4]   2> 516869 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 516870 T3005 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 516870 T3005 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002
   [junit4]   2> 516871 T3005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/'
   [junit4]   2> 516934 T3005 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/solr.xml
   [junit4]   2> 516948 T3005 oasc.CoreContainer.<init> New CoreContainer 536157303
   [junit4]   2> 516948 T3005 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/]
   [junit4]   2> 516949 T3005 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 516949 T3005 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 516950 T3005 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 516950 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 516950 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 516951 T3005 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 516951 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 516951 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 516952 T3005 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 516952 T3005 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 516953 T3005 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 516955 T3005 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 516955 T3005 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 516955 T3005 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 516956 T3005 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56919/solr
   [junit4]   2> 516956 T3005 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 516956 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 516958 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516959 T3033 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d3bd0f5 name:ZooKeeperConnection Watcher:127.0.0.1:56919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516959 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516959 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 516969 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 516970 T3036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56f4c2f7 name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 516970 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 516971 T3005 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 516973 T3005 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 516975 T3005 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 516981 T3005 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 516983 T3005 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 516985 T3005 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 516986 T3005 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43170_
   [junit4]   2> 516989 T3005 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43170_
   [junit4]   2> 516991 T3005 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 516992 T3005 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 516993 T3005 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 516995 T3005 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43170_
   [junit4]   2> 516995 T3005 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 516996 T3005 oasc.Overseer.start Overseer (id=93112576879034372-127.0.0.1:43170_-n_0000000000) starting
   [junit4]   2> 516998 T3005 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 517003 T3005 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 517021 T3038 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 517022 T3037 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 517026 T3005 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 517027 T3005 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 517028 T3005 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 517032 T3040 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 517033 T3040 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 517034 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 517035 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43170",
   [junit4]   2> 	  "node_name":"127.0.0.1:43170_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 517036 T3037 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43170",
   [junit4]   2> 	  "node_name":"127.0.0.1:43170_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 517036 T3037 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 517036 T3037 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 517037 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 517039 T3040 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 518040 T3040 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 518040 T3040 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 518040 T3040 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 518041 T3040 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 518041 T3040 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 518041 T3040 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/'
   [junit4]   2> 518042 T3040 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 518042 T3040 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 518043 T3040 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 518053 T3040 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 518055 T3040 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 518060 T3040 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 518073 T3040 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 518081 T3040 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 518081 T3040 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 518086 T3040 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 518221 T3040 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 518225 T3040 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 518227 T3040 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 518233 T3040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 518236 T3040 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 518237 T3040 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 518238 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 518238 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 518238 T3040 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 518239 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 518240 T3040 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 518240 T3040 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 518240 T3040 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 518240 T3040 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/data/
   [junit4]   2> 518241 T3040 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76eb4992
   [junit4]   2> 518241 T3040 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/data
   [junit4]   2> 518241 T3040 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/data/index/
   [junit4]   2> 518242 T3040 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 518242 T3040 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-002/collection1/data/index
   [junit4]   2> 518242 T3040 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.3984375, floorSegmentMB=1.7841796875, forceMergeDeletesPctAllowed=11.538031929360503, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 518243 T3040 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@698cb628 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@529929da),segFN=segments_1,generation=1}
   [junit4]   2> 518243 T3040 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 518245 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 518246 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 518246 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 518246 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 518246 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 518246 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 518247 T3040 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 518247 T3040 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 518247 T3040 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 518248 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 518248 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 518248 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 518248 T3040 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 518249 T3040 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 518249 T3040 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 518249 T3040 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 518250 T3040 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 518250 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 518250 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 518251 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 518251 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 518252 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 518252 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 518252 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 518253 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 518253 T3040 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 518253 T3040 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 518256 T3040 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 518257 T3040 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 518258 T3040 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 518259 T3040 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 518259 T3040 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 518260 T3040 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 518261 T3040 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 518261 T3040 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 518262 T3040 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@698cb628 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@529929da),segFN=segments_1,generation=1}
   [junit4]   2> 518262 T3040 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 518262 T3040 oass.SolrIndexSearcher.<init> Opening Searcher@7e2d1c49[collection1] main
   [junit4]   2> 518263 T3040 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 518263 T3040 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 518263 T3040 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 518264 T3040 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 518264 T3040 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 518264 T3040 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 518265 T3040 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 518265 T3040 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 518265 T3040 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 518265 T3040 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 518266 T3040 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 518266 T3041 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e2d1c49[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 518267 T3005 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 518267 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 518267 T3044 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:43170 collection:control_collection shard:shard1
   [junit4]   2> 518269 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 518270 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 518270 T3044 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 518271 T3047 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ca87fa name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 518271 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 518272 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 518273 T3005 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 518278 T3005 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 518281 T3044 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 518283 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 518284 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 518286 T3044 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 518286 T3044 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C642 name=collection1 org.apache.solr.core.SolrCore@346bebc3 url=https://127.0.0.1:43170/collection1 node=127.0.0.1:43170_ C642_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:43170, node_name=127.0.0.1:43170_, state=down}
   [junit4]   2> 518286 T3044 C642 P43170 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:43170/collection1/
   [junit4]   2> 518286 T3044 C642 P43170 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 518287 T3044 C642 P43170 oasc.SyncStrategy.syncToMe https://127.0.0.1:43170/collection1/ has no replicas
   [junit4]   2> 518287 T3044 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:43170/collection1/ shard1
   [junit4]   2> 518287 T3044 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 518290 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 518290 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43170",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 518394 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518395 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518445 T3044 oasc.ZkController.register We are https://127.0.0.1:43170/collection1/ and leader is https://127.0.0.1:43170/collection1/
   [junit4]   2> 518445 T3044 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:43170
   [junit4]   2> 518446 T3044 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 518446 T3044 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 518446 T3044 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 518448 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 518451 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43170",
   [junit4]   2> 	  "node_name":"127.0.0.1:43170_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 518452 T3037 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43170",
   [junit4]   2> 	  "node_name":"127.0.0.1:43170_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 518555 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518555 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 518620 T3005 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 518621 T3005 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 518622 T3005 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@40aedab9{/,null,AVAILABLE}
   [junit4]   2> 518624 T3005 oejs.AbstractConnector.doStart Started ServerConnector@4855a82a{SSL-http/1.1}{127.0.0.1:44277}
   [junit4]   2> 518624 T3005 oejs.Server.doStart Started @520087ms
   [junit4]   2> 518626 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 518626 T3005 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 518626 T3005 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003
   [junit4]   2> 518626 T3005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/'
   [junit4]   2> 518649 T3005 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/solr.xml
   [junit4]   2> 518658 T3005 oasc.CoreContainer.<init> New CoreContainer 1978043693
   [junit4]   2> 518659 T3005 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/]
   [junit4]   2> 518659 T3005 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 518660 T3005 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 518660 T3005 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 518660 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 518661 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 518661 T3005 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 518661 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 518661 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 518662 T3005 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 518662 T3005 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 518664 T3005 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 518665 T3005 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 518665 T3005 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 518666 T3005 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 518666 T3005 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56919/solr
   [junit4]   2> 518666 T3005 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 518667 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 518669 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 518693 T3061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bf65bd name:ZooKeeperConnection Watcher:127.0.0.1:56919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 518693 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 518694 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 518696 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 518697 T3064 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46c4a907 name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 518697 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 518701 T3005 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 519703 T3005 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44277_
   [junit4]   2> 519704 T3005 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44277_
   [junit4]   2> 519707 T3005 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 519709 T3065 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 519709 T3065 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 519709 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 519709 T3065 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 519710 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44277",
   [junit4]   2> 	  "node_name":"127.0.0.1:44277_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 519710 T3037 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44277",
   [junit4]   2> 	  "node_name":"127.0.0.1:44277_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 519711 T3037 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 519711 T3037 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 519812 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 519812 T3064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 519812 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 520710 T3065 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 520710 T3065 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 520711 T3065 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 520711 T3065 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 520712 T3065 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 520712 T3065 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/'
   [junit4]   2> 520712 T3065 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 520713 T3065 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 520713 T3065 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 520722 T3065 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 520723 T3065 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 520727 T3065 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 520736 T3065 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 520743 T3065 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 520744 T3065 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 520748 T3065 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 520863 T3065 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 520866 T3065 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 520867 T3065 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 520873 T3065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 520875 T3065 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 520876 T3065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 520877 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 520877 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 520877 T3065 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 520877 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 520878 T3065 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 520878 T3065 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 520878 T3065 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 520878 T3065 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/data/
   [junit4]   2> 520878 T3065 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76eb4992
   [junit4]   2> 520879 T3065 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/data
   [junit4]   2> 520879 T3065 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/data/index/
   [junit4]   2> 520879 T3065 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 520879 T3065 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-003/collection1/data/index
   [junit4]   2> 520879 T3065 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.3984375, floorSegmentMB=1.7841796875, forceMergeDeletesPctAllowed=11.538031929360503, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 520880 T3065 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d90ed50 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ba0c85c),segFN=segments_1,generation=1}
   [junit4]   2> 520880 T3065 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 520882 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 520882 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 520882 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 520882 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 520883 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 520883 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 520883 T3065 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 520883 T3065 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 520883 T3065 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 520884 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 520884 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 520884 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 520884 T3065 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 520885 T3065 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 520885 T3065 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 520885 T3065 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 520886 T3065 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 520886 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 520886 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 520886 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 520887 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 520887 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 520887 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 520888 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 520888 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 520888 T3065 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 520888 T3065 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 520890 T3065 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 520890 T3065 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 520891 T3065 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 520892 T3065 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 520892 T3065 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 520893 T3065 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 520893 T3065 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 520893 T3065 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 520894 T3065 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@d90ed50 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ba0c85c),segFN=segments_1,generation=1}
   [junit4]   2> 520894 T3065 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 520894 T3065 oass.SolrIndexSearcher.<init> Opening Searcher@67fb4e47[collection1] main
   [junit4]   2> 520894 T3065 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 520895 T3065 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 520895 T3065 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 520895 T3065 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 520895 T3065 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 520896 T3065 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 520896 T3065 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 520896 T3065 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 520896 T3065 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 520896 T3065 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 520897 T3066 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67fb4e47[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 520897 T3065 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 520898 T3069 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:44277 collection:collection1 shard:shard2
   [junit4]   2> 520898 T3005 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 520898 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 520898 T3069 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 520901 T3069 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 520902 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 520902 T3069 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 520902 T3069 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C643 name=collection1 org.apache.solr.core.SolrCore@36c3d75b url=https://127.0.0.1:44277/collection1 node=127.0.0.1:44277_ C643_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:44277, node_name=127.0.0.1:44277_, state=down}
   [junit4]   2> 520902 T3069 C643 P44277 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:44277/collection1/
   [junit4]   2> 520903 T3069 C643 P44277 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 520903 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 520903 T3069 C643 P44277 oasc.SyncStrategy.syncToMe https://127.0.0.1:44277/collection1/ has no replicas
   [junit4]   2> 520903 T3069 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:44277/collection1/ shard2
   [junit4]   2> 520903 T3069 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 520905 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 520906 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44277",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 521008 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521008 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521008 T3064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521056 T3069 oasc.ZkController.register We are https://127.0.0.1:44277/collection1/ and leader is https://127.0.0.1:44277/collection1/
   [junit4]   2> 521056 T3069 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:44277
   [junit4]   2> 521057 T3069 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 521057 T3069 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 521057 T3069 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 521058 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 521059 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44277",
   [junit4]   2> 	  "node_name":"127.0.0.1:44277_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 521059 T3037 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44277",
   [junit4]   2> 	  "node_name":"127.0.0.1:44277_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 521150 T3005 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 521151 T3005 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 521152 T3005 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7ca0703a{/,null,AVAILABLE}
   [junit4]   2> 521154 T3005 oejs.AbstractConnector.doStart Started ServerConnector@7f36141e{SSL-http/1.1}{127.0.0.1:41633}
   [junit4]   2> 521155 T3005 oejs.Server.doStart Started @522617ms
   [junit4]   2> 521156 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 521156 T3005 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 521156 T3005 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004
   [junit4]   2> 521156 T3005 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/'
   [junit4]   2> 521161 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521161 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521161 T3064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 521188 T3005 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/solr.xml
   [junit4]   2> 521197 T3005 oasc.CoreContainer.<init> New CoreContainer 2108412659
   [junit4]   2> 521198 T3005 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/]
   [junit4]   2> 521198 T3005 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 521198 T3005 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 521199 T3005 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 521199 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 521199 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 521200 T3005 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 521200 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 521200 T3005 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 521201 T3005 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 521201 T3005 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 521203 T3005 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 521204 T3005 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 521204 T3005 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 521204 T3005 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 521205 T3005 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56919/solr
   [junit4]   2> 521205 T3005 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 521205 T3005 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 521206 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 521207 T3083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78d0a5c4 name:ZooKeeperConnection Watcher:127.0.0.1:56919 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 521207 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 521208 T3005 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 521210 T3005 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 521211 T3086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41618cff name:ZooKeeperConnection Watcher:127.0.0.1:56919/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 521211 T3005 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 521214 T3005 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 522217 T3005 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41633_
   [junit4]   2> 522218 T3005 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41633_
   [junit4]   2> 522221 T3005 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 522224 T3087 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 522224 T3087 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 522225 T3036 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 522225 T3087 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 522225 T3037 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41633",
   [junit4]   2> 	  "node_name":"127.0.0.1:41633_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 6
   [junit4]   2> 522226 T3037 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:41633",
   [junit4]   2> 	  "node_name":"127.0.0.1:41633_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 522226 T3037 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 522226 T3037 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 522328 T3047 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 522328 T3086 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 522328 T3064 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 522328 T3036 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 523225 T3087 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 523225 T3087 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 523226 T3087 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 523226 T3087 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 523227 T3087 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 523227 T3087 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/'
   [junit4]   2> 523227 T3087 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 523227 T3087 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 523228 T3087 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest%20348F48E0FC0297BF-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 523238 T3087 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 523239 T3087 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 523242 T3087 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 523257 T3087 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 523265 T3087 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 523266 T3087 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 523270 T3087 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 523385 T3087 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 523388 T3087 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 523389 T3087 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 523395 T3087 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 523396 T3087 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 523398 T3087 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 523398 T3087 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 523399 T3087 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 523399 T3087 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 523399 T3087 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 523399 T3087 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 523399 T3087 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 523400 T3087 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 523400 T3087 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/data/
   [junit4]   2> 523400 T3087 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@76eb4992
   [junit4]   2> 523401 T3087 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/data
   [junit4]   2> 523401 T3087 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/data/index/
   [junit4]   2> 523401 T3087 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 523402 T3087 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-004/collection1/data/index
   [junit4]   2> 523402 T3087 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=64.3984375, floorSegmentMB=1.7841796875, forceMergeDeletesPctAllowed=11.538031929360503, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 523402 T3087 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ce8d339 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1a8902a0),segFN=segments_1,generation=1}
   [junit4]   2> 523403 T3087 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 523404 T3087 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 523405 T3087 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 523406 T3087 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 523406 T3087 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 523406 T3087 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 523406 T3087 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 523406 T3087 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 523407 T3087 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 523407 T3087 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 523407 T3087 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 523408 T3087 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 523408 T3087 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 523408 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 523409 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 523409 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 523409 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 523409 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 523410 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 523410 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 523410 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 523411 T3087 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 523411 T3087 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 523412 T3087 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 523413 T3087 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 523413 T3087 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 523414 T3087 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 523414 T3087 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 523415 T3087 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 523415 T3087 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 523416 T3087 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=46, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 523416 T3087 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3ce8d339 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1a8902a0),segFN=segments_1,generation=1}
   [junit4]   2> 523416 T3087 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 523416 T3087 oass.SolrIndexSearcher.<init> Opening Searcher@56a289a1[collection1] main
   [junit4]   2> 523417 T3087 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 523417 T3087 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 523417 T3087 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 523418 T3087 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 523418 T3087 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 523418 T3087 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 523418 T3087 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 523418 T3087 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 523418 T3087 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 523419 T3087 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 523419 T3088 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56a289a1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 523419 T3087 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 523420 T3091 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:41633 collection:collection1 shard:shard1
   [junit4]   2> 523420 T3005 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 523421 T3005 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 52

[...truncated too long message...]

.solr.core.SolrCore@c415e28
   [junit4]   2> 760851 T3005 oasc.ZkController.unRegisterConfListener  a listener was removed because of core close
   [junit4]   2> 760851 T3005 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 760851 T3005 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 760852 T3005 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 760852 T3005 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 760852 T3005 oasc.SolrCore.closeSearcher [ocptest_shardsplit2_shard2_1_replica1] Closing main searcher on request.
   [junit4]   2> 760867 T3005 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 760868 T3005 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data/index;done=false>>]
   [junit4]   2> 760868 T3005 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data/index
   [junit4]   2> 760869 T3005 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data;done=false>>]
   [junit4]   2> 760869 T3005 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001/tempDir-006/ocptest_shardsplit2_shard2_1_replica1/data
   [junit4]   2> 760871 T3005 oasc.Overseer.close Overseer (id=93112576879034381-127.0.0.1:60981_-n_0000000004) closing
   [junit4]   2> 760871 T3402 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:60981_
   [junit4]   2> 762382 T3395 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 762383 T3005 oejs.AbstractConnector.doStop Stopped ServerConnector@47aae8eb{SSL-http/1.1}{127.0.0.1:0}
   [junit4]   2> 762384 T3005 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@1109d796{/,null,UNAVAILABLE}
   [junit4]   2> 762384 T3005 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 762386 T3005 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56919 56919
   [junit4]   2> 763082 T3006 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:56919 56919
   [junit4]   2> 763084 T3006 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		5	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MultiThreadedOCPTest -Dtests.method=testDistribSearch -Dtests.seed=348F48E0FC0297BF -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_BA_#Latn -Dtests.timezone=Africa/Ndjamena -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    247s J1 | MultiThreadedOCPTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:44277: CLUSTERSTATUS the collection time out:180s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([348F48E0FC0297BF:B569C6F88B5DF783]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
   [junit4]    > 	at org.apache.solr.cloud.MultiThreadedOCPTest.testLongAndShortRunningParallelApiCalls(MultiThreadedOCPTest.java:249)
   [junit4]    > 	at org.apache.solr.cloud.MultiThreadedOCPTest.doTest(MultiThreadedOCPTest.java:75)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 763101 T3005 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest 348F48E0FC0297BF-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=BlockTreeOrds(blocksize=128), multiDefault=PostingsFormat(name=MockRandom), intDefault=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=Asserting), range_facet_l=BlockTreeOrds(blocksize=128), timestamp=BlockTreeOrds(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Lucene50)}, sim=DefaultSimilarity, locale=sr_BA_#Latn, timezone=Africa/Ndjamena
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=56888968,total=217374720
   [junit4]   2> NOTE: All tests run in this JVM: [RemoteQueryErrorTest, PrimitiveFieldTypeTest, TestDynamicFieldResource, ZkSolrClientTest, SolrCmdDistributorTest, TermsComponentTest, TestSearchPerf, ReturnFieldsTest, CoreMergeIndexesAdminHandlerTest, LeaderFailoverAfterPartitionTest, CollectionsAPIDistributedZkTest, TestLMJelinekMercerSimilarityFactory, TestConfig, TestSolrQueryParser, TestJmxMonitoredMap, FastVectorHighlighterTest, ResponseHeaderTest, TestSolrDeletionPolicy1, TestSolrIndexConfig, EnumFieldTest, TestConfigOverlay, ZkNodePropsTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestDefaultSearchFieldResource, CurrencyFieldOpenExchangeTest, StandardRequestHandlerTest, TestRequestStatusCollectionAPI, TestZkChroot, LeaderElectionIntegrationTest, TestSchemaSimilarityResource, IndexBasedSpellCheckerTest, DateRangeFieldTest, TestFileDictionaryLookup, ChaosMonkeySafeLeaderTest, SpatialRPTFieldTypeTest, RAMDirectoryFactoryTest, TestSolrXml, TestFoldingMultitermQuery, TestSuggestSpellingConverter, TestSolrDynamicMBean, TestImplicitCoreProperties, DistribDocExpirationUpdateProcessorTest, SpatialFilterTest, TestBadConfig, URLClassifyProcessorTest, XsltUpdateRequestHandlerTest, TestExactSharedStatsCache, TestBulkSchemaAPI, AddBlockUpdateTest, TestPostingsSolrHighlighter, TestBM25SimilarityFactory, TestSimpleQParserPlugin, SolrRequestParserTest, CustomCollectionTest, MultiThreadedOCPTest]
   [junit4] Completed on J1 in 246.87s, 1 test, 1 error <<< FAILURES!

[...truncated 941 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 460 suites, 1868 tests, 1 error, 47 ignored (22 assumptions)

Total time: 62 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b44) - Build # 11548 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11548/
Java: 64bit/jdk1.9.0-ea-b44 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
commitWithin did not work on node: http://127.0.0.1:42036/collection1 expected:<68> but was:<67>

Stack Trace:
java.lang.AssertionError: commitWithin did not work on node: http://127.0.0.1:42036/collection1 expected:<68> but was:<67>
	at __randomizedtesting.SeedInfo.seed([83EFE31591829766:2096D0DE6DDF75A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:345)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9004 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/init-core-data-001
   [junit4]   2> 449208 T3163 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 449210 T3163 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 449210 T3163 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 449211 T3164 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 449311 T3163 oasc.ZkTestServer.run start zk server on port:58510
   [junit4]   2> 449311 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449317 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449320 T3171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@235d18c4 name:ZooKeeperConnection Watcher:127.0.0.1:58510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449320 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449321 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449321 T3163 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 449327 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449327 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449329 T3174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79c14770 name:ZooKeeperConnection Watcher:127.0.0.1:58510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449329 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449329 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449329 T3163 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 449331 T3163 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 449332 T3163 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 449333 T3163 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 449334 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 449335 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 449336 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 449337 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 449341 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 449342 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 449343 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 449343 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 449344 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 449345 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 449346 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 449346 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 449347 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 449347 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 449349 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 449349 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 449351 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 449351 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 449352 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 449353 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 449354 T3163 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 449354 T3163 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 449616 T3163 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 449617 T3163 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@17e07c59{/,null,AVAILABLE}
   [junit4]   2> 449618 T3163 oejs.AbstractConnector.doStart Started ServerConnector@65514115{HTTP/1.1}{127.0.0.1:44831}
   [junit4]   2> 449618 T3163 oejs.Server.doStart Started @450757ms
   [junit4]   2> 449619 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@726f3b58
   [junit4]   2> 449619 T3163 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 449619 T3163 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002
   [junit4]   2> 449620 T3163 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002/'
   [junit4]   2> 449629 T3163 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002/solr.xml
   [junit4]   2> 449642 T3163 oasc.CoreContainer.<init> New CoreContainer 2013157250
   [junit4]   2> 449642 T3163 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002/]
   [junit4]   2> 449642 T3163 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 449642 T3163 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 449643 T3163 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 449643 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 449643 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 449643 T3163 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 449644 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 449644 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 449644 T3163 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 449644 T3163 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 449644 T3163 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 449645 T3163 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 449645 T3163 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 449645 T3163 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 449645 T3163 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58510/solr
   [junit4]   2> 449646 T3163 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 449646 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 449646 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449648 T3188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a2bf5a6 name:ZooKeeperConnection Watcher:127.0.0.1:58510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449648 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449648 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 449649 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449650 T3191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dd34fa8 name:ZooKeeperConnection Watcher:127.0.0.1:58510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449651 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449651 T3163 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 449652 T3163 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 449654 T3163 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 449655 T3163 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 449656 T3163 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 449661 T3163 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 449662 T3163 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44831_
   [junit4]   2> 449663 T3163 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44831_
   [junit4]   2> 449665 T3163 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 449666 T3163 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 449667 T3163 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 449668 T3163 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44831_
   [junit4]   2> 449668 T3163 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 449669 T3163 oasc.Overseer.start Overseer (id=93112002223079427-127.0.0.1:44831_-n_0000000000) starting
   [junit4]   2> 449671 T3163 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 449675 T3163 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 449676 T3193 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 449676 T3163 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 449677 T3163 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 449678 T3163 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 449680 T3192 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 449682 T3195 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 449682 T3195 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 449682 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 449682 T3195 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 449684 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44831",
   [junit4]   2> 	  "node_name":"127.0.0.1:44831_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 449684 T3192 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44831",
   [junit4]   2> 	  "node_name":"127.0.0.1:44831_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 449684 T3192 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 449685 T3192 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 449685 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 450683 T3195 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 450683 T3195 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 450684 T3195 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 450684 T3195 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 450684 T3195 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 450685 T3195 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002/collection1/'
   [junit4]   2> 450685 T3195 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 450685 T3195 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 450685 T3195 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 450695 T3195 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 450697 T3195 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 450703 T3195 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 450708 T3195 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 450726 T3195 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 450728 T3195 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 450731 T3195 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 450842 T3195 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 450845 T3195 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 450846 T3195 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 450852 T3195 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450854 T3195 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 450856 T3195 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450856 T3195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450857 T3195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450857 T3195 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 450857 T3195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 450858 T3195 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 450858 T3195 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 450858 T3195 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 450858 T3195 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/control/data/
   [junit4]   2> 450858 T3195 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c5c00ce
   [junit4]   2> 450859 T3195 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/control/data
   [junit4]   2> 450859 T3195 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/control/data/index/
   [junit4]   2> 450859 T3195 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 450859 T3195 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/control/data/index
   [junit4]   2> 450859 T3195 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4540401154164466]
   [junit4]   2> 450860 T3195 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@39ea09d3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41d79218),segFN=segments_1,generation=1}
   [junit4]   2> 450860 T3195 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450862 T3195 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 450862 T3195 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 450862 T3195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 450862 T3195 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450862 T3195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 450863 T3195 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 450863 T3195 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450863 T3195 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 450863 T3195 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 450863 T3195 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 450864 T3195 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 450864 T3195 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 450864 T3195 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 450864 T3195 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 450864 T3195 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 450865 T3195 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 450865 T3195 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 450865 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 450866 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 450866 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 450866 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 450866 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 450866 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 450867 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 450867 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 450867 T3195 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 450867 T3195 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 450869 T3195 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450869 T3195 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450870 T3195 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450870 T3195 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 450871 T3195 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 450872 T3195 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 450872 T3195 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 450872 T3195 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=74.9638671875, floorSegmentMB=1.9130859375, forceMergeDeletesPctAllowed=24.791266271844318, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 450873 T3195 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@39ea09d3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41d79218),segFN=segments_1,generation=1}
   [junit4]   2> 450873 T3195 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 450873 T3195 oass.SolrIndexSearcher.<init> Opening Searcher@b6d4bf[collection1] main
   [junit4]   2> 450873 T3195 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 450874 T3195 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 450874 T3195 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 450874 T3195 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 450874 T3195 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 450875 T3195 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450875 T3195 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 450875 T3195 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 450875 T3195 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 450875 T3195 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 450876 T3195 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 450876 T3196 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b6d4bf[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 450876 T3199 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44831 collection:control_collection shard:shard1
   [junit4]   2> 450877 T3163 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 450877 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450877 T3199 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 450878 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 450878 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 450879 T3202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e04e32c name:ZooKeeperConnection Watcher:127.0.0.1:58510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 450880 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 450880 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 450880 T3199 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 450880 T3163 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 450881 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 450882 T3199 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 450882 T3199 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 450882 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> ASYNC  NEW_CORE C1126 name=collection1 org.apache.solr.core.SolrCore@46bfae04 url=http://127.0.0.1:44831/collection1 node=127.0.0.1:44831_ C1126_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:44831, node_name=127.0.0.1:44831_, state=down}
   [junit4]   2> 450882 T3199 C1126 P44831 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44831/collection1/
   [junit4]   2> 450883 T3199 C1126 P44831 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 450883 T3199 C1126 P44831 oasc.SyncStrategy.syncToMe http://127.0.0.1:44831/collection1/ has no replicas
   [junit4]   2> 450883 T3199 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44831/collection1/ shard1
   [junit4]   2> 450883 T3163 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 450883 T3199 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 450890 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 450891 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44831",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 450994 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 450994 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 451042 T3199 oasc.ZkController.register We are http://127.0.0.1:44831/collection1/ and leader is http://127.0.0.1:44831/collection1/
   [junit4]   2> 451042 T3199 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44831
   [junit4]   2> 451042 T3199 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 451042 T3199 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 451043 T3199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 451044 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 451045 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44831",
   [junit4]   2> 	  "node_name":"127.0.0.1:44831_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 451045 T3192 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44831",
   [junit4]   2> 	  "node_name":"127.0.0.1:44831_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 451138 T3163 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 451141 T3163 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 451142 T3163 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@6b8e22b8{/,null,AVAILABLE}
   [junit4]   2> 451143 T3163 oejs.AbstractConnector.doStart Started ServerConnector@775c19d5{HTTP/1.1}{127.0.0.1:42036}
   [junit4]   2> 451143 T3163 oejs.Server.doStart Started @452282ms
   [junit4]   2> 451143 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@726f3b58
   [junit4]   2> 451144 T3163 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 451144 T3163 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003
   [junit4]   2> 451144 T3163 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003/'
   [junit4]   2> 451147 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 451147 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 451154 T3163 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003/solr.xml
   [junit4]   2> 451170 T3163 oasc.CoreContainer.<init> New CoreContainer 1387067435
   [junit4]   2> 451170 T3163 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003/]
   [junit4]   2> 451171 T3163 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 451171 T3163 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 451171 T3163 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 451172 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 451172 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 451172 T3163 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 451173 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 451173 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 451173 T3163 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 451173 T3163 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 451174 T3163 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 451175 T3163 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 451175 T3163 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 451175 T3163 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 451175 T3163 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58510/solr
   [junit4]   2> 451176 T3163 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 451176 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 451177 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451178 T3216 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18955e13 name:ZooKeeperConnection Watcher:127.0.0.1:58510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451179 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451180 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 451182 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451183 T3219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7705db18 name:ZooKeeperConnection Watcher:127.0.0.1:58510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451183 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451187 T3163 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 452192 T3163 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42036_
   [junit4]   2> 452193 T3163 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42036_
   [junit4]   2> 452195 T3163 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 452198 T3220 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 452198 T3220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 452199 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 452199 T3220 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 452199 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42036",
   [junit4]   2> 	  "node_name":"127.0.0.1:42036_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 452200 T3192 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42036",
   [junit4]   2> 	  "node_name":"127.0.0.1:42036_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 452200 T3192 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 452200 T3192 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 452302 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 452302 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 452302 T3219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453199 T3220 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 453199 T3220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 453200 T3220 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 453200 T3220 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 453201 T3220 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 453201 T3220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003/collection1/'
   [junit4]   2> 453201 T3220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 453201 T3220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 453202 T3220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 453211 T3220 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 453212 T3220 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 453216 T3220 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 453222 T3220 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 453239 T3220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453240 T3220 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 453245 T3220 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 453391 T3220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 453396 T3220 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 453397 T3220 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 453405 T3220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 453407 T3220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 453410 T3220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 453411 T3220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 453411 T3220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 453412 T3220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 453412 T3220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 453412 T3220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 453413 T3220 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 453413 T3220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 453413 T3220 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty1/
   [junit4]   2> 453413 T3220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c5c00ce
   [junit4]   2> 453414 T3220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty1
   [junit4]   2> 453414 T3220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty1/index/
   [junit4]   2> 453414 T3220 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 453415 T3220 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty1/index
   [junit4]   2> 453415 T3220 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4540401154164466]
   [junit4]   2> 453416 T3220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2cd37d75 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d2ceed6),segFN=segments_1,generation=1}
   [junit4]   2> 453416 T3220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 453419 T3220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 453419 T3220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 453420 T3220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 453420 T3220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 453420 T3220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 453420 T3220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 453420 T3220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 453421 T3220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 453421 T3220 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 453421 T3220 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 453422 T3220 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 453422 T3220 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 453422 T3220 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 453423 T3220 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 453423 T3220 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 453424 T3220 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 453425 T3220 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 453425 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 453426 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 453426 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 453426 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 453427 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 453427 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 453427 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 453428 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 453428 T3220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 453429 T3220 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 453431 T3220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 453432 T3220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 453432 T3220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 453433 T3220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 453434 T3220 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 453435 T3220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 453435 T3220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 453436 T3220 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=74.9638671875, floorSegmentMB=1.9130859375, forceMergeDeletesPctAllowed=24.791266271844318, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 453437 T3220 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2cd37d75 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d2ceed6),segFN=segments_1,generation=1}
   [junit4]   2> 453437 T3220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 453437 T3220 oass.SolrIndexSearcher.<init> Opening Searcher@19feb6fe[collection1] main
   [junit4]   2> 453437 T3220 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 453438 T3220 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 453438 T3220 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 453439 T3220 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 453439 T3220 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 453439 T3220 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 453439 T3220 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 453440 T3220 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 453440 T3220 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 453440 T3220 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 453441 T3221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19feb6fe[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 453441 T3220 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 453442 T3224 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42036 collection:collection1 shard:shard2
   [junit4]   2> 453442 T3163 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 453442 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 453442 T3224 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 453450 T3224 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 453451 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 453451 T3224 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 453452 T3224 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 453452 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C1127 name=collection1 org.apache.solr.core.SolrCore@7004e39c url=http://127.0.0.1:42036/collection1 node=127.0.0.1:42036_ C1127_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42036, node_name=127.0.0.1:42036_, state=down}
   [junit4]   2> 453452 T3224 C1127 P42036 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42036/collection1/
   [junit4]   2> 453452 T3224 C1127 P42036 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 453453 T3224 C1127 P42036 oasc.SyncStrategy.syncToMe http://127.0.0.1:42036/collection1/ has no replicas
   [junit4]   2> 453453 T3224 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42036/collection1/ shard2
   [junit4]   2> 453453 T3224 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 453455 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 453456 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42036",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 453558 T3219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453558 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453558 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453591 T3224 oasc.ZkController.register We are http://127.0.0.1:42036/collection1/ and leader is http://127.0.0.1:42036/collection1/
   [junit4]   2> 453591 T3224 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42036
   [junit4]   2> 453591 T3224 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 453592 T3224 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 453592 T3224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 453593 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 453594 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42036",
   [junit4]   2> 	  "node_name":"127.0.0.1:42036_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 453596 T3192 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42036",
   [junit4]   2> 	  "node_name":"127.0.0.1:42036_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 453698 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453698 T3219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453699 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 453723 T3163 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 453724 T3163 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 453725 T3163 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7c3b4981{/,null,AVAILABLE}
   [junit4]   2> 453726 T3163 oejs.AbstractConnector.doStart Started ServerConnector@2a520cae{HTTP/1.1}{127.0.0.1:43098}
   [junit4]   2> 453726 T3163 oejs.Server.doStart Started @454865ms
   [junit4]   2> 453726 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@726f3b58
   [junit4]   2> 453727 T3163 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 453727 T3163 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004
   [junit4]   2> 453727 T3163 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004/'
   [junit4]   2> 453737 T3163 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004/solr.xml
   [junit4]   2> 453757 T3163 oasc.CoreContainer.<init> New CoreContainer 1707482491
   [junit4]   2> 453757 T3163 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004/]
   [junit4]   2> 453758 T3163 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 453758 T3163 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 453758 T3163 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 453759 T3163 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 453760 T3163 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 453760 T3163 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 453760 T3163 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 453761 T3163 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 453761 T3163 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 453761 T3163 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58510/solr
   [junit4]   2> 453761 T3163 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 453761 T3163 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 453762 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453764 T3238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@722790c name:ZooKeeperConnection Watcher:127.0.0.1:58510 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453764 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453764 T3163 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 453766 T3163 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 453767 T3241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@390bc088 name:ZooKeeperConnection Watcher:127.0.0.1:58510/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 453767 T3163 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 453770 T3163 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 454773 T3163 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43098_
   [junit4]   2> 454774 T3163 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43098_
   [junit4]   2> 454777 T3163 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 454780 T3242 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 454781 T3242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 454781 T3242 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 454781 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 454782 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43098",
   [junit4]   2> 	  "node_name":"127.0.0.1:43098_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 6
   [junit4]   2> 454783 T3192 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43098",
   [junit4]   2> 	  "node_name":"127.0.0.1:43098_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 454783 T3192 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 454783 T3192 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 454885 T3202 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 454885 T3191 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 454885 T3241 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 454885 T3219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 455782 T3242 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 455782 T3242 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 455782 T3242 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 455782 T3242 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 455783 T3242 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 455783 T3242 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004/collection1/'
   [junit4]   2> 455784 T3242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 455784 T3242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 455784 T3242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest%2083EFE31591829766-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 455793 T3242 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 455795 T3242 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 455798 T3242 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 455802 T3242 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 455815 T3242 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 455816 T3242 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 455819 T3242 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 455936 T3242 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455939 T3242 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 455941 T3242 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 455948 T3242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455950 T3242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455952 T3242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455953 T3242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455953 T3242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455954 T3242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455954 T3242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455954 T3242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455954 T3242 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 455955 T3242 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 455955 T3242 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty2/
   [junit4]   2> 455955 T3242 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c5c00ce
   [junit4]   2> 455956 T3242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty2
   [junit4]   2> 455956 T3242 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty2/index/
   [junit4]   2> 455956 T3242 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 455956 T3242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty2/index
   [junit4]   2> 455956 T3242 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=22, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4540401154164466]
   [junit4]   2> 455957 T3242 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@55d3f813 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53a3a31f),segFN=segments_1,generation=1}
   [junit4]   2> 455957 T3242 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455959 T3242 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455960 T3242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455961 T3242 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 455961 T3242 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 455961 T3242 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 455961 T3242 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 455962 T3242 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 455962 T3242 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 455962 T3242 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 455963 T3242 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 455963 T3242 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 455964 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 455964 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 455965 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 455965 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 455965 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 455966 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 455966 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 455966 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 455967 T3242 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 455967 T3242 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 455969 T3242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 455969 T3242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 455970 T3242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 455970 T3242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 455971 T3242 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 455972 T3242 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 455972 T3242 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 455973 T3242 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=14, maxMergedSegmentMB=74.9638671875, floorSegmentMB=1.9130859375, forceMergeDeletesPctAllowed=24.791266271844318, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 455973 T3242 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@55d3f813 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53a3a31f),segFN=segments_1,generation=1}
   [junit4]   2> 455973 T3242 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455974 T3242 oass.SolrIndexSearcher.<init> Opening Searcher@3f8ddc5a[collection1] main
   [junit4]   2> 455974 T3242 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 455974 T3242 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 455975 T3242 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 455975 T3242 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 455975 T3242 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 455975 T3242 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 455975 T3242 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 455976 T3242 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 455976 T3242 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 455976 T3242 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 455977 T3243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f8ddc5a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 455977 T3242 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 455978 T3246 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43098 collection:collection1 shard:shard1
   [junit4]   2> 455978 T3163 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 455978 T3163 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 455978 T3246 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 455981 T3246 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 455981 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 455982 T3246 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 455982 T3246 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1128 name=collection1 org.apache.solr.core.SolrCore@7f24c66 url=http://127.0.0.1:43098/collection1 node=127.0.0.1:43098_ C1128_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:43098, node_name=127.0.0.1:43098_, state=down}
   [junit4]   2> 455982 T3246 C1128 P43098 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43098/collection1/
   [junit4]   2> 455982 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 7
   [junit4]   2> 455982 T3246 C1128 P43098 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 455982 T3246 C1128 P43098 oasc.SyncStrategy.syncToMe http://127.0.0.1:43098/collection1/ has no replicas
   [junit4]   2> 455982 T3246 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43098/collection1/ shard1
   [junit4]   2> 455982 T3246 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 455984 T3191 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 455985 T3192 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "sha

[...truncated too long message...]

runk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001/tempDir-001/jetty4
   [junit4]   2> 473199 T3163 oejs.AbstractConnector.doStop Stopped ServerConnector@16a22237{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 473200 T3163 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@136348aa{/,null,UNAVAILABLE}
   [junit4]   2> 473200 T3163 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 473201 T3163 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58510 58510
   [junit4]   2> 473743 T3164 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58510 58510
   [junit4]   2> 473744 T3164 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/aliases.json
   [junit4]   2> 		7	/solr/clusterstate.json
   [junit4]   2> 		5	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/live_nodes
   [junit4]   2> 		4	/solr/overseer/queue
   [junit4]   2> 		4	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> 473843 T3325 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@26497183 rejected from java.util.concurrent.ThreadPoolExecutor@634bfb2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2049)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:264)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 473844 T3325 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5262bc9b rejected from java.util.concurrent.ThreadPoolExecutor@634bfb2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2049)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:264)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 473844 T3325 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3640461 rejected from java.util.concurrent.ThreadPoolExecutor@634bfb2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 13]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2049)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:264)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=83EFE31591829766 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_US -Dtests.timezone=America/Scoresbysund -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 24.6s J0 | BasicDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: commitWithin did not work on node: http://127.0.0.1:42036/collection1 expected:<68> but was:<67>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([83EFE31591829766:2096D0DE6DDF75A]:0)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:345)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 473850 T3163 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.BasicDistributedZkTest 83EFE31591829766-001
   [junit4]   2> 24646 T3162 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {other_tl1=PostingsFormat(name=Direct), SubjectTerms_mfacet=PostingsFormat(name=Direct), n_l1=PostingsFormat(name=LuceneFixedGap), multiDefault=PostingsFormat(name=LuceneFixedGap), intDefault=PostingsFormat(name=Direct), n_td1=PostingsFormat(name=Direct), n_d1=PostingsFormat(name=LuceneFixedGap), range_facet_l=PostingsFormat(name=LuceneFixedGap), n_f1=PostingsFormat(name=Direct), n_tl1=PostingsFormat(name=Direct), oddField_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), n_tf1=PostingsFormat(name=LuceneFixedGap), a_t=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=Asserting), timestamp=PostingsFormat(name=LuceneFixedGap), foo_i=PostingsFormat(name=Direct), n_dt1=PostingsFormat(name=Direct), a_i1=PostingsFormat(name=LuceneFixedGap), n_ti1=PostingsFormat(name=LuceneVarGapDocFreqInterval), rnd_b=PostingsFormat(name=LuceneVarGapDocFreqInterval), _version_=PostingsFormat(name=Direct), foo_b=PostingsFormat(name=Asserting), n_tdt1=PostingsFormat(name=Direct), foo_d=PostingsFormat(name=LuceneVarGapDocFreqInterval), foo_f=PostingsFormat(name=Asserting)}, docValues:{timestamp=DocValuesFormat(name=Lucene50)}, sim=DefaultSimilarity, locale=en_US, timezone=America/Scoresbysund
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=141919976,total=294301696
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedQueryComponentOptimizationTest, SolrCoreTest, TestRecoveryHdfs, HdfsWriteToMultipleCollectionsTest, FieldAnalysisRequestHandlerTest, TestCollapseQParserPlugin, TestStandardQParsers, TestCollationFieldDocValues, TestFunctionQuery, TestSolrConfigHandlerCloud, TestPhraseSuggestions, SoftAutoCommitTest, TestXIncludeConfig, TestSchemaResource, NotRequiredUniqueKeyTest, AnalysisAfterCoreReloadTest, SuggesterFSTTest, TestShortCircuitedRequests, HdfsLockFactoryTest, TestStressVersions, DirectUpdateHandlerTest, RollingRestartTest, TestDefaultStatsCache, ResourceLoaderTest, TestOmitPositions, HighlighterConfigTest, AnalyticsMergeStrategyTest, HardAutoCommitTest, TestCloudInspectUtil, PeerSyncTest, TestSort, ConnectionManagerTest, DeleteInactiveReplicaTest, TestSweetSpotSimilarityFactory, BlockDirectoryTest, ClusterStateUpdateTest, LegacyHTMLStripCharFilterTest, DistributedQueryComponentCustomSortTest, AssignTest, TestPartialUpdateDeduplication, TestIBSimilarityFactory, DocValuesMultiTest, UnloadDistributedZkTest, AlternateDirectoryTest, PrimUtilsTest, OverseerTest, SolrInfoMBeanTest, DistributedSpellCheckComponentTest, ShardRoutingTest, DistribCursorPagingTest, DocValuesTest, BasicDistributedZkTest]
   [junit4] Completed on J0 in 24.92s, 1 test, 1 failure <<< FAILURES!

[...truncated 1094 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 460 suites, 1868 tests, 1 failure, 47 ignored (22 assumptions)

Total time: 60 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b44 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b20) - Build # 11547 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11547/
Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
some core start times did not change on reload

Stack Trace:
java.lang.AssertionError: some core start times did not change on reload
	at __randomizedtesting.SeedInfo.seed([2E00F5D6B0ABCC8B:AFE67BCEC7F4ACB7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:750)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:199)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9335 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/init-core-data-001
   [junit4]   2> 905053 T4841 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 905053 T4841 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 905056 T4841 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 905057 T4841 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 905057 T4842 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 905157 T4841 oasc.ZkTestServer.run start zk server on port:55658
   [junit4]   2> 905158 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905159 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905161 T4849 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@442bf832 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905161 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905161 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905161 T4841 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 905163 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905164 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905165 T4852 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@124c4170 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905165 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905166 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905166 T4841 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 905168 T4841 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 905170 T4841 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 905171 T4841 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 905172 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 905172 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 905174 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 905174 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 905175 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 905175 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 905176 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 905177 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 905177 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 905178 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 905179 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 905179 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 905180 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 905180 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 905181 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 905181 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 905182 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 905183 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 905183 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 905184 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 905185 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 905185 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 905186 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905187 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905187 T4855 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74173fa6 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905187 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905188 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905188 T4841 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 905189 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905189 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905190 T4858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51af96d9 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905190 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905190 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905191 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 905191 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 905192 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 905193 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 905194 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 905194 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 905196 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 905196 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 905197 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 905198 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 905199 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 905199 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 905201 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 905201 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 905203 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 905204 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 905205 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 905206 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 905207 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 905208 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 905209 T4841 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 905210 T4841 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 905461 T4841 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 905462 T4841 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@6cc4ba24{/,null,AVAILABLE}
   [junit4]   2> 905469 T4841 oejs.AbstractConnector.doStart Started ServerConnector@44ed49e{HTTP/1.1}{127.0.0.1:४९१०८}
   [junit4]   2> 905469 T4841 oejs.Server.doStart Started @९०६७७६ms
   [junit4]   2> 905470 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 905470 T4841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 905470 T4841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002
   [junit4]   2> 905470 T4841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/'
   [junit4]   2> 905480 T4841 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/solr.xml
   [junit4]   2> 905507 T4841 oasc.CoreContainer.<init> New CoreContainer 347002424
   [junit4]   2> 905508 T4841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/]
   [junit4]   2> 905508 T4841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 905509 T4841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 905509 T4841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 905509 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 905510 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 905510 T4841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 905511 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 905511 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 905511 T4841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 905511 T4841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 905512 T4841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 905512 T4841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 905513 T4841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 905513 T4841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 905513 T4841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55658/solr
   [junit4]   2> 905514 T4841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 905514 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905515 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905516 T4872 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ae4a149 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905516 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905516 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905518 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905519 T4875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69cb3dad name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905519 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905520 T4841 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 905521 T4841 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 905523 T4841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 905524 T4841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 905526 T4841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 905527 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 905528 T4841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49108_
   [junit4]   2> 905528 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49108_
   [junit4]   2> 905530 T4841 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 905531 T4841 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 905532 T4841 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 905533 T4841 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49108_
   [junit4]   2> 905533 T4841 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 905534 T4841 oasc.Overseer.start Overseer (id=93111499914412037-127.0.0.1:49108_-n_0000000000) starting
   [junit4]   2> 905536 T4841 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 905540 T4841 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 905541 T4877 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 905541 T4841 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 905542 T4841 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 905543 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 905545 T4876 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 905546 T4841 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 905547 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 905547 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905548 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905549 T4882 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bd23a73 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905549 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905549 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905550 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 905552 T4841 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 905553 T4841 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 905553 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905554 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905554 T4885 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@211f3788 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905554 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905555 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905556 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 905558 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 905558 T4876 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 905558 T4876 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 905559 T4875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 905559 T4882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 905865 T4841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 905866 T4841 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 905867 T4841 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@464054e{/,null,AVAILABLE}
   [junit4]   2> 905868 T4841 oejs.AbstractConnector.doStart Started ServerConnector@6ec67583{HTTP/1.1}{127.0.0.1:५४३७०}
   [junit4]   2> 905868 T4841 oejs.Server.doStart Started @९०७१७५ms
   [junit4]   2> 905868 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 905869 T4841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 905869 T4841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003
   [junit4]   2> 905869 T4841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/'
   [junit4]   2> 905880 T4841 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/solr.xml
   [junit4]   2> 905887 T4841 oasc.CoreContainer.<init> New CoreContainer 1394817990
   [junit4]   2> 905888 T4841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/]
   [junit4]   2> 905888 T4841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 905889 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 905890 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 905890 T4841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 905890 T4841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 905891 T4841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 905891 T4841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 905891 T4841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 905891 T4841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 905891 T4841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55658/solr
   [junit4]   2> 905892 T4841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 905892 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 905892 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905894 T4899 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@967cff4 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905894 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905894 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 905896 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 905897 T4902 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30ea37d9 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 905897 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 905899 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 906904 T4841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54370_
   [junit4]   2> 906905 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54370_
   [junit4]   2> 906910 T4841 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 906912 T4841 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 906912 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 907170 T4841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 907171 T4841 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 907172 T4841 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@3f06057b{/,null,AVAILABLE}
   [junit4]   2> 907172 T4841 oejs.AbstractConnector.doStart Started ServerConnector@733ab1a1{HTTP/1.1}{127.0.0.1:३६३६८}
   [junit4]   2> 907173 T4841 oejs.Server.doStart Started @९०८४७९ms
   [junit4]   2> 907173 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 907173 T4841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 907173 T4841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004
   [junit4]   2> 907174 T4841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/'
   [junit4]   2> 907184 T4841 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/solr.xml
   [junit4]   2> 907191 T4841 oasc.CoreContainer.<init> New CoreContainer 829017930
   [junit4]   2> 907191 T4841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/]
   [junit4]   2> 907192 T4841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 907192 T4841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 907192 T4841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 907192 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 907193 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 907193 T4841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 907193 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 907193 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 907193 T4841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 907194 T4841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 907194 T4841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 907194 T4841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 907194 T4841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 907195 T4841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 907195 T4841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55658/solr
   [junit4]   2> 907195 T4841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 907195 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 907196 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 907197 T4917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a2dc805 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 907197 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 907197 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 907199 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 907200 T4920 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@720d6072 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 907200 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 907204 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 908206 T4841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36368_
   [junit4]   2> 908207 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36368_
   [junit4]   2> 908210 T4841 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 908212 T4841 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 908213 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 908471 T4841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 908472 T4841 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 908474 T4841 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4551b0f2{/,null,AVAILABLE}
   [junit4]   2> 908474 T4841 oejs.AbstractConnector.doStart Started ServerConnector@401eb87a{HTTP/1.1}{127.0.0.1:५९१९४}
   [junit4]   2> 908474 T4841 oejs.Server.doStart Started @९०९७८१ms
   [junit4]   2> 908475 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 908475 T4841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 908475 T4841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005
   [junit4]   2> 908476 T4841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/'
   [junit4]   2> 908486 T4841 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/solr.xml
   [junit4]   2> 908493 T4841 oasc.CoreContainer.<init> New CoreContainer 1406077458
   [junit4]   2> 908493 T4841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/]
   [junit4]   2> 908494 T4841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 908494 T4841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 908494 T4841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 908494 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 908495 T4841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 908496 T4841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 908496 T4841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 908496 T4841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 908496 T4841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 908497 T4841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55658/solr
   [junit4]   2> 908497 T4841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 908497 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 908498 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 908499 T4935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5eea5fa1 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 908499 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 908499 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 908501 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 908501 T4938 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@145e03d name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 908501 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 908504 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 909507 T4841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59194_
   [junit4]   2> 909508 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59194_
   [junit4]   2> 909511 T4841 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 909513 T4841 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 909513 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 909766 T4841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 909767 T4841 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 909769 T4841 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@fd024d4{/,null,AVAILABLE}
   [junit4]   2> 909769 T4841 oejs.AbstractConnector.doStart Started ServerConnector@69f42c5{HTTP/1.1}{127.0.0.1:४१७८५}
   [junit4]   2> 909770 T4841 oejs.Server.doStart Started @९११०७६ms
   [junit4]   2> 909770 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 909771 T4841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 909771 T4841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006
   [junit4]   2> 909771 T4841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/'
   [junit4]   2> 909783 T4841 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/solr.xml
   [junit4]   2> 909790 T4841 oasc.CoreContainer.<init> New CoreContainer 1019640708
   [junit4]   2> 909790 T4841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/]
   [junit4]   2> 909791 T4841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 909791 T4841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 909791 T4841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 909791 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 909792 T4841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 909793 T4841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 909793 T4841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 909793 T4841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 909793 T4841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 909794 T4841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55658/solr
   [junit4]   2> 909794 T4841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 909794 T4841 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 909795 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 909796 T4953 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24e0dc65 name:ZooKeeperConnection Watcher:127.0.0.1:55658 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 909796 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 909797 T4841 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 909798 T4841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 909799 T4956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73e64314 name:ZooKeeperConnection Watcher:127.0.0.1:55658/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 909799 T4841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 909802 T4841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 910806 T4841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41785_
   [junit4]   2> 910807 T4841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41785_
   [junit4]   2> 910811 T4841 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 910813 T4841 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 910814 T4841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 910820 T4892 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&collection.configName=conf1&name=nodes_used_collection&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 910820 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 910821 T4877 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"nodes_used_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 910822 T4959 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"nodes_used_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2"}
   [junit4]   2> 910822 T4959 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/nodes_used_collection 
   [junit4]   2> 910822 T4959 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 910824 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 910825 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"nodes_used_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2"} current state version: 1
   [junit4]   2> 910825 T4876 oasco.ClusterStateMutator.createCollection building a new cName: nodes_used_collection
   [junit4]   2> 910825 T4882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 910826 T4902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 910826 T4938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 910826 T4875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 910826 T4920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 910826 T4956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911024 T4959 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection nodes_used_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 911025 T4959 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:54370_
   [junit4]   2> 911025 T4959 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:49108_
   [junit4]   2> 911026 T4959 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:59194_
   [junit4]   2> 911027 T4893 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=nodes_used_collection_shard1_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 911028 T4865 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=nodes_used_collection_shard1_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard1&wt=javabin&version=2
   [junit4]   2> 911027 T4959 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:36368_
   [junit4]   2> 911029 T4928 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=nodes_used_collection_shard2_replica1&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 911029 T4893 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down collection=nodes_used_collection
   [junit4]   2> 911029 T4865 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down collection=nodes_used_collection
   [junit4]   2> 911030 T4910 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=nodes_used_collection_shard2_replica2&action=CREATE&numShards=2&collection=nodes_used_collection&shard=shard2&wt=javabin&version=2
   [junit4]   2> 911030 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 911031 T4893 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 911031 T4865 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 911031 T4928 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down collection=nodes_used_collection
   [junit4]   2> 911031 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54370",
   [junit4]   2> 	  "node_name":"127.0.0.1:54370_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 911031 T4910 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down collection=nodes_used_collection
   [junit4]   2> 911032 T4928 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 911032 T4876 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54370",
   [junit4]   2> 	  "node_name":"127.0.0.1:54370_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 911032 T4910 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 911033 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49108",
   [junit4]   2> 	  "node_name":"127.0.0.1:49108_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 911034 T4876 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49108",
   [junit4]   2> 	  "node_name":"127.0.0.1:49108_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 911035 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 911035 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59194",
   [junit4]   2> 	  "node_name":"127.0.0.1:59194_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 911035 T4876 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59194",
   [junit4]   2> 	  "node_name":"127.0.0.1:59194_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 911036 T4876 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36368",
   [junit4]   2> 	  "node_name":"127.0.0.1:36368_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 911037 T4876 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36368",
   [junit4]   2> 	  "node_name":"127.0.0.1:36368_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"nodes_used_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 911038 T4875 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 911038 T4882 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911038 T4938 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911038 T4902 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911039 T4956 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911038 T4875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 911038 T4920 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 912031 T4893 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica1
   [junit4]   2> 912031 T4865 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica2
   [junit4]   2> 912031 T4893 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 912031 T4865 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 912032 T4893 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 912032 T4928 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica1
   [junit4]   2> 912032 T4865 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 912032 T4865 oascc.ZkStateReader.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 912032 T4893 oascc.ZkStateReader.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 912032 T4910 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica2
   [junit4]   2> 912033 T4910 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 912032 T4928 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 912033 T4865 oascc.ZkStateReader.readConfigName path=/collections/nodes_used_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 912034 T4910 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 912034 T4910 oascc.ZkStateReader.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 912034 T4928 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 912034 T4928 oascc.ZkStateReader.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 912034 T4865 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/'
   [junit4]   2> 912034 T4910 oascc.ZkStateReader.readConfigName path=/collections/nodes_used_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 912034 T4893 oascc.ZkStateReader.readConfigName path=/collections/nodes_used_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 912035 T4910 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/'
   [junit4]   2> 912035 T4928 oascc.ZkStateReader.readConfigName path=/collections/nodes_used_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 912036 T4928 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/'
   [junit4]   2> 912035 T4893 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/'
   [junit4]   2> 912045 T4865 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 912047 T4865 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 912050 T4928 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 912051 T4865 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 912051 T4928 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 912053 T4893 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 912070 T4893 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 912071 T4928 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 912071 T4865 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 912073 T4910 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 912076 T4910 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 912078 T4893 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 912080 T4928 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 912081 T4910 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 912085 T4893 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 912088 T4910 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 912091 T4865 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 912092 T4865 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 912112 T4928 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 912115 T4893 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 912115 T4865 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
   [junit4]   2> 912115 T4928 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 912117 T4910 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 912119 T4910 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 912119 T4893 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 912122 T4928 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
   [junit4]   2> 912127 T4893 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
   [junit4]   2> 912127 T4910 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
   [junit4]   2> 912319 T4893 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 912334 T4893 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 912335 T4893 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 912342 T4893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912344 T4893 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912347 T4893 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912348 T4865 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 912350 T4928 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 912350 T4893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912350 T4893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912351 T4893 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912351 T4893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912352 T4893 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912352 T4893 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using configuration from collection nodes_used_collection
   [junit4]   2> 912352 T4865 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 912352 T4893 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 912353 T4893 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/data/
   [junit4]   2> 912353 T4893 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@340d8780
   [junit4]   2> 912354 T4910 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 912354 T4893 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/data
   [junit4]   2> 912354 T4865 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 912355 T4893 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/data/index/
   [junit4]   2> 912354 T4928 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 912355 T4893 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 912356 T4893 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-003/nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 912356 T4893 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=71.017578125, floorSegmentMB=1.7919921875, forceMergeDeletesPctAllowed=14.872684109700682, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8122364988930778
   [junit4]   2> 912357 T4893 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1ccc689 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11705458),segFN=segments_1,generation=1}
   [junit4]   2> 912357 T4893 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 912357 T4928 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 912358 T4910 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 912360 T4910 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 912360 T4893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 912361 T4893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 912361 T4893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 912362 T4893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912362 T4893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912362 T4893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 912362 T4893 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 912363 T4893 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 912363 T4893 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 912362 T4865 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912365 T4928 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912366 T4865 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912366 T4893 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 912366 T4928 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912367 T4893 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 912367 T4893 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 912368 T4893 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 912368 T4865 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912368 T4893 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 912368 T4928 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912369 T4893 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 912369 T4865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912369 T4865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912369 T4928 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912370 T4928 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912369 T4893 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 912370 T4910 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912370 T4893 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 912370 T4928 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912371 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 912369 T4865 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912371 T4928 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912372 T4928 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912372 T4928 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using configuration from collection nodes_used_collection
   [junit4]   2> 912371 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 912372 T4910 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 912372 T4928 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 912372 T4865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912373 T4865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912372 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 912373 T4865 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using configuration from collection nodes_used_collection
   [junit4]   2> 912373 T4865 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 912374 T4910 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912373 T4928 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/data/
   [junit4]   2> 912374 T4865 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/data/
   [junit4]   2> 912373 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 912375 T4910 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912375 T4910 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912375 T4865 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@340d8780
   [junit4]   2> 912374 T4928 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@340d8780
   [junit4]   2> 912375 T4910 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 912376 T4865 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/data
   [junit4]   2> 912375 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 912376 T4910 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 912377 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 912376 T4865 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/data/index/
   [junit4]   2> 912376 T4928 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/data
   [junit4]   2> 912377 T4865 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 912377 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 912378 T4865 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-002/nodes_used_collection_shard1_replica2/data/index
   [junit4]   2> 912377 T4910 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 912378 T4865 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=71.017578125, floorSegmentMB=1.7919921875, forceMergeDeletesPctAllowed=14.872684109700682, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8122364988930778
   [junit4]   2> 912378 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 912377 T4928 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/data/index/
   [junit4]   2> 912379 T4893 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 912379 T4865 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@29d620c4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6301d9e0),segFN=segments_1,generation=1}
   [junit4]   2> 912380 T4865 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 912378 T4910 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using configuration from collection nodes_used_collection
   [junit4]   2> 912379 T4893 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 912379 T4928 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 912381 T4928 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-005/nodes_used_collection_shard2_replica1/data/index
   [junit4]   2> 912381 T4928 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=71.017578125, floorSegmentMB=1.7919921875, forceMergeDeletesPctAllowed=14.872684109700682, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8122364988930778
   [junit4]   2> 912380 T4910 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 912382 T4910 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/data/
   [junit4]   2> 912382 T4865 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 912382 T4928 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@459ed68a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@29c7abc2),segFN=segments_1,generation=1}
   [junit4]   2> 912382 T4928 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 912382 T4865 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 912382 T4910 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@340d8780
   [junit4]   2> 912382 T4893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 912383 T4865 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 912384 T4910 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/data
   [junit4]   2> 912384 T4865 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912384 T4893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 912384 T4910 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/data/index/
   [junit4]   2> 912385 T4928 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 912384 T4865 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 912385 T4928 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 912385 T4893 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 912385 T4910 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-004/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 912385 T4928 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 912386 T4910 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/

[...truncated too long message...]

ndexWriter with IndexWriterCloser
   [junit4]   2> 971848 T4841 oasc.SolrCore.closeSearcher [awholynewcollection_0_shard1_replica3] Closing main searcher on request.
   [junit4]   2> 971860 T4841 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 971861 T4841 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data;done=false>>]
   [junit4]   2> 971861 T4841 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data
   [junit4]   2> 971861 T4841 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data/index;done=false>>]
   [junit4]   2> 971862 T4841 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001/tempDir-006/awholynewcollection_0_shard1_replica3/data/index
   [junit4]   2> 971863 T4841 oasc.Overseer.close Overseer (id=93111499914412047-127.0.0.1:41785_-n_0000000004) closing
   [junit4]   2> 971863 T5123 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:41785_
   [junit4]   2> 973365 T5100 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 973371 T4841 oejs.AbstractConnector.doStop Stopped ServerConnector@69f42c5{HTTP/1.1}{127.0.0.1:०}
   [junit4]   2> 973371 T4841 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@fd024d4{/,null,UNAVAILABLE}
   [junit4]   2> 973372 T4841 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 973373 T4841 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55658 55658
   [junit4]   2> 974090 T4842 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:55658 55658
   [junit4]   2> 974091 T4842 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/aliases.json
   [junit4]   2> 		8	/solr/clusterstate.json
   [junit4]   2> 		5	/solr/collections/awholynewcollection_0/state.json
   [junit4]   2> 		5	/solr/configs/conf2
   [junit4]   2> 		4	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent data watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		2	/solr/collections/awholynewcollection_0/leader_elect/shard1/election/93111499914412047-core_node2-n_0000000000
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=2E00F5D6B0ABCC8B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hi_IN -Dtests.timezone=Greenwich -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 69.0s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: some core start times did not change on reload
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E00F5D6B0ABCC8B:AFE67BCEC7F4ACB7]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCollectionsAPI(CollectionsAPIDistributedZkTest.java:750)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:199)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 974100 T4841 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest 2E00F5D6B0ABCC8B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {other_tl1=PostingsFormat(name=LuceneFixedGap), _version_=PostingsFormat(name=LuceneFixedGap), multiDefault=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=LuceneFixedGap), a_i1=PostingsFormat(name=Direct), id=Lucene50(blocksize=128), text=Lucene50(blocksize=128), range_facet_l=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct)}, docValues:{timestamp=DocValuesFormat(name=Lucene50)}, sim=DefaultSimilarity, locale=hi_IN, timezone=Greenwich
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=29799344,total=409780224
   [junit4]   2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderTest, TermVectorComponentDistributedTest, DirectSolrSpellCheckerTest, TestSortingResponseWriter, TestLuceneMatchVersion, TestNRTOpen, StatelessScriptUpdateProcessorFactoryTest, TestCollectionAPI, TestManagedSchemaDynamicFieldResource, AddBlockUpdateTest, TestDistributedMissingSort, PolyFieldTest, TestCharFilters, CSVRequestHandlerTest, TimeZoneUtilsTest, UpdateParamsTest, PrimitiveFieldTypeTest, TestBM25SimilarityFactory, TestManagedResourceStorage, AddSchemaFieldsUpdateProcessorFactoryTest, SolrXmlInZkTest, TestValueSourceCache, TestSolr4Spatial, DistributedFacetPivotSmallTest, TestFiltering, RemoteQueryErrorTest, TestSolrIndexConfig, BlockCacheTest, ExternalCollectionsTest, ZkSolrClientTest, FileBasedSpellCheckerTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestExpandComponent, SignatureUpdateProcessorFactoryTest, MBeansHandlerTest, TestJoin, TestInitQParser, TestPostingsSolrHighlighter, HighlighterMaxOffsetTest, TestManagedSynonymFilterFactory, MergeStrategyTest, TestNoOpRegenerator, ZkNodePropsTest, TestJmxMonitoredMap, TestSearchPerf, TestExceedMaxTermLength, SpellCheckCollatorTest, TermVectorComponentTest, TestShardHandlerFactory, CoreMergeIndexesAdminHandlerTest, TestUtils, DistributedTermsComponentTest, TestReload, TestZkChroot, CloudExitableDirectoryReaderTest, DirectUpdateHandlerOptimizeTest, CursorMarkTest, TestSolrXml, SuggesterTSTTest, TestRTGBase, EchoParamsTest, SOLR749Test, SampleTest, TestPivotHelperCode, TestRemoteStreaming, TestDocBasedVersionConstraints, TestConfigOverlay, HighlighterTest, SharedFSAutoReplicaFailoverUtilsTest, TestCloudPivotFacet, TestFileDictionaryLookup, TestSolrXmlPersistence, TestDefaultSearchFieldResource, SimplePostToolTest, SolrTestCaseJ4Test, TestConfigSets, QueryEqualityTest, CoreAdminRequestStatusTest, OpenCloseCoreStressTest, TestImplicitCoreProperties, TermsComponentTest, ExternalFileFieldSortTest, TestSimpleQParserPlugin, StandardRequestHandlerTest, TestLMJelinekMercerSimilarityFactory, TestFastLRUCache, TestRecovery, TestClassNameShortening, WordBreakSolrSpellCheckerTest, TestMaxScoreQueryParser, BadComponentTest, TestSolrXmlPersistor, TestSolrJ, MultiThreadedOCPTest, TestElisionMultitermQuery, TestOverriddenPrefixQueryForCustomFieldType, FastVectorHighlighterTest, MoreLikeThisHandlerTest, ShardSplitTest, TestTrie, HdfsChaosMonkeySafeLeaderTest, TestSolr4Spatial2, BasicZkTest, SynonymTokenizerTest, SolrCmdDistributorTest, TestDynamicFieldResource, FacetPivotSmallTest, XsltUpdateRequestHandlerTest, TestExactStatsCache, TestPerFieldSimilarity, LoggingHandlerTest, BasicDistributedZkTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 69.07s, 1 test, 1 failure <<< FAILURES!

[...truncated 775 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 460 suites, 1868 tests, 1 failure, 47 ignored (22 assumptions)

Total time: 57 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



RE: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11546 - Still Failing!

Posted by Uwe Schindler <uw...@thetaphi.de>.
Seems to still have problems. But this time looks more like wrong config. I nuked the IVY cache, lets see if it helps:

resolve:
[ivy:retrieve] 
[ivy:retrieve] :: problems summary ::
[ivy:retrieve] :::: WARNINGS
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] 		::          UNRESOLVED DEPENDENCIES         ::
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] 		:: org.restlet.jee#org.restlet.ext.servlet;2.3.0: configuration not found in org.restlet.jee#org.restlet.ext.servlet;2.3.0: 'master'. It was required from org.apache.solr#core;working@serv1.sd-datasolutions.de compile
[ivy:retrieve] 		::::::::::::::::::::::::::::::::::::::::::::::
[ivy:retrieve] :::: ERRORS
[ivy:retrieve] 	unknown resolver cloudera
[ivy:retrieve] 	unknown resolver cloudera
[ivy:retrieve] 
[ivy:retrieve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

To me the message 
"[ivy:retrieve] 	unknown resolver cloudera"
Is strange. Wasn't that one removed?

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Policeman Jenkins Server [mailto:jenkins@thetaphi.de]
> Sent: Friday, January 09, 2015 12:51 AM
> To: uwe@thetaphi.de; dev@lucene.apache.org
> Subject: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) -
> Build # 11546 - Still Failing!
> 
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11546/
> Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC
> 
> All tests passed
> 
> Build Log:
> [...truncated 7976 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-
> build.xml:510: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-
> build.xml:463: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-
> build.xml:376: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-
> Linux/solr/core/build.xml:65: impossible to resolve dependencies:
> 	resolve failed - see output for details
> 
> Total time: 32 minutes 38 seconds
> Build step 'Invoke Ant' marked build as failure [description-setter]
> Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -server -
> XX:+UseConcMarkSweepGC Archiving artifacts Recording test results Email
> was triggered for: Failure - Any Sending email for trigger: Failure - Any
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org