You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/01/06 04:17:02 UTC

[JENKINS] Lucene-Solr-4.10-Linux (32bit/jdk1.7.0_67) - Build # 213 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.10-Linux/213/
Java: 32bit/jdk1.7.0_67 -server -XX:+UseConcMarkSweepGC (asserts: true)

1 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrServerTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([466C2494292A3631:C78AAA8C5E75560D]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.client.solrj.impl.CloudSolrServerTest.doTest(CloudSolrServerTest.java:204)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:871)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12903 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.impl.CloudSolrServerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/init-core-data-001
   [junit4]   2> 36927 T114 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 36928 T114 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /volcx/io
   [junit4]   2> 36931 T114 oasc.AbstractZkTestCase.<clinit> WARN TEST_HOME() does not exist - solrj test?
   [junit4]   2> 36938 T114 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 36940 T114 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 36943 T115 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 37042 T114 oasc.ZkTestServer.run start zk server on port:38765
   [junit4]   2> 37082 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37120 T121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ea9cd8 name:ZooKeeperConnection Watcher:127.0.0.1:38765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37120 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37122 T114 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 37146 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37148 T123 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c97a40 name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37148 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37150 T114 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 37156 T114 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 37162 T114 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 37166 T114 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 37173 T114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 37173 T114 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 37182 T114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 37183 T114 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 37189 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 37190 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 37191 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 37191 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 37192 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 37193 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 37193 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 37193 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 37194 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 37261 T114 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 37265 T114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58197
   [junit4]   2> 37265 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 37266 T114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 37266 T114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002
   [junit4]   2> 37266 T114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002/'
   [junit4]   2> 37276 T114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002/solr.xml
   [junit4]   2> 37309 T114 oasc.CoreContainer.<init> New CoreContainer 18503492
   [junit4]   2> 37309 T114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002/]
   [junit4]   2> 37310 T114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 37310 T114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 37311 T114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 37311 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 37311 T114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 37312 T114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 37312 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 37312 T114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 37312 T114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 37313 T114 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 37313 T114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 37314 T114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 37314 T114 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 37314 T114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38765/solr
   [junit4]   2> 37324 T114 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 37325 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37327 T134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11626cf name:ZooKeeperConnection Watcher:127.0.0.1:38765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37327 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37333 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 37335 T136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cdd240 name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 37335 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 37344 T114 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 37354 T114 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 37363 T114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 37371 T114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 37379 T114 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 37394 T114 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 37397 T114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58197_volcx%2Fio
   [junit4]   2> 37400 T114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58197_volcx%2Fio
   [junit4]   2> 37407 T114 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 37412 T114 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 37416 T114 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 37423 T114 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58197_volcx%2Fio
   [junit4]   2> 37423 T114 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 37427 T114 oasc.Overseer.start Overseer (id=93094811379171331-127.0.0.1:58197_volcx%2Fio-n_0000000000) starting
   [junit4]   2> 37437 T114 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 37465 T114 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 37470 T138 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 37471 T114 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 37476 T114 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 37480 T114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 37496 T137 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 37504 T140 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 37505 T140 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 37512 T140 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 37514 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37521 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "node_name":"127.0.0.1:58197_volcx%2Fio",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58197/volcx/io",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 37521 T137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 37534 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 37545 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 38513 T140 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 38514 T140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 38515 T140 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 38516 T140 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 38518 T140 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 38519 T140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002/collection1/'
   [junit4]   2> 38565 T140 oasc.SolrConfig.<init> Using Lucene MatchVersion: 4.10.3
   [junit4]   2> 38590 T140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 38593 T140 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 38609 T140 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 38862 T140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 38864 T140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 38865 T140 oass.IndexSchema.loadCopyFields WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 38866 T140 oass.IndexSchema.loadCopyFields WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 38867 T140 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 38867 T140 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 38867 T140 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-002/collection1/, dataDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/
   [junit4]   2> 38867 T140 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 38868 T140 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data
   [junit4]   2> 38868 T140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/index/
   [junit4]   2> 38868 T140 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 38869 T140 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/index
   [junit4]   2> 38871 T140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@11e07f7 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 38871 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 38872 T140 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 38872 T140 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38873 T140 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38873 T140 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38873 T140 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38874 T140 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 38874 T140 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 38874 T140 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 38874 T140 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 38874 T140 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38875 T140 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 38875 T140 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 38875 T140 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 38877 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38878 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38879 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38880 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38882 T140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38884 T140 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 38884 T140 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 38885 T140 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@11e07f7 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/control/data/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 38885 T140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 38885 T140 oass.SolrIndexSearcher.<init> Opening Searcher@1d4269b[collection1] main
   [junit4]   2> 38886 T140 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 38888 T140 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 38888 T140 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 38891 T140 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 38891 T140 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 38891 T140 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 38892 T140 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 38893 T140 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 38894 T140 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 38900 T140 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 38900 T140 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 38901 T140 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 38903 T140 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 38904 T142 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d4269b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 38905 T145 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58197/volcx/io collection:control_collection shard:shard1
   [junit4]   2> 38906 T114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 38907 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 38923 T145 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 38929 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 38931 T147 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14552fd name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 38931 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 38933 T114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 38938 T114 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 38946 T145 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 38949 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 38950 T145 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 38950 T145 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@a98388 url=http://127.0.0.1:58197/volcx/io/collection1 node=127.0.0.1:58197_volcx%2Fio C29_STATE=coll:control_collection core:collection1 props:{core=collection1, node_name=127.0.0.1:58197_volcx%2Fio, state=down, base_url=http://127.0.0.1:58197/volcx/io}
   [junit4]   2> 38951 T145 C29 P58197 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58197/volcx/io/collection1/
   [junit4]   2> 38952 T145 C29 P58197 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 38952 T145 C29 P58197 oasc.SyncStrategy.syncToMe http://127.0.0.1:58197/volcx/io/collection1/ has no replicas
   [junit4]   2> 38952 T145 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58197/volcx/io/collection1/ shard1
   [junit4]   2> 38953 T145 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 38959 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39002 T114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 39003 T114 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 39005 T114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56564
   [junit4]   2> 39006 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 39007 T114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 39007 T114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003
   [junit4]   2> 39007 T114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003/'
   [junit4]   2> 39022 T114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003/solr.xml
   [junit4]   2> 39073 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39074 T114 oasc.CoreContainer.<init> New CoreContainer 20704697
   [junit4]   2> 39075 T114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003/]
   [junit4]   2> 39075 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39084 T114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 39085 T114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 39085 T114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 39085 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 39086 T114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 39086 T114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 39087 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 39087 T114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 39088 T114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 39088 T114 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 39089 T114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 39089 T114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 39090 T114 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 39090 T114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38765/solr
   [junit4]   2> 39091 T114 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 39092 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 39094 T159 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1430e4a name:ZooKeeperConnection Watcher:127.0.0.1:38765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 39094 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 39097 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 39098 T161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a021a name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 39099 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 39108 T114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 39110 T145 oasc.ZkController.register We are http://127.0.0.1:58197/volcx/io/collection1/ and leader is http://127.0.0.1:58197/volcx/io/collection1/
   [junit4]   2> 39111 T145 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58197/volcx/io
   [junit4]   2> 39111 T145 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 39111 T145 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 39111 T145 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 39113 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39116 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:58197_volcx%2Fio",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58197/volcx/io",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 39222 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39223 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 39223 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 40113 T114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56564_volcx%2Fio
   [junit4]   2> 40114 T114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56564_volcx%2Fio
   [junit4]   2> 40122 T114 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 40128 T163 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 40128 T163 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 40131 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 40131 T163 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 40134 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:56564_volcx%2Fio",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56564/volcx/io",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 40134 T137 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 40134 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 40241 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40242 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40244 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41132 T163 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 41132 T163 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 41133 T163 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 41133 T163 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 41135 T163 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 41135 T163 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003/collection1/'
   [junit4]   2> 41157 T163 oasc.SolrConfig.<init> Using Lucene MatchVersion: 4.10.3
   [junit4]   2> 41183 T163 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 41185 T163 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 41194 T163 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 41433 T163 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 41435 T163 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 41437 T163 oass.IndexSchema.loadCopyFields WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 41437 T163 oass.IndexSchema.loadCopyFields WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 41438 T163 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 41438 T163 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 41438 T163 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-003/collection1/, dataDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/
   [junit4]   2> 41438 T163 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 41439 T163 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1
   [junit4]   2> 41439 T163 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/index/
   [junit4]   2> 41439 T163 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 41439 T163 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/index
   [junit4]   2> 41441 T163 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@da78f9 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 41441 T163 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 41442 T163 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 41442 T163 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 41442 T163 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 41443 T163 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 41444 T163 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 41444 T163 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 41444 T163 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 41444 T163 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 41446 T163 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 41447 T163 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 41448 T163 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 41449 T163 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 41450 T163 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 41452 T163 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 41452 T163 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 41453 T163 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@da78f9 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty1/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 41453 T163 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 41453 T163 oass.SolrIndexSearcher.<init> Opening Searcher@81739e[collection1] main
   [junit4]   2> 41453 T163 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 41455 T163 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 41455 T163 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 41456 T163 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 41456 T163 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 41456 T163 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 41458 T163 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 41458 T163 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 41458 T163 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 41458 T163 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 41460 T164 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@81739e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 41460 T163 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 41461 T167 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56564/volcx/io collection:collection1 shard:shard1
   [junit4]   2> 41461 T114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 41462 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 41462 T167 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 41471 T167 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 41478 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 41479 T167 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 41479 T167 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@1c7c93e url=http://127.0.0.1:56564/volcx/io/collection1 node=127.0.0.1:56564_volcx%2Fio C30_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:56564_volcx%2Fio, state=down, base_url=http://127.0.0.1:56564/volcx/io}
   [junit4]   2> 41479 T167 C30 P56564 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56564/volcx/io/collection1/
   [junit4]   2> 41480 T167 C30 P56564 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 41480 T167 C30 P56564 oasc.SyncStrategy.syncToMe http://127.0.0.1:56564/volcx/io/collection1/ has no replicas
   [junit4]   2> 41480 T167 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56564/volcx/io/collection1/ shard1
   [junit4]   2> 41481 T167 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 41490 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 41534 T114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 41535 T114 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 41557 T114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58644
   [junit4]   2> 41558 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 41558 T114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 41559 T114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004
   [junit4]   2> 41559 T114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004/'
   [junit4]   2> 41586 T114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004/solr.xml
   [junit4]   2> 41597 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41598 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41597 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41644 T114 oasc.CoreContainer.<init> New CoreContainer 20486997
   [junit4]   2> 41644 T167 oasc.ZkController.register We are http://127.0.0.1:56564/volcx/io/collection1/ and leader is http://127.0.0.1:56564/volcx/io/collection1/
   [junit4]   2> 41644 T114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004/]
   [junit4]   2> 41645 T167 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56564/volcx/io
   [junit4]   2> 41645 T167 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 41645 T167 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 41646 T167 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 41646 T114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 41646 T114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 41647 T114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 41647 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 41648 T114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 41648 T114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 41648 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 41649 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 41649 T114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 41650 T114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 41651 T114 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 41651 T114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 41652 T114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 41652 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:56564_volcx%2Fio",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56564/volcx/io",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 41652 T114 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 41653 T114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38765/solr
   [junit4]   2> 41653 T114 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 41655 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41656 T178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15dfc8a name:ZooKeeperConnection Watcher:127.0.0.1:38765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41657 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41660 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41661 T116 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14abd3cdab00007, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 41664 T180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@826436 name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41664 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41672 T114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 41765 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41765 T181 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41765 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 41769 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 42676 T114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58644_volcx%2Fio
   [junit4]   2> 42678 T114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58644_volcx%2Fio
   [junit4]   2> 42682 T114 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 42687 T182 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 42687 T182 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42689 T182 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 42689 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42691 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:58644_volcx%2Fio",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58644/volcx/io",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 42692 T137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 42692 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 42796 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42796 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42796 T181 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42796 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 43689 T182 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 43689 T182 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 43690 T182 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 43690 T182 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 43691 T182 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 43692 T182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004/collection1/'
   [junit4]   2> 43718 T182 oasc.SolrConfig.<init> Using Lucene MatchVersion: 4.10.3
   [junit4]   2> 43743 T182 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 43745 T182 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 43754 T182 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 43989 T182 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 43991 T182 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 43992 T182 oass.IndexSchema.loadCopyFields WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 43992 T182 oass.IndexSchema.loadCopyFields WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 43993 T182 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 43993 T182 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 43993 T182 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-004/collection1/, dataDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/
   [junit4]   2> 43994 T182 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 43994 T182 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2
   [junit4]   2> 43994 T182 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/index/
   [junit4]   2> 43994 T182 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 43995 T182 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/index
   [junit4]   2> 43996 T182 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@1afb661 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 43996 T182 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 43997 T182 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 43997 T182 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 43997 T182 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 43998 T182 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 43998 T182 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 43998 T182 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 43998 T182 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 43998 T182 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 43999 T182 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 43999 T182 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 43999 T182 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 43999 T182 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 43999 T182 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 44001 T182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44002 T182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44003 T182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44004 T182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44005 T182 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44006 T182 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 44007 T182 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 44007 T182 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@1afb661 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty2/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 44008 T182 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44008 T182 oass.SolrIndexSearcher.<init> Opening Searcher@60c733[collection1] main
   [junit4]   2> 44008 T182 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 44010 T182 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 44010 T182 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 44011 T182 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 44011 T182 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 44011 T182 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44021 T182 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 44022 T182 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44022 T182 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 44022 T182 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 44023 T183 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@60c733[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 44023 T182 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 44024 T186 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58644/volcx/io collection:collection1 shard:shard2
   [junit4]   2> 44024 T114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 44025 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 44025 T186 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 44030 T186 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 44032 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44033 T186 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 44033 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@1d01b31 url=http://127.0.0.1:58644/volcx/io/collection1 node=127.0.0.1:58644_volcx%2Fio C31_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:58644_volcx%2Fio, state=down, base_url=http://127.0.0.1:58644/volcx/io}
   [junit4]   2> 44034 T186 C31 P58644 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58644/volcx/io/collection1/
   [junit4]   2> 44034 T186 C31 P58644 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 44034 T186 C31 P58644 oasc.SyncStrategy.syncToMe http://127.0.0.1:58644/volcx/io/collection1/ has no replicas
   [junit4]   2> 44035 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58644/volcx/io/collection1/ shard2
   [junit4]   2> 44035 T186 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 44044 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44073 T114 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 44074 T114 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 44076 T114 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47101
   [junit4]   2> 44077 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 44077 T114 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 44077 T114 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005
   [junit4]   2> 44078 T114 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005/'
   [junit4]   2> 44087 T114 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005/solr.xml
   [junit4]   2> 44117 T114 oasc.CoreContainer.<init> New CoreContainer 18533137
   [junit4]   2> 44118 T114 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005/]
   [junit4]   2> 44118 T114 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 44119 T114 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 44119 T114 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 44119 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 44120 T114 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 44120 T114 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 44120 T114 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 44120 T114 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 44121 T114 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 44121 T114 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 44122 T114 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 44122 T114 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 44122 T114 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 44123 T114 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38765/solr
   [junit4]   2> 44123 T114 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 44124 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 44125 T197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9f5c4 name:ZooKeeperConnection Watcher:127.0.0.1:38765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 44125 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 44128 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 44130 T199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f7c45d name:ZooKeeperConnection Watcher:127.0.0.1:38765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 44130 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 44137 T114 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 44150 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44150 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44150 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44151 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44151 T181 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44196 T186 oasc.ZkController.register We are http://127.0.0.1:58644/volcx/io/collection1/ and leader is http://127.0.0.1:58644/volcx/io/collection1/
   [junit4]   2> 44196 T186 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58644/volcx/io
   [junit4]   2> 44197 T186 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 44197 T186 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 44197 T186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 44199 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44201 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:58644_volcx%2Fio",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:58644/volcx/io",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 44334 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44334 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44334 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44335 T181 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44335 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 45140 T114 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47101_volcx%2Fio
   [junit4]   2> 45142 T114 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47101_volcx%2Fio
   [junit4]   2> 45148 T114 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 45157 T201 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 45157 T201 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 45158 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 45158 T201 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 45162 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:47101_volcx%2Fio",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47101/volcx/io",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 45163 T137 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 45163 T137 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 45271 T181 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 45272 T157 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 45272 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 45272 T162 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 45272 T141 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 46159 T201 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 46160 T201 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 46161 T201 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 46162 T201 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 46164 T201 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 46165 T201 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005/collection1/'
   [junit4]   2> 46203 T201 oasc.SolrConfig.<init> Using Lucene MatchVersion: 4.10.3
   [junit4]   2> 46228 T201 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 46229 T201 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 46238 T201 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 46459 T201 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 46465 T201 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 46466 T201 oass.IndexSchema.loadCopyFields WARN Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 46466 T201 oass.IndexSchema.loadCopyFields WARN Field text is not multivalued and destination for multiple copyFields (3)
   [junit4]   2> 46467 T201 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 46467 T201 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 46467 T201 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-005/collection1/, dataDir=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/
   [junit4]   2> 46467 T201 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 46468 T201 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3
   [junit4]   2> 46468 T201 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/index/
   [junit4]   2> 46468 T201 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 46469 T201 oasc.CachingDirectoryFactory.get return new directory for ./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/index
   [junit4]   2> 46471 T201 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@395e27 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 46471 T201 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 46472 T201 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 46472 T201 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 46472 T201 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 46472 T201 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 46473 T201 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 46473 T201 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 46473 T201 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 46473 T201 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 46473 T201 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 46474 T201 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 46474 T201 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 46474 T201 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 46474 T201 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: solr.PingRequestHandler
   [junit4]   2> 46476 T201 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46477 T201 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46478 T201 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46479 T201 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46479 T201 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 46481 T201 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 46481 T201 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 46482 T201 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@395e27 lockFactory=NativeFSLockFactory@./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001/tempDir-001/jetty3/index; maxCacheMB=0.53515625 maxMergeSizeMB=0.1943359375)),segFN=segments_1,generation=1}
   [junit4]   2> 46482 T201 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 46483 T201 oass.SolrIndexSearcher.<init> Opening Searcher@1819223[collection1] main
   [junit4]   2> 46484 T201 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 46485 T201 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 46485 T201 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 46486 T201 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 46486 T201 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 46486 T201 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 46487 T201 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 46487 T201 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 46488 T201 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 46488 T201 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 46489 T202 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1819223[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 46489 T201 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 46490 T205 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47101/volcx/io collection:collection1 shard:shard1
   [junit4]   2> 46491 T114 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0
   [junit4]   2> 46491 T114 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 46493 T205 oasc.ZkController.register We are http://127.0.0.1:47101/volcx/io/collection1/ and leader is http://127.0.0.1:56564/volcx/io/collection1/
   [junit4]   2> 46493 T205 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47101/volcx/io
   [junit4]   2> 46494 T205 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 46494 T205 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 46494 T114 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
   [junit4]   2> 46495 T114 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
   [junit4]   2> ASYNC  NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@758e43 url=http://127.0.0.1:47101/volcx/io/collection1 node=127.0.0.1:47101_volcx%2Fio C32_STATE=coll:collection1 core:collection1 props:{core=collection1, node_name=127.0.0.1:47101_volcx%2Fio, state=down, base_url=http://127.0.0.1:47101/volcx/io}
   [junit4]   2> 46498 T206 C32 P47101 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 46499 T206 C32 P47101 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 46499 T206 C32 P47101 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:56564/volcx/io/collection1/ and I am http://127.0.0.1:47101/volcx/io/collection1/
   [junit4]   2> 46499 T206 C32 P47101 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 46500 T206 C32 P47101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 46501 T141 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 46504 T206 C32 P47101 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:56564/volcx/io; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A47101_volcx%252Fio&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
   [junit4]   2> 46504 T137 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:47101_volcx%2Fio",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:47101/volcx/io",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "roles":null}
   [junit4]   2> 46506 T153 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
   [junit4]   2> 46507 T153 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
   [junit4]   2> 46508 T15

[...truncated too long message...]

 because it doesn't exist
   [junit4]   2> 78116 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 78116 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 78117 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 78117 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 78119 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 78120 T251 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 78120 T251 oaz.ClientCnxn$SendThread.run WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.SocketException: Network is unreachable
   [junit4]   2> 	at sun.nio.ch.Net.connect0(Native Method)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:465)
   [junit4]   2> 	at sun.nio.ch.Net.connect(Net.java:457)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:967)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 
   [junit4]   2> 79222 T251 oaz.ClientCnxnSocketNIO.connect ERROR Unable to open socket to ff01::114/ff01:0:0:0:0:0:0:114:33332
   [junit4]   2> 79324 T114 oas.SolrTestCaseJ4.tearDown ###Ending testShutdown
   [junit4]   2> 79325 T114 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59877 59877
   [junit4]   2> 80001 T242 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59877 59877
   [junit4]   2> 80012 T114 oas.SolrTestCaseJ4.setUp ###Starting testWrongZkChrootTest
   [junit4]   2> 80014 T114 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 80015 T253 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 80114 T114 oasc.ZkTestServer.run start zk server on port:58445
   [junit4]   2> 80116 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 80118 T259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ce0ae2 name:ZooKeeperConnection Watcher:127.0.0.1:58445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 80118 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 80118 T114 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 80122 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 80124 T261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10261c8 name:ZooKeeperConnection Watcher:127.0.0.1:58445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 80124 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 80124 T114 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 80127 T114 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 80129 T114 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 80131 T114 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 80138 T114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 80139 T114 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 80144 T114 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 80145 T114 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 80149 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml because it doesn't exist
   [junit4]   2> 80149 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/stopwords.txt because it doesn't exist
   [junit4]   2> 80150 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/protwords.txt because it doesn't exist
   [junit4]   2> 80150 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/currency.xml because it doesn't exist
   [junit4]   2> 80151 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/enumsConfig.xml because it doesn't exist
   [junit4]   2> 80151 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/open-exchange-rates.json because it doesn't exist
   [junit4]   2> 80152 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/mapping-ISOLatin1Accent.txt because it doesn't exist
   [junit4]   2> 80152 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/old_synonyms.txt because it doesn't exist
   [junit4]   2> 80153 T114 oasc.AbstractZkTestCase.putConfig skipping /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/solrj/src/test-files/solrj/solr/collection1/conf/synonyms.txt because it doesn't exist
   [junit4]   2> 80156 T114 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 80157 T263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b17258 name:ZooKeeperConnection Watcher:127.0.0.1:58445/solr/xyz/foo got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 80157 T114 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 80159 T114 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 80162 T114 oas.SolrTestCaseJ4.tearDown ###Ending testWrongZkChrootTest
   [junit4]   2> 80163 T114 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58445 58445
   [junit4]   2> 80702 T253 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58445 58445
   [junit4]   2> 80706 T114 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build/solr-solrj/test/J0/./solr.client.solrj.impl.CloudSolrServerTest-466C2494292A3631-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene410), sim=DefaultSimilarity, locale=tr_TR, timezone=Africa/Dakar
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.7.0_67 (32-bit)/cpus=8,threads=1,free=26382472,total=64880640
   [junit4]   2> NOTE: All tests run in this JVM: [SolrPingTest, ContentStreamTest, DocumentAnalysisResponseTest, JettyWebappTest, TermsResponseTest, TestDocumentObjectBinder, TestJavaBinCodec, TestJsonRecordReader, TestRetryUtil, ExternalHttpClientTest, ModifiableSolrParamsTest, SolrExampleXMLTest, TestCoreAdmin, CloudSolrServerTest]
   [junit4] Completed on J0 in 43.83s, 3 tests, 1 failure <<< FAILURES!

[...truncated 82 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:474: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/build.xml:202: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/solr/common-build.xml:494: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.10-Linux/lucene/common-build.xml:966: There were test failures: 52 suites, 309 tests, 1 failure

Total time: 110 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_67 -server -XX:+UseConcMarkSweepGC (asserts: true)
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any