You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/10/11 17:17:04 UTC
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 654 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/654/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 943 from http://127.0.0.1:65501/ot/s/collection1lastClient and got 945 from http://127.0.0.1:11446/ot/s/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 943 from http://127.0.0.1:65501/ot/s/collection1lastClient and got 945 from http://127.0.0.1:11446/ot/s/collection1
at __randomizedtesting.SeedInfo.seed([184858502002211B:99AED648575D4127]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1255)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:232)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12518 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/init-core-data-001
[junit4] 2> 2407538 T6032 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ot/s
[junit4] 2> 2407544 T6032 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 2407545 T6032 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2407546 T6033 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 2407647 T6032 oasc.ZkTestServer.run start zk server on port:28495
[junit4] 2> 2407648 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2407649 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2407655 T6039 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dcbf0c5 name:ZooKeeperConnection Watcher:127.0.0.1:28495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2407655 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2407656 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2407656 T6032 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2407659 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2407659 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2407661 T6041 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f5d39a3 name:ZooKeeperConnection Watcher:127.0.0.1:28495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2407661 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2407661 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2407662 T6032 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 2407664 T6032 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 2407665 T6032 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 2407667 T6032 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 2407669 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2407669 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 2407672 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2407673 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 2407777 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2407778 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2407780 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2407781 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 2407783 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2407783 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 2407785 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2407786 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 2407788 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2407789 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 2407790 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2407791 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 2407793 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2407794 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2407795 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2407796 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 2407798 T6032 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2407799 T6032 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 2409197 T6032 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2409202 T6032 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28498
[junit4] 2> 2409202 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2409203 T6032 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2409203 T6032 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002
[junit4] 2> 2409203 T6032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/'
[junit4] 2> 2409238 T6032 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/solr.xml
[junit4] 2> 2409311 T6032 oasc.CoreContainer.<init> New CoreContainer 1448556625
[junit4] 2> 2409312 T6032 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/]
[junit4] 2> 2409312 T6032 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2409313 T6032 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2409313 T6032 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2409313 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2409313 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 2409314 T6032 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2409314 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2409314 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2409314 T6032 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2409314 T6032 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2409315 T6032 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2409315 T6032 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2409315 T6032 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2409316 T6032 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2409316 T6032 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28495/solr
[junit4] 2> 2409316 T6032 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2409316 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2409317 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2409319 T6052 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78327a1f name:ZooKeeperConnection Watcher:127.0.0.1:28495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2409319 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2409320 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2409321 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2409322 T6054 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60b4ed5f name:ZooKeeperConnection Watcher:127.0.0.1:28495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2409322 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2409323 T6032 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 2409325 T6032 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 2409327 T6032 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 2409329 T6032 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 2409331 T6032 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 2409332 T6032 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 2409333 T6032 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28498_ot%2Fs
[junit4] 2> 2409334 T6032 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28498_ot%2Fs
[junit4] 2> 2409335 T6032 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 2409337 T6032 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 2409338 T6032 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2409339 T6032 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:28498_ot%2Fs
[junit4] 2> 2409339 T6032 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 2409341 T6032 oasc.Overseer.start Overseer (id=92604829481631747-127.0.0.1:28498_ot%2Fs-n_0000000000) starting
[junit4] 2> 2409342 T6032 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 2409347 T6032 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 2409348 T6056 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 2409348 T6032 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 2409350 T6032 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 2409351 T6032 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2409353 T6055 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 2409357 T6058 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 2409357 T6058 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2409358 T6058 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2409359 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2409363 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:28498_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:28498/ot/s",
[junit4] 2> "state":"down"}
[junit4] 2> 2409364 T6055 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 2409364 T6055 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 2409364 T6055 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2409366 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2410360 T6058 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2410361 T6058 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 2410362 T6058 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2410362 T6058 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2410362 T6058 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2410363 T6058 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/collection1/'
[junit4] 2> 2410364 T6058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 2410365 T6058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 2410366 T6058 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 2410434 T6058 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 2410483 T6058 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2410585 T6058 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2410608 T6058 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2410992 T6058 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2410995 T6058 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2410998 T6058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2411003 T6058 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2411020 T6058 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 2411021 T6058 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2411021 T6058 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/
[junit4] 2> 2411021 T6058 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10cae060
[junit4] 2> 2411024 T6058 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data
[junit4] 2> 2411025 T6058 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/index/
[junit4] 2> 2411026 T6058 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 2411027 T6058 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/index
[junit4] 2> 2411028 T6058 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=3, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 2411051 T6058 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2411052 T6058 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2411056 T6058 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2411056 T6058 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2411057 T6058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2411057 T6058 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2411057 T6058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2411057 T6058 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2411058 T6058 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2411058 T6058 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2411058 T6058 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2411059 T6058 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2411060 T6058 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2411060 T6058 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2411060 T6058 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2411061 T6058 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2411061 T6058 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2411061 T6058 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2411062 T6058 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2411062 T6058 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2411063 T6058 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2411063 T6058 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2411083 T6058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2411086 T6058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2411087 T6058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2411089 T6058 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2411093 T6058 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2411093 T6058 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2411094 T6058 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=266024004, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2411096 T6058 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 2411096 T6058 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2411097 T6058 oass.SolrIndexSearcher.<init> Opening Searcher@27018c17[collection1] main
[junit4] 2> 2411097 T6058 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 2411102 T6058 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2411102 T6058 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2411103 T6058 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2411103 T6058 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2411104 T6058 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2411104 T6058 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 2411104 T6058 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2411105 T6058 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 2411107 T6058 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 2411107 T6058 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2411107 T6058 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2411111 T6060 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@27018c17[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2411112 T6058 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2411113 T6063 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28498/ot/s collection:control_collection shard:shard1
[junit4] 2> 2411114 T6032 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 2411114 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2411114 T6063 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 2411115 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2411116 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2411117 T6065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dbf8fc1 name:ZooKeeperConnection Watcher:127.0.0.1:28495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2411117 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2411118 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2411119 T6032 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2411120 T6063 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2411121 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2411122 T6063 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2411122 T6063 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 2411122 T6032 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> ASYNC NEW_CORE C1421 name=collection1 org.apache.solr.core.SolrCore@6bc52213 url=http://127.0.0.1:28498/ot/s/collection1 node=127.0.0.1:28498_ot%2Fs C1421_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:28498_ot%2Fs, core=collection1, base_url=http://127.0.0.1:28498/ot/s, state=down}
[junit4] 2> 2411122 T6063 C1421 P28498 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28498/ot/s/collection1/
[junit4] 2> 2411123 T6063 C1421 P28498 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2411123 T6063 C1421 P28498 oasc.SyncStrategy.syncToMe http://127.0.0.1:28498/ot/s/collection1/ has no replicas
[junit4] 2> 2411123 T6063 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28498/ot/s/collection1/ shard1
[junit4] 2> 2411124 T6063 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 2411126 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2411126 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2411132 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2411133 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2411133 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2411180 T6063 oasc.ZkController.register We are http://127.0.0.1:28498/ot/s/collection1/ and leader is http://127.0.0.1:28498/ot/s/collection1/
[junit4] 2> 2411181 T6063 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28498/ot/s
[junit4] 2> 2411181 T6063 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2411181 T6063 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 2411181 T6063 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2411183 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2411186 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "node_name":"127.0.0.1:28498_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:28498/ot/s",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "state":"active"}
[junit4] 2> 2411290 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2411290 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 2412402 T6032 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 2412403 T6032 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2412406 T6032 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28502
[junit4] 2> 2412406 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2412406 T6032 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2412407 T6032 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003
[junit4] 2> 2412407 T6032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/'
[junit4] 2> 2412437 T6032 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/solr.xml
[junit4] 2> 2412492 T6032 oasc.CoreContainer.<init> New CoreContainer 652636090
[junit4] 2> 2412492 T6032 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/]
[junit4] 2> 2412493 T6032 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2412493 T6032 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2412493 T6032 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2412494 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2412494 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 2412494 T6032 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2412494 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2412494 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2412495 T6032 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2412495 T6032 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2412495 T6032 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2412496 T6032 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2412496 T6032 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2412496 T6032 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2412496 T6032 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28495/solr
[junit4] 2> 2412497 T6032 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2412497 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2412498 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2412499 T6077 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@62b13049 name:ZooKeeperConnection Watcher:127.0.0.1:28495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2412500 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2412500 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2412501 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2412503 T6079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a92975e name:ZooKeeperConnection Watcher:127.0.0.1:28495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2412503 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2412508 T6032 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2413512 T6032 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28502_ot%2Fs
[junit4] 2> 2413513 T6032 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28502_ot%2Fs
[junit4] 2> 2413517 T6032 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2413523 T6081 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2413524 T6081 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2413524 T6081 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2413525 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2413527 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:28502_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:28502/ot/s",
[junit4] 2> "state":"down"}
[junit4] 2> 2413527 T6055 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 2413527 T6055 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 2413527 T6055 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 2413530 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2413530 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2413530 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2414525 T6081 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2414526 T6081 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 2414527 T6081 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2414527 T6081 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2414528 T6081 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2414528 T6081 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/collection1/'
[junit4] 2> 2414529 T6081 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 2414530 T6081 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 2414531 T6081 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 2414599 T6081 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 2414676 T6081 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2414778 T6081 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2414796 T6081 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2415051 T6081 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2415053 T6081 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2415055 T6081 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2415058 T6081 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2415074 T6081 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 2415074 T6081 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2415074 T6081 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/
[junit4] 2> 2415075 T6081 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10cae060
[junit4] 2> 2415076 T6081 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1
[junit4] 2> 2415077 T6081 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/index/
[junit4] 2> 2415077 T6081 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 2415079 T6081 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/index
[junit4] 2> 2415079 T6081 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=3, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 2415083 T6081 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2415083 T6081 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2415087 T6081 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2415087 T6081 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2415087 T6081 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2415087 T6081 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2415088 T6081 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2415088 T6081 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2415088 T6081 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2415088 T6081 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2415089 T6081 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2415089 T6081 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2415089 T6081 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2415090 T6081 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2415090 T6081 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2415091 T6081 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2415091 T6081 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2415091 T6081 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2415091 T6081 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2415092 T6081 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2415092 T6081 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2415092 T6081 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2415105 T6081 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2415107 T6081 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2415108 T6081 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2415109 T6081 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2415112 T6081 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2415112 T6081 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2415113 T6081 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=266024004, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2415114 T6081 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 2415115 T6081 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2415115 T6081 oass.SolrIndexSearcher.<init> Opening Searcher@4e00f701[collection1] main
[junit4] 2> 2415115 T6081 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2415116 T6081 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2415116 T6081 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2415117 T6081 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2415117 T6081 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2415117 T6081 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2415118 T6081 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 2415118 T6081 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2415118 T6081 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 2415118 T6081 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2415124 T6081 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2415124 T6082 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e00f701[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2415125 T6085 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28502/ot/s collection:collection1 shard:shard2
[junit4] 2> 2415125 T6032 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 2415125 T6085 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 2415125 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2415129 T6085 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 2415130 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2415130 T6085 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2415130 T6085 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1422 name=collection1 org.apache.solr.core.SolrCore@6d968ad6 url=http://127.0.0.1:28502/ot/s/collection1 node=127.0.0.1:28502_ot%2Fs C1422_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:28502_ot%2Fs, core=collection1, base_url=http://127.0.0.1:28502/ot/s, state=down}
[junit4] 2> 2415131 T6085 C1422 P28502 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28502/ot/s/collection1/
[junit4] 2> 2415131 T6085 C1422 P28502 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2415131 T6085 C1422 P28502 oasc.SyncStrategy.syncToMe http://127.0.0.1:28502/ot/s/collection1/ has no replicas
[junit4] 2> 2415131 T6085 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28502/ot/s/collection1/ shard2
[junit4] 2> 2415131 T6085 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 2415134 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415134 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415134 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415139 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2415139 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415139 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415139 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415185 T6085 oasc.ZkController.register We are http://127.0.0.1:28502/ot/s/collection1/ and leader is http://127.0.0.1:28502/ot/s/collection1/
[junit4] 2> 2415186 T6085 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28502/ot/s
[junit4] 2> 2415186 T6085 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2415186 T6085 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 2415186 T6085 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2415188 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2415191 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "node_name":"127.0.0.1:28502_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:28502/ot/s",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "state":"active"}
[junit4] 2> 2415294 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415294 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2415294 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 2416587 T6032 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 2416588 T6032 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2416591 T6032 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:57476
[junit4] 2> 2416592 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2416593 T6032 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2416593 T6032 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004
[junit4] 2> 2416593 T6032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/'
[junit4] 2> 2416643 T6032 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/solr.xml
[junit4] 2> 2416732 T6032 oasc.CoreContainer.<init> New CoreContainer 401168257
[junit4] 2> 2416732 T6032 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/]
[junit4] 2> 2416733 T6032 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2416734 T6032 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2416734 T6032 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2416734 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2416735 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 2416735 T6032 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2416735 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2416736 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2416736 T6032 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2416736 T6032 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2416737 T6032 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2416738 T6032 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2416738 T6032 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2416738 T6032 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2416739 T6032 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28495/solr
[junit4] 2> 2416739 T6032 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2416739 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2416740 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2416742 T6096 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@586fe26a name:ZooKeeperConnection Watcher:127.0.0.1:28495 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2416743 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2416743 T6032 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 2416745 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2416747 T6098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@728ae8ec name:ZooKeeperConnection Watcher:127.0.0.1:28495/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2416747 T6032 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2416752 T6032 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 2417758 T6032 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57476_ot%2Fs
[junit4] 2> 2417759 T6032 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57476_ot%2Fs
[junit4] 2> 2417763 T6032 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 2417771 T6100 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 2417771 T6100 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2417772 T6100 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 2417772 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2417775 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "node_name":"127.0.0.1:57476_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:57476/ot/s",
[junit4] 2> "state":"down"}
[junit4] 2> 2417775 T6055 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 2417775 T6055 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 2417778 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2417778 T6099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2417778 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2417778 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2418773 T6100 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 2418774 T6100 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 2418775 T6100 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 2418775 T6100 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2418776 T6100 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2418776 T6100 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/collection1/'
[junit4] 2> 2418778 T6100 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 2418779 T6100 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 2418780 T6100 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 2418847 T6100 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 2418897 T6100 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 2418998 T6100 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 2419013 T6100 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 2419375 T6100 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 2419378 T6100 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 2419380 T6100 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2419385 T6100 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 2419402 T6100 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 2419403 T6100 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 2419403 T6100 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/
[junit4] 2> 2419404 T6100 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10cae060
[junit4] 2> 2419405 T6100 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2
[junit4] 2> 2419406 T6100 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/index/
[junit4] 2> 2419407 T6100 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 2419408 T6100 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/index
[junit4] 2> 2419409 T6100 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=3, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 2419414 T6100 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2419414 T6100 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2419419 T6100 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 2419419 T6100 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 2419420 T6100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 2419420 T6100 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 2419420 T6100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 2419420 T6100 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 2419421 T6100 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2419421 T6100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 2419421 T6100 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 2419422 T6100 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2419423 T6100 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2419423 T6100 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 2419424 T6100 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 2419424 T6100 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 2419425 T6100 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 2419425 T6100 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 2419425 T6100 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 2419426 T6100 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 2419426 T6100 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 2419427 T6100 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 2419447 T6100 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2419449 T6100 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2419451 T6100 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2419453 T6100 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 2419457 T6100 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 2419457 T6100 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 2419458 T6100 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=266024004, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2419460 T6100 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 2419460 T6100 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 2419461 T6100 oass.SolrIndexSearcher.<init> Opening Searcher@3c007afe[collection1] main
[junit4] 2> 2419461 T6100 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 2419462 T6100 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 2419462 T6100 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 2419463 T6100 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2419463 T6100 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 2419463 T6100 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2419464 T6100 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 2419464 T6100 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2419464 T6100 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 2419465 T6100 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 2419469 T6101 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c007afe[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2419472 T6100 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 2419473 T6104 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57476/ot/s collection:collection1 shard:shard1
[junit4] 2> 2419473 T6032 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 2419474 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 2419474 T6104 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 2419478 T6104 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 2419480 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2419480 T6104 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 2419481 T6104 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1423 name=collection1 org.apache.solr.core.SolrCore@8d18882 url=http://127.0.0.1:57476/ot/s/collection1 node=127.0.0.1:57476_ot%2Fs C1423_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:57476_ot%2Fs, core=collection1, base_url=http://127.0.0.1:57476/ot/s, state=down}
[junit4] 2> 2419481 T6104 C1423 P57476 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57476/ot/s/collection1/
[junit4] 2> 2419481 T6104 C1423 P57476 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 2419481 T6104 C1423 P57476 oasc.SyncStrategy.syncToMe http://127.0.0.1:57476/ot/s/collection1/ has no replicas
[junit4] 2> 2419481 T6104 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57476/ot/s/collection1/ shard1
[junit4] 2> 2419482 T6104 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 2419485 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419485 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419485 T6099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419485 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419491 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2419492 T6099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419492 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419492 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419492 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419537 T6104 oasc.ZkController.register We are http://127.0.0.1:57476/ot/s/collection1/ and leader is http://127.0.0.1:57476/ot/s/collection1/
[junit4] 2> 2419538 T6104 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57476/ot/s
[junit4] 2> 2419538 T6104 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 2419538 T6104 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 2419538 T6104 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 2419540 T6059 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 2419542 T6055 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "node_name":"127.0.0.1:57476_ot%2Fs",
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "base_url":"http://127.0.0.1:57476/ot/s",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "state":"active"}
[junit4] 2> 2419646 T6059 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419646 T6066 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419646 T6080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2419646 T6099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2420940 T6032 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 2420941 T6032 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 2420944 T6032 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64132
[junit4] 2> 2420944 T6032 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 2420945 T6032 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 2420945 T6032 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-005
[junit4] 2> 2420945 T6032 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-005/'
[junit4] 2> 2420977 T6032 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-005/solr.xml
[junit4] 2> 2421035 T6032 oasc.CoreContainer.<init> New CoreContainer 1969587337
[junit4] 2> 2421035 T6032 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-005/]
[junit4] 2> 2421036 T6032 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 2421036 T6032 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 2421037 T6032 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 2421037 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 2421037 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 2421037 T6032 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 2421038 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 2421038 T6032 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 2421038 T6032 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 2421038 T6032 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 2421039 T6032 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 2421039 T6032 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 2421039 T6032 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 2421039 T6032 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 2421040 T6032 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:28495/solr
[junit4] 2> 2421040 T6032 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 2421040 T6032 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 2421041 T6032 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2421042 T6115 oascc.ConnectionManager.process Watcher org.a
[...truncated too long message...]
3594 T6032 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 2713594 T6032 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 2713594 T6032 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8;done=false>>]
[junit4] 2> 2713595 T6032 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8
[junit4] 2> 2713595 T6032 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8/index.20141011142733122 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8/index.20141011142733122;done=false>>]
[junit4] 2> 2713595 T6032 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001/tempDir-001/jetty8/index.20141011142733122
[junit4] 2> 2713596 T6032 oasc.Overseer.close Overseer (id=92604829481631768-127.0.0.1:11937_ot%2Fs-n_0000000010) closing
[junit4] 2> 2713596 T6541 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:11937_ot%2Fs
[junit4] 2> 2713598 T6540 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2713599 T6032 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/ot/s,null}
[junit4] 2> 2718600 T6032 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 2718602 T6032 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:28495 28495
[junit4] 2> 2718712 T6033 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:28495 28495
[junit4] 2> 2718714 T6033 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/clusterstate.json
[junit4] 2> 14 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/overseer_elect/election/92604829481631760-127.0.0.1:10702_ot%2Fs-n_0000000006
[junit4] 2> 3 /solr/collections/collection1/leader_elect/shard1/election/92604829481631760-core_node6-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/92604829481631747-127.0.0.1:28498_ot%2Fs-n_0000000000
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard2/election/92604829481631758-core_node5-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/92604829481631750-127.0.0.1:28502_ot%2Fs-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 10 /solr/live_nodes
[junit4] 2> 6 /solr/overseer/queue
[junit4] 2> 6 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=184858502002211B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_MA -Dtests.timezone=Greenwich -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 311s J1 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: shard2 is not consistent. Got 943 from http://127.0.0.1:65501/ot/s/collection1lastClient and got 945 from http://127.0.0.1:11446/ot/s/collection1
[junit4] > at __randomizedtesting.SeedInfo.seed([184858502002211B:99AED648575D4127]:0)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1255)
[junit4] > at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:232)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2718730 T6032 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest-184858502002211B-001
[junit4] 2> 311198 T6031 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene50): {id=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Memory doPackFST= false), rnd_b=PostingsFormat(name=Direct), _version_=PostingsFormat(name=Memory doPackFST= false), a_i=PostingsFormat(name=Direct)}, docValues:{}, sim=DefaultSimilarity, locale=ar_MA, timezone=Greenwich
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=173040624,total=374865920
[junit4] 2> NOTE: All tests run in this JVM: [TestStressUserVersions, SolrCoreTest, DeleteShardTest, ExternalFileFieldSortTest, DebugComponentTest, AnalyticsMergeStrategyTest, PreAnalyzedUpdateProcessorTest, ExternalCollectionsTest, TestManagedSchemaFieldTypeResource, TestRecovery, TestSolrXmlPersistor, TestShardHandlerFactory, TestCollapseQParserPlugin, TestFieldCollectionResource, HttpPartitionTest, DistributedSuggestComponentTest, TestPartialUpdateDeduplication, DisMaxRequestHandlerTest, JSONWriterTest, TestCloudPivotFacet, TestSweetSpotSimilarityFactory, TestCollationFieldDocValues, PreAnalyzedFieldTest, RankQueryTest, DocumentBuilderTest, UpdateParamsTest, DirectSolrSpellCheckerTest, TestSystemIdResolver, TestClassNameShortening, SpatialFilterTest, TestManagedSchemaFieldResource, FullSolrCloudDistribCmdsTest, SharedFSAutoReplicaFailoverTest, CurrencyFieldOpenExchangeTest, TestIntervalFaceting, TestDocumentBuilder, StressHdfsTest, TestRandomMergePolicy, TestInitParams, TestFastOutputStream, SolrRequestParserTest, TestSchemaVersionResource, ExitableDirectoryReaderTest, NotRequiredUniqueKeyTest, TestComponentsName, TestManagedResource, TestBinaryField, TestHighFrequencyDictionaryFactory, FastVectorHighlighterTest, TestCodecSupport, TestStressVersions, LeaderElectionIntegrationTest, ShardRoutingTest, ClusterStateUpdateTest, TestRandomFaceting, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, DistributedTermsComponentTest, TestIndexSearcher, SolrIndexSplitterTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, PolyFieldTest, SolrCoreCheckLockOnStartupTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, DocValuesMultiTest, DistanceFunctionTest, LukeRequestHandlerTest, MoreLikeThisHandlerTest, TestArbitraryIndexDir, RegexBoostProcessorTest, TestCollationField, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestLFUCache, CopyFieldTest, BadComponentTest, SampleTest, TestSearchPerf, MinimalSchemaTest, OutputWriterTest, TestSolrCoreProperties, TestIBSimilarityFactory, TestDefaultSimilarityFactory, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 311.99s, 1 test, 1 failure <<< FAILURES!
[...truncated 296 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 437 suites, 1784 tests, 1 failure, 53 ignored (9 assumptions)
Total time: 211 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #651
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 693727176 bytes
Compression is 0.0%
Took 2 min 0 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 656 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/656/
1 tests failed.
REGRESSION: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
Error Message:
IOException occured when talking to server at: http://127.0.0.1:13373/w/d/collection1
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:13373/w/d/collection1
at __randomizedtesting.SeedInfo.seed([7E0577D042F3955E:FFE3F9C835ACF562]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:583)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
at org.apache.http.impl.io.AbstractSessionOutputBuffer.writeLine(AbstractSessionOutputBuffer.java:258)
at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:59)
at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:46)
at org.apache.http.impl.io.AbstractMessageWriter.write(AbstractMessageWriter.java:109)
at org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:254)
at org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader(DefaultClientConnection.java:270)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestHeader(ManagedClientConnectionImpl.java:209)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:203)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
... 49 more
Build Log:
[...truncated 12079 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/init-core-data-001
[junit4] 2> 1922968 T11190 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /w/d
[junit4] 2> 1922975 T11190 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 1922976 T11190 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1922977 T11191 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1923077 T11190 oasc.ZkTestServer.run start zk server on port:13370
[junit4] 2> 1923078 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1923079 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1923084 T11197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27fc461a name:ZooKeeperConnection Watcher:127.0.0.1:13370 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1923085 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1923085 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1923085 T11190 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1923090 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1923093 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1923094 T11199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ac27f8 name:ZooKeeperConnection Watcher:127.0.0.1:13370/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1923095 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1923095 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1923095 T11190 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1923097 T11190 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1923099 T11190 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1923100 T11190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1923103 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1923103 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1923106 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1923107 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1923209 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1923210 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1923212 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1923212 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1923215 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1923215 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1923217 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1923218 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1923220 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1923220 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1923222 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1923223 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1923225 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1923225 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1923227 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1923228 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1923230 T11190 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1923231 T11190 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1924528 T11190 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1924538 T11190 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13373
[junit4] 2> 1924539 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1924540 T11190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1924540 T11190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002
[junit4] 2> 1924541 T11190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/'
[junit4] 2> 1924586 T11190 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/solr.xml
[junit4] 2> 1924668 T11190 oasc.CoreContainer.<init> New CoreContainer 1536505534
[junit4] 2> 1924669 T11190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/]
[junit4] 2> 1924670 T11190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1924670 T11190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1924671 T11190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1924671 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1924671 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1924672 T11190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1924672 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1924672 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1924673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1924673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1924674 T11190 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1924674 T11190 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1924675 T11190 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1924675 T11190 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1924675 T11190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13370/solr
[junit4] 2> 1924676 T11190 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1924676 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1924677 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1924678 T11210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@213d22eb name:ZooKeeperConnection Watcher:127.0.0.1:13370 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1924680 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1924681 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1924682 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1924683 T11212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a0e824d name:ZooKeeperConnection Watcher:127.0.0.1:13370/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1924683 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1924684 T11190 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1924686 T11190 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1924687 T11190 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1924689 T11190 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1924691 T11190 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1924692 T11190 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1924693 T11190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13373_w%2Fd
[junit4] 2> 1924694 T11190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13373_w%2Fd
[junit4] 2> 1924695 T11190 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1924696 T11190 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1924698 T11190 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1924699 T11190 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:13373_w%2Fd
[junit4] 2> 1924699 T11190 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1924700 T11190 oasc.Overseer.start Overseer (id=92619499739611139-127.0.0.1:13373_w%2Fd-n_0000000000) starting
[junit4] 2> 1924702 T11190 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1924706 T11190 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 1924707 T11214 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1924707 T11190 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1924708 T11190 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1924709 T11190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1924711 T11213 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1924715 T11216 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1924715 T11216 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1924716 T11216 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1924716 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1924718 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "numShards":"1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13373_w%2Fd",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:13373/w/d"}
[junit4] 2> 1924718 T11213 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1924718 T11213 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 1924718 T11213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1924720 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1925716 T11216 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1925717 T11216 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1925718 T11216 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1925718 T11216 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1925719 T11216 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1925719 T11216 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/collection1/'
[junit4] 2> 1925721 T11216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 1925722 T11216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 1925723 T11216 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 1925787 T11216 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1925832 T11216 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1925934 T11216 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1925963 T11216 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1926305 T11216 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1926308 T11216 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1926309 T11216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1926314 T11216 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1926332 T11216 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1926333 T11216 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1926333 T11216 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/control/data/
[junit4] 2> 1926333 T11216 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@598d4938
[junit4] 2> 1926334 T11216 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/control/data
[junit4] 2> 1926334 T11216 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/control/data/index/
[junit4] 2> 1926334 T11216 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1926335 T11216 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/control/data/index
[junit4] 2> 1926335 T11216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=31.3515625, floorSegmentMB=0.287109375, forceMergeDeletesPctAllowed=17.871216511253312, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1926336 T11216 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-fa290c5)),segFN=segments_1,generation=1}
[junit4] 2> 1926336 T11216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1926339 T11216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1926340 T11216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1926340 T11216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1926340 T11216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1926340 T11216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1926341 T11216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1926341 T11216 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1926341 T11216 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1926342 T11216 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1926342 T11216 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1926343 T11216 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1926343 T11216 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1926344 T11216 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1926344 T11216 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1926344 T11216 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1926344 T11216 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1926345 T11216 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1926345 T11216 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1926345 T11216 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1926346 T11216 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1926365 T11216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1926367 T11216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1926369 T11216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1926371 T11216 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1926376 T11216 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1926376 T11216 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1926377 T11216 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4275883296971833]
[junit4] 2> 1926378 T11216 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-fa290c5)),segFN=segments_1,generation=1}
[junit4] 2> 1926378 T11216 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1926378 T11216 oass.SolrIndexSearcher.<init> Opening Searcher@614a1823[collection1] main
[junit4] 2> 1926379 T11216 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1926379 T11216 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1926380 T11216 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1926380 T11216 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1926380 T11216 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1926381 T11216 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1926381 T11216 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1926381 T11216 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1926382 T11216 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 1926384 T11216 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 1926384 T11216 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1926384 T11216 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1926387 T11218 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@614a1823[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1926388 T11216 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1926389 T11221 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13373/w/d collection:control_collection shard:shard1
[junit4] 2> 1926389 T11190 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1926389 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1926389 T11221 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1926390 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1926391 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1926392 T11223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14bed9b4 name:ZooKeeperConnection Watcher:127.0.0.1:13370/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1926393 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1926393 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1926394 T11190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1926395 T11221 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1926396 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1926396 T11221 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1926397 T11221 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2964 name=collection1 org.apache.solr.core.SolrCore@90b7e7c url=http://127.0.0.1:13373/w/d/collection1 node=127.0.0.1:13373_w%2Fd C2964_STATE=coll:control_collection core:collection1 props:{node_name=127.0.0.1:13373_w%2Fd, core=collection1, state=down, base_url=http://127.0.0.1:13373/w/d}
[junit4] 2> 1926397 T11221 C2964 P13373 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13373/w/d/collection1/
[junit4] 2> 1926397 T11221 C2964 P13373 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1926397 T11190 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1926397 T11221 C2964 P13373 oasc.SyncStrategy.syncToMe http://127.0.0.1:13373/w/d/collection1/ has no replicas
[junit4] 2> 1926398 T11221 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13373/w/d/collection1/ shard1
[junit4] 2> 1926398 T11221 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1926400 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1926400 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1926406 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1926406 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1926406 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1926453 T11221 oasc.ZkController.register We are http://127.0.0.1:13373/w/d/collection1/ and leader is http://127.0.0.1:13373/w/d/collection1/
[junit4] 2> 1926454 T11221 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13373/w/d
[junit4] 2> 1926454 T11221 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1926454 T11221 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1926454 T11221 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1926456 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1926459 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13373_w%2Fd",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":"shard1",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:13373/w/d"}
[junit4] 2> 1926562 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1926562 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1927625 T11190 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1927626 T11190 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1927630 T11190 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13381
[junit4] 2> 1927630 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1927631 T11190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1927631 T11190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003
[junit4] 2> 1927632 T11190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/'
[junit4] 2> 1927676 T11190 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/solr.xml
[junit4] 2> 1927763 T11190 oasc.CoreContainer.<init> New CoreContainer 1411907026
[junit4] 2> 1927763 T11190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/]
[junit4] 2> 1927765 T11190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1927765 T11190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1927765 T11190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1927766 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1927766 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1927766 T11190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1927767 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1927767 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1927767 T11190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1927768 T11190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1927768 T11190 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1927769 T11190 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1927769 T11190 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1927770 T11190 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1927770 T11190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13370/solr
[junit4] 2> 1927770 T11190 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1927771 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1927772 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1927773 T11235 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d891128 name:ZooKeeperConnection Watcher:127.0.0.1:13370 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1927775 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1927775 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1927777 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1927779 T11237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@772fa434 name:ZooKeeperConnection Watcher:127.0.0.1:13370/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1927779 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1927783 T11190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1928787 T11190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13381_w%2Fd
[junit4] 2> 1928788 T11190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13381_w%2Fd
[junit4] 2> 1928790 T11190 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1928795 T11239 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1928796 T11239 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1928796 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1928797 T11239 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1928798 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13381_w%2Fd",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:13381/w/d"}
[junit4] 2> 1928798 T11213 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1928799 T11213 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
[junit4] 2> 1928799 T11213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1928801 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1928801 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1928801 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1929797 T11239 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1929798 T11239 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1929798 T11239 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1929799 T11239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1929799 T11239 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1929799 T11239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/collection1/'
[junit4] 2> 1929801 T11239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 1929801 T11239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 1929802 T11239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 1929853 T11239 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1929887 T11239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1929988 T11239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1929995 T11239 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1930355 T11239 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1930359 T11239 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1930360 T11239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1930365 T11239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1930383 T11239 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1930384 T11239 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1930384 T11239 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty1/
[junit4] 2> 1930384 T11239 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@598d4938
[junit4] 2> 1930385 T11239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty1
[junit4] 2> 1930385 T11239 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty1/index/
[junit4] 2> 1930386 T11239 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1930388 T11239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty1/index
[junit4] 2> 1930388 T11239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=31.3515625, floorSegmentMB=0.287109375, forceMergeDeletesPctAllowed=17.871216511253312, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1930389 T11239 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-4875a97b)),segFN=segments_1,generation=1}
[junit4] 2> 1930390 T11239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1930393 T11239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1930393 T11239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1930393 T11239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1930394 T11239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1930394 T11239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1930394 T11239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1930395 T11239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1930395 T11239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1930395 T11239 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1930396 T11239 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1930396 T11239 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1930397 T11239 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1930397 T11239 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1930398 T11239 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1930398 T11239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1930398 T11239 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1930398 T11239 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1930399 T11239 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1930399 T11239 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1930400 T11239 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1930424 T11239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1930426 T11239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1930428 T11239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1930430 T11239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1930435 T11239 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1930435 T11239 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1930436 T11239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4275883296971833]
[junit4] 2> 1930436 T11239 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-4875a97b)),segFN=segments_1,generation=1}
[junit4] 2> 1930437 T11239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1930437 T11239 oass.SolrIndexSearcher.<init> Opening Searcher@4f637b4f[collection1] main
[junit4] 2> 1930437 T11239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1930438 T11239 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1930438 T11239 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1930439 T11239 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1930439 T11239 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1930439 T11239 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1930440 T11239 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 1930441 T11239 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1930441 T11239 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 1930441 T11239 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1930444 T11240 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4f637b4f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1930447 T11239 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1930448 T11243 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13381/w/d collection:collection1 shard:shard2
[junit4] 2> 1930448 T11190 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1930449 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1930449 T11243 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1930454 T11243 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1930455 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1930456 T11243 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1930456 T11243 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2965 name=collection1 org.apache.solr.core.SolrCore@50aaa1fb url=http://127.0.0.1:13381/w/d/collection1 node=127.0.0.1:13381_w%2Fd C2965_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:13381_w%2Fd, core=collection1, state=down, base_url=http://127.0.0.1:13381/w/d}
[junit4] 2> 1930456 T11243 C2965 P13381 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13381/w/d/collection1/
[junit4] 2> 1930456 T11243 C2965 P13381 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1930457 T11243 C2965 P13381 oasc.SyncStrategy.syncToMe http://127.0.0.1:13381/w/d/collection1/ has no replicas
[junit4] 2> 1930457 T11243 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13381/w/d/collection1/ shard2
[junit4] 2> 1930457 T11243 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1930460 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930460 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930460 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930466 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1930467 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930467 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930467 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930512 T11243 oasc.ZkController.register We are http://127.0.0.1:13381/w/d/collection1/ and leader is http://127.0.0.1:13381/w/d/collection1/
[junit4] 2> 1930513 T11243 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13381/w/d
[junit4] 2> 1930513 T11243 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1930513 T11243 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1930513 T11243 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1930515 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1930517 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13381_w%2Fd",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":"shard2",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:13381/w/d"}
[junit4] 2> 1930621 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930621 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1930621 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1932495 T11190 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1932496 T11190 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1932502 T11190 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13389
[junit4] 2> 1932503 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1932503 T11190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1932503 T11190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004
[junit4] 2> 1932504 T11190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/'
[junit4] 2> 1932554 T11190 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/solr.xml
[junit4] 2> 1932670 T11190 oasc.CoreContainer.<init> New CoreContainer 1818097372
[junit4] 2> 1932670 T11190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/]
[junit4] 2> 1932671 T11190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1932672 T11190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1932672 T11190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1932673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1932673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 1932673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1932673 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1932674 T11190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1932674 T11190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1932674 T11190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1932675 T11190 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 1932676 T11190 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1932676 T11190 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1932676 T11190 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1932677 T11190 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13370/solr
[junit4] 2> 1932677 T11190 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1932677 T11190 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 1932678 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1932680 T11254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cfa1d19 name:ZooKeeperConnection Watcher:127.0.0.1:13370 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1932682 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1932682 T11190 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 1932684 T11190 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1932685 T11256 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e58e7b5 name:ZooKeeperConnection Watcher:127.0.0.1:13370/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1932685 T11190 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1932691 T11190 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1933695 T11190 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13389_w%2Fd
[junit4] 2> 1933696 T11190 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13389_w%2Fd
[junit4] 2> 1933699 T11190 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 1933706 T11258 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1933706 T11258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1933707 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1933707 T11258 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1933709 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13389_w%2Fd",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":null,
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:13389/w/d"}
[junit4] 2> 1933709 T11213 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1933709 T11213 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1933712 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1933712 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1933712 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1933712 T11257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1934707 T11258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1934708 T11258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1934709 T11258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1934709 T11258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1934710 T11258 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1934710 T11258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/collection1/'
[junit4] 2> 1934711 T11258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 1934712 T11258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 1934713 T11258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 1934778 T11258 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 1934823 T11258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1934924 T11258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1934933 T11258 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1935285 T11258 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1935287 T11258 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1935289 T11258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1935294 T11258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1935314 T11258 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1935314 T11258 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1935314 T11258 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty2/
[junit4] 2> 1935315 T11258 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@598d4938
[junit4] 2> 1935315 T11258 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty2
[junit4] 2> 1935316 T11258 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty2/index/
[junit4] 2> 1935316 T11258 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1935324 T11258 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-001/jetty2/index
[junit4] 2> 1935325 T11258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=31.3515625, floorSegmentMB=0.287109375, forceMergeDeletesPctAllowed=17.871216511253312, segmentsPerTier=28.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 1935326 T11258 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-52659d83)),segFN=segments_1,generation=1}
[junit4] 2> 1935326 T11258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1935330 T11258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1935330 T11258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1935330 T11258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1935331 T11258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1935331 T11258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1935331 T11258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1935332 T11258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1935332 T11258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1935332 T11258 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1935333 T11258 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1935333 T11258 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1935334 T11258 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 1935334 T11258 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1935335 T11258 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1935335 T11258 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1935335 T11258 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1935336 T11258 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1935336 T11258 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1935337 T11258 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1935337 T11258 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1935360 T11258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1935363 T11258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1935365 T11258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1935367 T11258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1935372 T11258 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1935372 T11258 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1935373 T11258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4275883296971833]
[junit4] 2> 1935373 T11258 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-52659d83)),segFN=segments_1,generation=1}
[junit4] 2> 1935374 T11258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1935374 T11258 oass.SolrIndexSearcher.<init> Opening Searcher@6c534461[collection1] main
[junit4] 2> 1935374 T11258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1935375 T11258 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1935375 T11258 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1935376 T11258 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1935376 T11258 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1935376 T11258 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1935377 T11258 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 1935377 T11258 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1935378 T11258 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 1935378 T11258 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1935381 T11259 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6c534461[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1935383 T11258 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1935384 T11262 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13389/w/d collection:collection1 shard:shard3
[junit4] 2> 1935384 T11190 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1935385 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1935385 T11262 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1935389 T11262 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 1935391 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1935391 T11262 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1935392 T11262 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C2966 name=collection1 org.apache.solr.core.SolrCore@509a9835 url=http://127.0.0.1:13389/w/d/collection1 node=127.0.0.1:13389_w%2Fd C2966_STATE=coll:collection1 core:collection1 props:{node_name=127.0.0.1:13389_w%2Fd, core=collection1, state=down, base_url=http://127.0.0.1:13389/w/d}
[junit4] 2> 1935392 T11262 C2966 P13389 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13389/w/d/collection1/
[junit4] 2> 1935392 T11262 C2966 P13389 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1935392 T11262 C2966 P13389 oasc.SyncStrategy.syncToMe http://127.0.0.1:13389/w/d/collection1/ has no replicas
[junit4] 2> 1935393 T11262 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13389/w/d/collection1/ shard3
[junit4] 2> 1935393 T11262 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1935396 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935396 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935396 T11257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935396 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935401 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1935402 T11257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935402 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935402 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935402 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935448 T11262 oasc.ZkController.register We are http://127.0.0.1:13389/w/d/collection1/ and leader is http://127.0.0.1:13389/w/d/collection1/
[junit4] 2> 1935448 T11262 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13389/w/d
[junit4] 2> 1935448 T11262 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1935448 T11262 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1935449 T11262 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1935450 T11217 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1935452 T11213 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "numShards":"3",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:13389_w%2Fd",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "operation":"state",
[junit4] 2> "shard":"shard3",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:13389/w/d"}
[junit4] 2> 1935556 T11217 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935556 T11224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935556 T11257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1935556 T11238 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1937289 T11190 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1937290 T11190 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1937296 T11190 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13395
[junit4] 2> 1937296 T11190 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1937297 T11190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1937297 T11190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-005
[junit4] 2> 1937298 T11190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-005/'
[junit4] 2> 1937360 T11190 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDistribCmdsTest-7E0577D042F3955E-001/tempDir-005/solr.xml
[junit4] 2> 1937466 T11190 oasc.CoreContainer.<init> New CoreContainer 2023977510
[junit4] 2> 1937467 T11190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.FullSolrCloudDis
[...truncated too long message...]
ne41VarGapFixedInterval), 3child1_s=PostingsFormat(name=Asserting), 25parent_f1_s=FSTOrd41, 11grand5_s=FSTOrd41, 10grand7_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 47parent_f1_s=FSTOrd41, 8parent_f2_s=FSTOrd41, 9parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 3parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 44parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 10grand0_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 12parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 47parent_f2_s=PostingsFormat(name=Asserting), 55parent_f2_s=FSTOrd41, 11grand7_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 5parent_f2_s=PostingsFormat(name=Asserting), 37parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 7parent_f1_s=FSTOrd41, 10grand4_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 6grand4_s=PostingsFormat(name=Asserting), 10parent_f1_s=FSTOrd41, 17parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 57parent_f1_s=PostingsFormat(name=Asserting), 5parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), text_t=PostingsFormat(name=Lucene41VarGapFixedInterval), 45parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 42parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 38parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 4child2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 10grand6_s=FSTOrd41, 10grand8_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 3grand2_s=FSTOrd41, 38parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 1grand0_s=FSTOrd41, 6child2_s=PostingsFormat(name=Asserting), 5grand3_s=PostingsFormat(name=Asserting), 14parent_f1_s=FSTOrd41, 29parent_f1_s=FSTOrd41, 2grand0_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 40parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 6child3_s=FSTOrd41, 36parent_f1_s=FSTOrd41, 2child1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 11grand4_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 56parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 5child1_s=PostingsFormat(name=Asserting), 11grand9_s=FSTOrd41, 23parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 6grand1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 31parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 50parent_f1_s=FSTOrd41, 16parent_f2_s=PostingsFormat(name=Asserting), 27parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 54parent_f1_s=FSTOrd41, 8parent_f1_s=PostingsFormat(name=Asserting), 3child2_s=FSTOrd41, 30parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 29parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 6child4_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 18parent_f1_s=FSTOrd41, 35parent_f1_s=PostingsFormat(name=Asserting), type_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 7grand6_s=FSTOrd41, 0parent_f1_s=PostingsFormat(name=Asserting), 51parent_f2_s=FSTOrd41, 4parent_f2_s=FSTOrd41, 9grand4_s=FSTOrd41, 40parent_f2_s=FSTOrd41, 4child1_s=FSTOrd41, 33parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 54parent_f2_s=PostingsFormat(name=Asserting), 21parent_f2_s=PostingsFormat(name=Asserting), 50parent_f2_s=PostingsFormat(name=Asserting), 32parent_f1_s=FSTOrd41, 32parent_f2_s=PostingsFormat(name=Asserting), 4grand0_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 49parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 53parent_f1_s=PostingsFormat(name=Asserting), 8grand2_s=PostingsFormat(name=Asserting), 8grand1_s=FSTOrd41, 11grand10_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 48parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 19parent_f2_s=FSTOrd41, 31parent_f1_s=PostingsFormat(name=Asserting), 3grand0_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 11grand2_s=PostingsFormat(name=Asserting), 13parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 5grand1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 48parent_f2_s=FSTOrd41, 30parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 12parent_f2_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 4parent_f1_s=PostingsFormat(name=Asserting), 41parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 8grand7_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 53parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 10grand1_s=PostingsFormat(name=Asserting), 36parent_f2_s=PostingsFormat(name=Asserting), 15parent_f2_s=FSTOrd41, 3child0_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 34parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 49parent_f1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 11grand1_s=FSTOrd41, 7grand4_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 33parent_f2_s=FSTOrd41, 4child3_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 2grand1_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 46parent_f2_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 9grand6_s=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 10grand9_s=PostingsFormat(name=Asserting), rnd_b=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), 51parent_f1_s=PostingsFormat(name=Lucene41VarGapFixedInterval), 5grand0_s=FSTOrd41, _version_=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{}, sim=DefaultSimilarity, locale=es_PE, timezone=America/Knox_IN
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=182300440,total=477102080
[junit4] 2> NOTE: All tests run in this JVM: [TestTrie, TestCloudSchemaless, TestCursorMarkWithoutUniqueKey, TestRequestStatusCollectionAPI, BlockCacheTest, TestReplicationHandlerBackup, TestSimpleQParserPlugin, TestValueSourceCache, TestIBSimilarityFactory, TestDistribDocBasedVersion, TriLevelCompositeIdRoutingTest, DirectSolrConnectionTest, CSVRequestHandlerTest, PrimUtilsTest, TestSolrXml, HdfsCollectionsAPIDistributedZkTest, AlternateDirectoryTest, TestUtils, MoreLikeThisHandlerTest, DistributedFacetPivotSmallTest, IndexSchemaRuntimeFieldTest, CacheHeaderTest, TestSolrDeletionPolicy2, TestWriterPerf, DistributedQueryElevationComponentTest, SoftAutoCommitTest, ConnectionManagerTest, CoreAdminCreateDiscoverTest, BasicDistributedZkTest, TestBinaryResponseWriter, TestStressReorder, ParsingFieldUpdateProcessorsTest, ChangedSchemaMergeTest, TestBulkSchemaConcurrent, DistanceFunctionTest, TestReRankQParserPlugin, MultiThreadedOCPTest, ZkSolrClientTest, TestArbitraryIndexDir, SolrCmdDistributorTest, ReturnFieldsTest, TestDefaultSimilarityFactory, TestDefaultSearchFieldResource, TestOrdValues, DistributedSpellCheckComponentTest, DateRangeFieldTest, SynonymTokenizerTest, TestWordDelimiterFilterFactory, BadComponentTest, HdfsBasicDistributedZkTest, MinimalSchemaTest, TestSolrDeletionPolicy1, TestRestManager, MigrateRouteKeyTest, TestFileDictionaryLookup, CollectionsAPIDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest]
[junit4] Completed on J0 in 150.69s, 1 test, 1 error <<< FAILURES!
[...truncated 801 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 437 suites, 1784 tests, 1 error, 53 ignored (9 assumptions)
Total time: 240 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #651
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 685462704 bytes
Compression is 0.0%
Took 2 min 2 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 655 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/655/
4 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<2>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<2>
at __randomizedtesting.SeedInfo.seed([7CEAB97D489AD51F:FD0C37653FC5B523]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:153)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup: 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
at __randomizedtesting.SeedInfo.seed([7CEAB97D489AD51F]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandlerBackup
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at sun.net.NetworkClient.doConnect(NetworkClient.java:180) at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323) at java.net.URL.openStream(URL.java:1037) at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
at java.net.URL.openStream(URL.java:1037)
at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
at __randomizedtesting.SeedInfo.seed([7CEAB97D489AD51F]:0)
Build Log:
[...truncated 11703 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/init-core-data-001
[junit4] 2> 939285 T2921 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 939285 T2921 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 939292 T2921 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 939293 T2921 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 939294 T2922 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 939394 T2921 oasc.ZkTestServer.run start zk server on port:31640
[junit4] 2> 939395 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 939396 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939400 T2928 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f7edb14 name:ZooKeeperConnection Watcher:127.0.0.1:31640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939400 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939400 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 939401 T2921 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 939403 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 939404 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939405 T2930 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c80e286 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939405 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939406 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 939406 T2921 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 939408 T2921 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 939409 T2921 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 939410 T2921 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 939412 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 939413 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 939415 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 939416 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 939518 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 939518 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 939520 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 939521 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 939523 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 939524 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 939525 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 939526 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 939528 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 939528 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 939530 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 939531 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 939532 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 939533 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 939535 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 939536 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 939537 T2921 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 939538 T2921 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 939540 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 939541 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939542 T2932 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5e3634 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939542 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939543 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 941109 T2921 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 941115 T2921 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 941118 T2921 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31654
[junit4] 2> 941122 T2921 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 941122 T2921 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 941122 T2921 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002
[junit4] 2> 941123 T2921 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/'
[junit4] 2> 941184 T2921 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/solr.xml
[junit4] 2> 941273 T2921 oasc.CoreContainer.<init> New CoreContainer 2014182616
[junit4] 2> 941273 T2921 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/]
[junit4] 2> 941274 T2921 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 941275 T2921 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 941275 T2921 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 941275 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 941275 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 941275 T2921 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 941276 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 941276 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 941276 T2921 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 941276 T2921 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 941278 T2921 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 941280 T2921 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 941280 T2921 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 941280 T2921 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 941281 T2921 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:31640/solr
[junit4] 2> 941281 T2921 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 941281 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 941282 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 941283 T2943 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c26dd7b name:ZooKeeperConnection Watcher:127.0.0.1:31640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 941284 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 941285 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 941286 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 941287 T2945 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ec88a0 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 941287 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 941288 T2921 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 941290 T2921 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 941291 T2921 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 941293 T2921 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 941294 T2921 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 941296 T2921 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 941297 T2921 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31654_
[junit4] 2> 941297 T2921 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31654_
[junit4] 2> 941299 T2921 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 941300 T2921 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 941301 T2921 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 941302 T2921 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:31654_
[junit4] 2> 941302 T2921 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 941303 T2921 oasc.Overseer.start Overseer (id=92612672838565892-127.0.0.1:31654_-n_0000000000) starting
[junit4] 2> 941305 T2921 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 941310 T2921 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
[junit4] 2> 941310 T2947 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 941311 T2921 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 941312 T2921 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 941313 T2921 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 941315 T2946 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 941319 T2949 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 941319 T2949 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 941319 T2949 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 941319 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941321 T2946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "node_name":"127.0.0.1:31654_",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "base_url":"https://127.0.0.1:31654"}
[junit4] 2> 941321 T2946 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 941321 T2946 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
[junit4] 2> 941322 T2946 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 941323 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 942320 T2949 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 942321 T2949 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 942321 T2949 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 942322 T2949 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 942322 T2949 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 942322 T2949 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/collection1/'
[junit4] 2> 942324 T2949 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 942325 T2949 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 942325 T2949 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 942390 T2949 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 942435 T2949 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 942536 T2949 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 942545 T2949 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 942881 T2949 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 942883 T2949 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 942885 T2949 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 942890 T2949 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 942906 T2949 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 942906 T2949 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 942907 T2949 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/
[junit4] 2> 942907 T2949 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@484ff8fa
[junit4] 2> 942910 T2949 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data
[junit4] 2> 942911 T2949 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/index/
[junit4] 2> 942912 T2949 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 942913 T2949 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/index
[junit4] 2> 942914 T2949 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=29.9375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=2.5492460313621357, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31738124865380357
[junit4] 2> 942939 T2949 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 942939 T2949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 942943 T2949 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 942943 T2949 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 942944 T2949 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 942944 T2949 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 942944 T2949 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 942944 T2949 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 942945 T2949 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 942945 T2949 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 942945 T2949 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 942946 T2949 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 942946 T2949 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 942947 T2949 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 942947 T2949 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 942948 T2949 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 942948 T2949 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 942948 T2949 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 942949 T2949 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 942949 T2949 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 942949 T2949 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 942950 T2949 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 942968 T2949 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 942970 T2949 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 942972 T2949 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 942974 T2949 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 942977 T2949 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 942978 T2949 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 942978 T2949 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 942980 T2949 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 942980 T2949 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 942980 T2949 oass.SolrIndexSearcher.<init> Opening Searcher@1e677c93[collection1] main
[junit4] 2> 942981 T2949 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 942982 T2949 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 942982 T2949 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 942982 T2949 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 942982 T2949 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 942983 T2949 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 942983 T2949 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 942983 T2949 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 942984 T2949 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 942985 T2949 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 942986 T2949 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 942986 T2949 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 942988 T2951 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e677c93[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 942991 T2949 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 942992 T2954 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:31654 collection:control_collection shard:shard1
[junit4] 2> 942992 T2921 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 942992 T2921 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 942993 T2954 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 942997 T2954 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 942998 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 942998 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942998 T2954 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 942998 T2954 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 942998 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> ASYNC NEW_CORE C558 name=collection1 org.apache.solr.core.SolrCore@78ca96a7 url=https://127.0.0.1:31654/collection1 node=127.0.0.1:31654_ C558_STATE=coll:control_collection core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:31654_, base_url=https://127.0.0.1:31654}
[junit4] 2> 942999 T2954 C558 P31654 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:31654/collection1/
[junit4] 2> 942999 T2954 C558 P31654 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 943000 T2954 C558 P31654 oasc.SyncStrategy.syncToMe https://127.0.0.1:31654/collection1/ has no replicas
[junit4] 2> 943000 T2954 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:31654/collection1/ shard1
[junit4] 2> 943000 T2956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75490bd8 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 943000 T2954 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 943000 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 943001 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 943002 T2921 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 943003 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943003 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
[junit4] 2> 943004 T2921 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 943004 T2921 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
[junit4] 2> 943005 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 943005 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 943007 T2959 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ae5f6be name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 943007 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 943007 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 943008 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943009 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943009 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943012 T2946 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
[junit4] 2> 943012 T2946 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 943012 T2946 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
[junit4] 2> 943014 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943014 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
[junit4] 2> 943015 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943015 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943054 T2954 oasc.ZkController.register We are https://127.0.0.1:31654/collection1/ and leader is https://127.0.0.1:31654/collection1/
[junit4] 2> 943055 T2954 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:31654
[junit4] 2> 943055 T2954 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 943055 T2954 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 943055 T2954 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 943057 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943059 T2946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "state":"active",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "core":"collection1",
[junit4] 2> "node_name":"127.0.0.1:31654_",
[junit4] 2> "numShards":"3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "base_url":"https://127.0.0.1:31654"}
[junit4] 2> 943162 T2950 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943162 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 944215 T2921 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 944215 T2921 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 944219 T2921 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 944221 T2921 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31665
[junit4] 2> 944223 T2921 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 944223 T2921 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 944224 T2921 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003
[junit4] 2> 944224 T2921 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/'
[junit4] 2> 944284 T2921 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/solr.xml
[junit4] 2> 944366 T2921 oasc.CoreContainer.<init> New CoreContainer 923543013
[junit4] 2> 944366 T2921 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/]
[junit4] 2> 944367 T2921 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 944368 T2921 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 944368 T2921 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 944368 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 944369 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 944369 T2921 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 944369 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 944370 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 944370 T2921 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 944370 T2921 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 944373 T2921 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 944376 T2921 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 944376 T2921 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 944377 T2921 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 944377 T2921 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:31640/solr
[junit4] 2> 944377 T2921 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 944378 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 944379 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944380 T2970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@556ac6fd name:ZooKeeperConnection Watcher:127.0.0.1:31640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944381 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944382 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 944383 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944384 T2972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fa78d29 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944384 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944389 T2921 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 945393 T2921 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31665_
[junit4] 2> 945394 T2921 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31665_
[junit4] 2> 945396 T2921 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 945404 T2974 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 945405 T2974 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 945406 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945406 T2974 oasc.ZkController.preRegister Registering watch for external collection collection1
[junit4] 2> 945407 T2974 oascc.ZkStateReader.addZkWatch addZkWatch collection1
[junit4] 2> 945407 T2974 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper...
[junit4] 2> 945408 T2946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "node_name":"127.0.0.1:31665_",
[junit4] 2> "numShards":"3",
[junit4] 2> "shard":null,
[junit4] 2> "base_url":"https://127.0.0.1:31665"}
[junit4] 2> 945408 T2974 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0
[junit4] 2> 945408 T2974 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 945410 T2946 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 945410 T2946 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 945414 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
[junit4] 2> 945414 T2973 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 945415 T2973 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1
[junit4] 2> 946409 T2974 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 946410 T2974 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 946410 T2974 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 946411 T2974 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 946411 T2974 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 946411 T2974 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/collection1/'
[junit4] 2> 946413 T2974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 946414 T2974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 946414 T2974 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 946477 T2974 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 946522 T2974 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 946623 T2974 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 946632 T2974 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 946973 T2974 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 946976 T2974 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 946978 T2974 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 946983 T2974 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 946999 T2974 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 947000 T2974 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 947000 T2974 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/
[junit4] 2> 947000 T2974 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@484ff8fa
[junit4] 2> 947002 T2974 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1
[junit4] 2> 947003 T2974 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/index/
[junit4] 2> 947004 T2974 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 947005 T2974 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/index
[junit4] 2> 947006 T2974 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=29.9375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=2.5492460313621357, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31738124865380357
[junit4] 2> 947011 T2974 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 947011 T2974 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 947015 T2974 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 947016 T2974 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 947016 T2974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 947016 T2974 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 947017 T2974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 947017 T2974 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 947017 T2974 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 947018 T2974 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 947018 T2974 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 947019 T2974 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 947019 T2974 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 947020 T2974 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 947020 T2974 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 947021 T2974 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 947021 T2974 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 947021 T2974 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 947022 T2974 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 947022 T2974 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 947023 T2974 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 947023 T2974 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 947043 T2974 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 947045 T2974 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 947048 T2974 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 947050 T2974 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 947053 T2974 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 947053 T2974 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 947054 T2974 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 947056 T2974 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 947056 T2974 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 947056 T2974 oass.SolrIndexSearcher.<init> Opening Searcher@298e93a6[collection1] main
[junit4] 2> 947057 T2974 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 947057 T2974 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 947058 T2974 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 947058 T2974 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 947058 T2974 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 947058 T2974 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 947059 T2974 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 947059 T2974 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 947060 T2974 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 947060 T2974 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 947063 T2975 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@298e93a6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 947066 T2974 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 947067 T2978 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:31665 collection:collection1 shard:shard1
[junit4] 2> 947067 T2921 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 947068 T2921 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 947068 T2978 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 947072 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 947073 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 947074 T2978 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 947074 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C559 name=collection1 org.apache.solr.core.SolrCore@7bb72f2c url=https://127.0.0.1:31665/collection1 node=127.0.0.1:31665_ C559_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:31665_, base_url=https://127.0.0.1:31665}
[junit4] 2> 947074 T2978 C559 P31665 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:31665/collection1/
[junit4] 2> 947074 T2978 C559 P31665 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 947075 T2978 C559 P31665 oasc.SyncStrategy.syncToMe https://127.0.0.1:31665/collection1/ has no replicas
[junit4] 2> 947075 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:31665/collection1/ shard1
[junit4] 2> 947075 T2978 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 947077 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
[junit4] 2> 947078 T2973 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 947079 T2973 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2
[junit4] 2> 947083 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 947083 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
[junit4] 2> 947084 T2973 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 947088 T2973 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3
[junit4] 2> 947129 T2978 oasc.ZkController.register We are https://127.0.0.1:31665/collection1/ and leader is https://127.0.0.1:31665/collection1/
[junit4] 2> 947130 T2978 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:31665
[junit4] 2> 947130 T2978 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 947130 T2978 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 947130 T2978 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 947132 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 947134 T2978 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3
[junit4] 2> 947134 T2946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "state":"active",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "node_name":"127.0.0.1:31665_",
[junit4] 2> "numShards":"3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "base_url":"https://127.0.0.1:31665"}
[junit4] 2> 947138 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
[junit4] 2> 947138 T2973 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 947139 T2973 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 948329 T2921 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 948330 T2921 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 948335 T2921 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 948336 T2921 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31685
[junit4] 2> 948338 T2921 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 948339 T2921 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 948339 T2921 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004
[junit4] 2> 948339 T2921 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/'
[junit4] 2> 948369 T2921 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/solr.xml
[junit4] 2> 948423 T2921 oasc.CoreContainer.<init> New CoreContainer 1167823090
[junit4] 2> 948423 T2921 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/]
[junit4] 2> 948424 T2921 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 948424 T2921 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 948425 T2921 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 948425 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 948425 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
[junit4] 2> 948425 T2921 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 948425 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 948426 T2921 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 948426 T2921 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 948426 T2921 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 948428 T2921 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
[junit4] 2> 948430 T2921 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 948430 T2921 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 948430 T2921 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 948431 T2921 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:31640/solr
[junit4] 2> 948431 T2921 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 948431 T2921 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
[junit4] 2> 948432 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 948433 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3904473d name:ZooKeeperConnection Watcher:127.0.0.1:31640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 948434 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 948435 T2921 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
[junit4] 2> 948436 T2921 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 948437 T2991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78d50b00 name:ZooKeeperConnection Watcher:127.0.0.1:31640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 948437 T2921 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 948441 T2921 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 949445 T2921 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31685_
[junit4] 2> 949446 T2921 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31685_
[junit4] 2> 949448 T2921 oasc.Overseer.close Overseer (id=null) closing
[junit4] 2> 949456 T2993 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 949456 T2993 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 949457 T2950 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 949458 T2993 oasc.ZkController.preRegister Registering watch for external collection collection1
[junit4] 2> 949458 T2993 oascc.ZkStateReader.addZkWatch addZkWatch collection1
[junit4] 2> 949459 T2993 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper...
[junit4] 2> 949459 T2946 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "roles":null,
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "collection":"collection1",
[junit4] 2> "core":"collection1",
[junit4] 2> "node_name":"127.0.0.1:31685_",
[junit4] 2> "numShards":"3",
[junit4] 2> "shard":null,
[junit4] 2> "base_url":"https://127.0.0.1:31685"}
[junit4] 2> 949459 T2993 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4
[junit4] 2> 949460 T2993 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 949461 T2946 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 949462 T2946 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 949465 T2946 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
[junit4] 2> 949466 T2992 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 949466 T2973 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating...
[junit4] 2> 949467 T2992 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5
[junit4] 2> 949467 T2973 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5
[junit4] 2> 950460 T2993 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 950461 T2993 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 950461 T2993 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 950462 T2993 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 950462 T2993 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 950463 T2993 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/collection1/'
[junit4] 2> 950465 T2993 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 950466 T2993 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 950466 T2993 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 950530 T2993 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
[junit4] 2> 950576 T2993 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 950677 T2993 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 950686 T2993 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 951023 T2993 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 951026 T2993 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 951028 T2993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 951032 T2993 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 951049 T2993 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 951049 T2993 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 951049 T2993 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2/
[junit4] 2> 951050 T2993 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@484ff8fa
[junit4] 2> 951051 T2993 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2
[junit4] 2> 951052 T2993 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2/index/
[junit4] 2> 951053 T2993 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 951054 T2993 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2/index
[junit4] 2> 951055 T2993 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=29.9375, floorSegmentMB=1.666015625, forceMergeDeletesPctAllowed=2.5492460313621357, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31738124865380357
[junit4] 2> 951077 T2993 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-7CEAB97D489AD51F-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 951086 T2993 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 951090 T2993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 951091 T2993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 951091 T2993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 951091 T2993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 951092 T2993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 951092 T2993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 951092 T2993 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 951093 T2993 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 951093 T2993 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 951094 T2993 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 951094 T2993 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 951095 T2993 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
[junit4] 2> 951095 T2993 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 951096 T2993 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 951096 T2993 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 951096 T2993 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 951097 T2993 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 951097 T2993 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 951098 T2993 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 951098 T2993 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 951117 T2993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 951119 T2993 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4]
[...truncated too long message...]
kup.java:318)
[junit4] 2> 27076 T5020 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] 2> 30084 T5020 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] 2> at java.net.Socket.connect(Socket.java:579)
[junit4] 2> at java.net.Socket.connect(Socket.java:528)
[junit4] 2> at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] 2> at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] 2> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] 2> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] 2> at java.net.URL.openStream(URL.java:1037)
[junit4] 2> at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
[junit4] 2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=hr_HR, timezone=Europe/Oslo
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=2,free=150141464,total=308281344
[junit4] 2> NOTE: All tests run in this JVM: [TestOverriddenPrefixQueryForCustomFieldType, TestUniqueKeyFieldResource, ZkNodePropsTest, TestReplicationHandler, TestReplicaProperties, HdfsUnloadDistributedZkTest, XsltUpdateRequestHandlerTest, TestSolrQueryParserResource, TestDFRSimilarityFactory, TestMultiCoreConfBootstrap, RemoteQueryErrorTest, DocValuesTest, DateMathParserTest, TestManagedResourceStorage, TestDistributedMissingSort, TestQuerySenderListener, TestCloudInspectUtil, TestSolr4Spatial, TestUpdate, TestSchemaManager, BJQParserTest, TestStressLucene, TestSolrXmlPersistence, TestOmitPositions, TestReversedWildcardFilterFactory, CollectionsAPIAsyncDistributedZkTest, AliasIntegrationTest, DistributedDebugComponentTest, TestHashPartitioner, ModifyConfFileTest, StatsComponentTest, UpdateRequestProcessorFactoryTest, RAMDirectoryFactoryTest, NumericFieldsTest, TestMissingGroups, PolyFieldTest, AnalysisAfterCoreReloadTest, ShardSplitTest, BinaryUpdateRequestHandlerTest, CursorPagingTest, OverseerCollectionProcessorTest, LeaderElectionIntegrationTest, TestSearcherReuse, BasicDistributedZk2Test, MultiTermTest, XmlUpdateRequestHandlerTest, TestManagedSchema, TestRemoteStreaming, TestLuceneMatchVersion, TestStressRecovery, OverseerStatusTest, BasicFunctionalityTest, TestIndexingPerformance, SimpleFacetsTest, SolrCoreTest, TestSort, TestLazyCores, HighlighterTest, ShowFileRequestHandlerTest, CoreAdminHandlerTest, SolrCoreCheckLockOnStartupTest, QueryEqualityTest, DocValuesMultiTest, TestSurroundQueryParser, TestQueryUtils, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, DocumentBuilderTest, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaTest, QueryParsingTest, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, SOLR749Test, CopyFieldTest, BadComponentTest, TestElisionMultitermQuery, NotRequiredUniqueKeyTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerRolesTest, SharedFSAutoReplicaFailoverTest, SolrXmlInZkTest, TestCloudPivotFacet, TestShortCircuitedRequests, QueryResultKeyTest, TestInfoStreamLogging, TestNonNRTOpen, TestShardHandlerFactory, TestReplicationHandlerBackup]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestReplicationHandlerBackup -Dtests.seed=7CEAB97D489AD51F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=hr_HR -Dtests.timezone=Europe/Oslo -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J2 | TestReplicationHandlerBackup (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandlerBackup:
[junit4] > 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
[junit4] > at __randomizedtesting.SeedInfo.seed([7CEAB97D489AD51F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=5041, name=Thread-2046, state=RUNNABLE, group=TGRP-TestReplicationHandlerBackup]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at java.net.Socket.connect(Socket.java:528)
[junit4] > at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
[junit4] > at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
[junit4] > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
[junit4] > at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
[junit4] > at java.net.URL.openStream(URL.java:1037)
[junit4] > at org.apache.solr.handler.TestReplicationHandlerBackup$BackupThread.run(TestReplicationHandlerBackup.java:318)
[junit4] > at __randomizedtesting.SeedInfo.seed([7CEAB97D489AD51F]:0)
[junit4] Completed on J2 in 30.12s, 1 test, 2 errors <<< FAILURES!
[...truncated 221 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 437 suites, 1784 tests, 2 suite-level errors, 1 error, 1 failure, 238 ignored (9 assumptions)
Total time: 208 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #651
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 633418189 bytes
Compression is 0.0%
Took 1 min 48 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure