You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/11 18:59:58 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build #
10787 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10787/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([139B43890EE76870]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Captured an uncaught exception in thread: Thread[id=751, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=751, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Caused by: java.lang.AssertionError: liveDocs.count()=4 info.docCount=9 info.getDelCount()=6
at __randomizedtesting.SeedInfo.seed([139B43890EE76870]:0)
at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:115)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:133)
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:185)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:95)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:393)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:204)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1489)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1690)
at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([139B43890EE76870]:0)
Build Log:
[...truncated 11889 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/init-core-data-001
[junit4] 2> 649 T15 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 670 T15 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 681 T15 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 686 T15 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 691 T16 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 789 T15 oasc.ZkTestServer.run start zk server on port:39566
[junit4] 2> 843 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 884 T17 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
[junit4] 2> 2810 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed6dd9f name:ZooKeeperConnection Watcher:127.0.0.1:39566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2811 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2812 T15 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 2835 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2837 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c628a88 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2837 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 2842 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 2847 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 2853 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 2856 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 2859 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2871 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 2877 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2878 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 2883 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2884 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2894 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2895 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 2907 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2909 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 2913 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2914 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 2917 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2917 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 2920 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2921 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 2923 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2924 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2928 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2928 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 2932 T15 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2933 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 2942 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 2944 T26 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2146a305 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 2945 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 3427 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 3778 T15 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 3803 T15 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34322
[junit4] 2> 4065 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 4091 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 4092 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002
[junit4] 2> 4092 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/'
[junit4] 2> 4238 T15 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/solr.xml
[junit4] 2> 4457 T15 oasc.CoreContainer.<init> New CoreContainer 1743143706
[junit4] 2> 4460 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/]
[junit4] 2> 4500 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 4500 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 4505 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 4506 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 4506 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 4507 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 4508 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 4508 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 4509 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 4554 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 4556 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 4558 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 4559 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39566/solr
[junit4] 2> 4571 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 4574 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 4583 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6130ec05 name:ZooKeeperConnection Watcher:127.0.0.1:39566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 4584 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 4592 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 4594 T40 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66e507e1 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 4595 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 4604 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 4618 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 4628 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 4632 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 4639 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 4658 T15 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 4661 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34322_
[junit4] 2> 4665 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34322_
[junit4] 2> 4670 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 4673 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 4683 T15 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34322_
[junit4] 2> 4684 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 4689 T15 oasc.Overseer.start Overseer (id=92084026733166596-127.0.0.1:34322_-n_0000000000) starting
[junit4] 2> 4700 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 4732 T42 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 4733 T15 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 4742 T15 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 4746 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 4757 T41 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 4836 T43 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 4836 T43 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 4839 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 4839 T43 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 4842 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 4843 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34322",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34322_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 4844 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 4851 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 4855 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 4859 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 5840 T43 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 5841 T43 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 5842 T43 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 5843 T43 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 5845 T43 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 5846 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/collection1/'
[junit4] 2> 5848 T43 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 5849 T43 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 5851 T43 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 6016 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 6171 T43 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 6185 T43 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 6299 T43 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 6740 T43 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 6741 T43 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 6746 T43 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 6752 T43 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 6939 T43 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 6962 T43 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 6976 T43 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/
[junit4] 2> 6979 T43 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c33d55
[junit4] 2> 7022 T43 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data
[junit4] 2> 7027 T43 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/index/
[junit4] 2> 7028 T43 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 7040 T43 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/index
[junit4] 2> 7043 T43 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 7116 T43 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 7118 T43 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 7391 T43 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 7454 T43 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 7464 T43 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 7465 T43 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 7466 T43 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 7467 T43 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 7518 T43 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 7519 T43 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 7519 T43 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 7541 T43 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 7550 T43 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 7551 T43 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 7553 T43 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 7570 T43 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 7572 T43 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 7597 T43 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 7608 T43 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 7651 T43 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 7702 T43 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 7703 T43 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 7706 T43 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 7723 T43 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 7724 T43 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 7754 T43 oass.SolrIndexSearcher.<init> Opening Searcher@5d486a82[collection1] main
[junit4] 2> 7771 T43 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 7774 T43 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 7775 T43 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 7781 T43 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 7781 T43 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 7812 T43 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 7816 T43 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 7818 T43 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 7820 T43 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 7824 T43 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 7825 T43 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 7825 T43 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 7874 T44 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d486a82[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 7882 T43 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 7886 T47 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:34322 collection:control_collection shard:shard1
[junit4] 2> 7887 T15 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 7887 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 7897 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 7911 T47 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 7915 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 7921 T47 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 7921 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 7922 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 7923 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> ASYNC NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@4cbac94d url=https://127.0.0.1:34322/collection1 node=127.0.0.1:34322_ C0_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:34322, core=collection1, node_name=127.0.0.1:34322_}
[junit4] 2> 7926 T47 C0 P34322 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34322/collection1/
[junit4] 2> 7927 T49 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75cedf2f name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 7929 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 7929 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 7929 T47 C0 P34322 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 7931 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 7931 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 7932 T47 C0 P34322 oasc.SyncStrategy.syncToMe https://127.0.0.1:34322/collection1/ has no replicas
[junit4] 2> 7933 T47 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34322/collection1/ shard1
[junit4] 2> 7936 T47 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 7938 T15 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 7944 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 7946 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 7949 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 8052 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8052 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8096 T47 oasc.ZkController.register We are https://127.0.0.1:34322/collection1/ and leader is https://127.0.0.1:34322/collection1/
[junit4] 2> 8097 T47 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:34322
[junit4] 2> 8097 T47 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 8098 T47 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 8098 T47 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 8107 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 8108 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 8108 T47 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 8108 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 8111 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 8112 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:34322",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34322_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 8115 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 8217 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8217 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 8286 T15 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 8287 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 8292 T15 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 8295 T15 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34301
[junit4] 2> 8297 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 8298 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 8298 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003
[junit4] 2> 8299 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/'
[junit4] 2> 8318 T15 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/solr.xml
[junit4] 2> 8335 T15 oasc.CoreContainer.<init> New CoreContainer 654077730
[junit4] 2> 8336 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/]
[junit4] 2> 8337 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 8337 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 8338 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 8338 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 8338 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 8339 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 8339 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 8339 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 8340 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 8342 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 8343 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 8343 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 8343 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39566/solr
[junit4] 2> 8344 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 8345 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 8347 T60 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@513e5432 name:ZooKeeperConnection Watcher:127.0.0.1:39566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 8348 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 8350 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 8353 T62 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@679f00f4 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 8353 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 8360 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 9364 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34301_
[junit4] 2> 9366 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34301_
[junit4] 2> 9387 T63 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 9388 T63 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 9390 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9390 T63 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 9391 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9392 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9395 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 9397 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34301",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34301_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 9397 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 9398 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 9403 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 9507 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9507 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 9507 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 10391 T63 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 10392 T63 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 10393 T63 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 10393 T63 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 10395 T63 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 10395 T63 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/collection1/'
[junit4] 2> 10396 T63 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 10397 T63 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 10397 T63 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 10437 T63 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 10466 T63 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 10469 T63 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 10509 T63 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 10762 T63 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 10763 T63 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 10765 T63 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 10770 T63 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 10817 T63 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 10818 T63 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 10818 T63 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/
[junit4] 2> 10818 T63 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c33d55
[junit4] 2> 10820 T63 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1
[junit4] 2> 10821 T63 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/index/
[junit4] 2> 10821 T63 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 10822 T63 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/index
[junit4] 2> 10822 T63 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 11221 T63 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 11222 T63 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 11232 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 11233 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 11233 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 11234 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 11234 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 11235 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 11236 T63 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 11237 T63 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 11237 T63 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 11238 T63 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 11244 T63 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 11245 T63 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 11245 T63 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 11247 T63 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 11248 T63 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 11249 T63 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 11251 T63 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 11257 T63 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 11262 T63 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 11262 T63 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 11264 T63 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 11266 T63 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 11267 T63 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 11268 T63 oass.SolrIndexSearcher.<init> Opening Searcher@cb844ba[collection1] main
[junit4] 2> 11268 T63 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 11269 T63 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 11270 T63 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 11271 T63 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 11271 T63 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 11272 T63 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 11274 T63 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 11274 T63 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 11275 T63 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 11276 T63 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 11289 T64 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cb844ba[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 11296 T63 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 11299 T67 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:34301 collection:collection1 shard:shard2
[junit4] 2> 11299 T15 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 11304 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 11304 T67 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 11319 T67 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 11322 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11323 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11323 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11323 T67 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 11324 T67 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 11326 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@44178458 url=https://127.0.0.1:34301/collection1 node=127.0.0.1:34301_ C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:34301, core=collection1, node_name=127.0.0.1:34301_}
[junit4] 2> 11324 T67 C1 P34301 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34301/collection1/
[junit4] 2> 11330 T67 C1 P34301 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 11331 T67 C1 P34301 oasc.SyncStrategy.syncToMe https://127.0.0.1:34301/collection1/ has no replicas
[junit4] 2> 11331 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11332 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11332 T67 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34301/collection1/ shard2
[junit4] 2> 11332 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11332 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11341 T67 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 11356 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11357 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11360 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11472 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11472 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11472 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11507 T67 oasc.ZkController.register We are https://127.0.0.1:34301/collection1/ and leader is https://127.0.0.1:34301/collection1/
[junit4] 2> 11508 T67 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:34301
[junit4] 2> 11508 T67 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 11509 T67 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 11509 T67 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 11511 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11512 T67 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11512 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11512 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11514 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 11515 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:34301",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34301_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 11521 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 11626 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11626 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11627 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 11693 T15 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 11698 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 11715 T15 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 11725 T15 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36487
[junit4] 2> 11731 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 11732 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 11732 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004
[junit4] 2> 11733 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/'
[junit4] 2> 11751 T15 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/solr.xml
[junit4] 2> 11781 T15 oasc.CoreContainer.<init> New CoreContainer 2001795855
[junit4] 2> 11782 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/]
[junit4] 2> 11783 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 11784 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 11784 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 11785 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 11785 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 11786 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 11786 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 11786 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 11787 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 11794 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 11795 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 11796 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 11796 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39566/solr
[junit4] 2> 11797 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 11807 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11810 T78 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@159a8c24 name:ZooKeeperConnection Watcher:127.0.0.1:39566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11811 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11827 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 11829 T80 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bfef57 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 11831 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 11841 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 12845 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36487_
[junit4] 2> 12847 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36487_
[junit4] 2> 12863 T81 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 12864 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 12866 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12866 T81 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 12866 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12867 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12869 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 12872 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:36487",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36487_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 12872 T41 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 12873 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 12877 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 12981 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 12981 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 12981 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 12981 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 13867 T81 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 13867 T81 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 13868 T81 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 13868 T81 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 13869 T81 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 13870 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/collection1/'
[junit4] 2> 13870 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 13871 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 13871 T81 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 13904 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 13920 T81 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 13922 T81 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 13944 T81 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 14086 T81 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 14088 T81 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 14090 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 14094 T81 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 14152 T81 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 14153 T81 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 14153 T81 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/
[junit4] 2> 14154 T81 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c33d55
[junit4] 2> 14155 T81 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2
[junit4] 2> 14156 T81 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/index/
[junit4] 2> 14156 T81 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 14157 T81 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/index
[junit4] 2> 14158 T81 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=34, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 14201 T81 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 14201 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 14211 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 14211 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 14211 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 14212 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 14212 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 14212 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 14213 T81 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 14213 T81 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 14213 T81 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 14214 T81 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 14215 T81 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 14216 T81 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 14216 T81 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 14217 T81 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 14218 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 14218 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 14219 T81 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 14221 T81 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 14223 T81 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 14226 T81 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 14228 T81 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 14232 T81 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 14233 T81 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 14234 T81 oass.SolrIndexSearcher.<init> Opening Searcher@6e55e68e[collection1] main
[junit4] 2> 14234 T81 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 14236 T81 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 14236 T81 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 14237 T81 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 14237 T81 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 14237 T81 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 14239 T81 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 14239 T81 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 14239 T81 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 14240 T81 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 14248 T82 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e55e68e[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 14258 T81 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 14260 T85 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36487 collection:collection1 shard:shard3
[junit4] 2> 14260 T15 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 14261 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 14263 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 14270 T85 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 14273 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14273 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14273 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14273 T85 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 14275 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 14276 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@3109617c url=https://127.0.0.1:36487/collection1 node=127.0.0.1:36487_ C2_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:36487, core=collection1, node_name=127.0.0.1:36487_}
[junit4] 2> 14276 T85 C2 P36487 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36487/collection1/
[junit4] 2> 14278 T85 C2 P36487 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 14279 T85 C2 P36487 oasc.SyncStrategy.syncToMe https://127.0.0.1:36487/collection1/ has no replicas
[junit4] 2> 14279 T85 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36487/collection1/ shard3
[junit4] 2> 14279 T85 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 14284 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14291 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14294 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14400 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14400 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14400 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14400 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14442 T85 oasc.ZkController.register We are https://127.0.0.1:36487/collection1/ and leader is https://127.0.0.1:36487/collection1/
[junit4] 2> 14443 T85 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36487
[junit4] 2> 14443 T85 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 14443 T85 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 14443 T85 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 14446 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14446 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14446 T85 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 14446 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14448 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 14449 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:36487",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36487_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 14452 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 14551 T15 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 14552 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 14554 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14554 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14554 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14554 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 14557 T15 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 14560 T15 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42268
[junit4] 2> 14563 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 14563 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 14564 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005
[junit4] 2> 14564 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/'
[junit4] 2> 14586 T15 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/solr.xml
[junit4] 2> 14599 T15 oasc.CoreContainer.<init> New CoreContainer 1604537321
[junit4] 2> 14600 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/]
[junit4] 2> 14601 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 14601 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 14602 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 14602 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 14603 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 14603 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 14603 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 14604 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 14604 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 14609 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 14609 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 14610 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 14610 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39566/solr
[junit4] 2> 14611 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 14612 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 14615 T96 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7004d206 name:ZooKeeperConnection Watcher:127.0.0.1:39566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 14615 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 14618 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 14620 T98 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e539862 name:ZooKeeperConnection Watcher:127.0.0.1:39566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 14620 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 14626 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 15629 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42268_
[junit4] 2> 15631 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42268_
[junit4] 2> 15649 T99 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 15650 T99 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 15652 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15652 T99 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 15652 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15653 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15657 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 15659 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:42268",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42268_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 15660 T41 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 15660 T41 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 15666 T40 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 15770 T62 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 15770 T98 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 15770 T80 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 15770 T40 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 15770 T49 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 16653 T99 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 16653 T99 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 16654 T99 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 16654 T99 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 16655 T99 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 16655 T99 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/collection1/'
[junit4] 2> 16656 T99 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/collection1/lib/.svn/' to classloader
[junit4] 2> 16656 T99 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-139B43890EE76870-001/tempDir-005/collection1/lib/classes/' to classloader
[junit4] 2> 16656 T99 oasc.SolrResourceLoader.replaceCla
[...truncated too long message...]
org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 55) Thread[id=1031, name=searcherExecutor-410-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 56) Thread[id=1034, name=coreZkRegister-404-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 57) Thread[id=1017, name=qtp1254350210-1017 Acceptor0 SslSelectChannelConnector@127.0.0.1:52939, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 58) Thread[id=1041, name=qtp23793548-1041 Acceptor0 SslSelectChannelConnector@127.0.0.1:40205, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 59) Thread[id=1092, name=Thread-156-SendThread(127.0.0.1:39566), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2> 60) Thread[id=86, name=HashSessionScavenger-3, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 61) Thread[id=965, name=searcherExecutor-381-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 62) Thread[id=1081, name=HashSessionScavenger-45, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> NOTE: test params are: codec=Lucene49: {rnd_b=PostingsFormat(name=Memory doPackFST= true), _version_=Pulsing41(freqCutoff=18 minBlockSize=55 maxBlockSize=197), a_t=Pulsing41(freqCutoff=18 minBlockSize=55 maxBlockSize=197), a_i=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=DefaultSimilarity, locale=mt, timezone=America/Dawson
[junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=76,free=93061248,total=213385216
[junit4] 2> NOTE: All tests run in this JVM: [UUIDFieldTest, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=139B43890EE76870 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt -Dtests.timezone=America/Dawson -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([139B43890EE76870]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=751, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] > Caused by: java.lang.AssertionError: liveDocs.count()=4 info.docCount=9 info.getDelCount()=6
[junit4] > at __randomizedtesting.SeedInfo.seed([139B43890EE76870]:0)
[junit4] > at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
[junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:115)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:133)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:185)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:95)
[junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:393)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:269)
[junit4] > at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:204)
[junit4] > at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1489)
[junit4] > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1690)
[junit4] > at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
[junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
[junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
[junit4] Completed on J1 in 7225.02s, 1 test, 3 errors <<< FAILURES!
[...truncated 312 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 418 suites, 1710 tests, 2 suite-level errors, 1 error, 403 ignored (16 assumptions)
Total time: 156 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60) - Build #
10788 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10788/
Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseSerialGC
All tests passed
Build Log:
[...truncated 59435 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:406: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:87: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:179: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/strange-overgeneration.aff
* ./lucene/analysis/common/src/test/org/apache/lucene/analysis/hunspell/strange-overgeneration.dic
Total time: 82 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any