You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/06/17 15:21:32 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_60) - Build # 10465
- Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10465/
Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([620D1A4C5F907DEC]:0)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([620D1A4C5F907DEC]:0)
Build Log:
[...truncated 12169 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/init-core-data-001
[junit4] 2> 971245 T3750 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 971246 T3750 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 971251 T3750 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 971252 T3750 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 971252 T3751 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 971352 T3750 oasc.ZkTestServer.run start zk server on port:44001
[junit4] 2> 971353 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 972822 T3754 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1467ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 972823 T3757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36d960 name:ZooKeeperConnection Watcher:127.0.0.1:44001 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 972824 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 972824 T3750 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 972829 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 972830 T3759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b661f9 name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 972830 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 972830 T3750 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 972832 T3750 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 972834 T3750 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 972835 T3750 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 972836 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 972837 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 972839 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 972840 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 972842 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 972843 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 972844 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 972845 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 972846 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 972847 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 972848 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 972848 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 972850 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 972850 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 972856 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 972856 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 972858 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 972858 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 972860 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 972860 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 972862 T3750 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 972862 T3750 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 972865 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 972866 T3761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ca6cb name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 972866 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 973125 T3750 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 973127 T3750 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 973130 T3750 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50200
[junit4] 2> 973132 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 973132 T3750 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 973133 T3750 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002
[junit4] 2> 973133 T3750 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/'
[junit4] 2> 973148 T3750 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/solr.xml
[junit4] 2> 973179 T3750 oasc.CoreContainer.<init> New CoreContainer 5827696
[junit4] 2> 973180 T3750 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/]
[junit4] 2> 973180 T3750 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 973181 T3750 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 973181 T3750 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 973182 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 973182 T3750 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 973182 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 973183 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 973183 T3750 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 973183 T3750 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 973185 T3750 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 973186 T3750 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 973186 T3750 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 973187 T3750 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44001/solr
[junit4] 2> 973187 T3750 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 973188 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 973189 T3772 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba1d0f name:ZooKeeperConnection Watcher:127.0.0.1:44001 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 973189 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 973192 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 973192 T3774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e791ec name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 973193 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 973194 T3750 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 973197 T3750 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 973201 T3750 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 973204 T3750 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 973206 T3750 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 973209 T3750 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 973210 T3750 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50200_
[junit4] 2> 973210 T3750 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50200_
[junit4] 2> 973214 T3750 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 973218 T3750 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 973227 T3750 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50200_
[junit4] 2> 973228 T3750 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 973231 T3750 oasc.Overseer.start Overseer (id=91947273902686212-127.0.0.1:50200_-n_0000000000) starting
[junit4] 2> 973236 T3750 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 973246 T3776 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 973247 T3750 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 973248 T3750 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 973250 T3750 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 973253 T3775 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 973258 T3777 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 973259 T3777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 973261 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 973261 T3777 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 973262 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 973263 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:50200",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50200_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 973263 T3775 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 973263 T3775 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 973265 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 973266 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 974262 T3777 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 974262 T3777 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 974263 T3777 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 974263 T3777 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 974264 T3777 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 974264 T3777 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/collection1/'
[junit4] 2> 974265 T3777 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 974266 T3777 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 974266 T3777 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 974318 T3777 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 974363 T3777 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 974364 T3777 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 974378 T3777 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 974731 T3777 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 974733 T3777 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 974735 T3777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 974738 T3777 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 974754 T3777 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 974754 T3777 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 974755 T3777 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/
[junit4] 2> 974755 T3777 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ba3c3
[junit4] 2> 974755 T3777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data
[junit4] 2> 974756 T3777 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/index/
[junit4] 2> 974756 T3777 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 974757 T3777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/index
[junit4] 2> 974757 T3777 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=24.404296875, floorSegmentMB=0.7412109375, forceMergeDeletesPctAllowed=10.323174166504653, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15830744598373175
[junit4] 2> 974761 T3777 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 974762 T3777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 974764 T3777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 974765 T3777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 974765 T3777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 974765 T3777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 974765 T3777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 974766 T3777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 974766 T3777 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 974766 T3777 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 974767 T3777 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 974767 T3777 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 974768 T3777 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 974768 T3777 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 974768 T3777 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 974768 T3777 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 974769 T3777 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 974769 T3777 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 974772 T3777 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 974774 T3777 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 974774 T3777 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 974775 T3777 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 974776 T3777 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 974776 T3777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 974776 T3777 oass.SolrIndexSearcher.<init> Opening Searcher@9473e7[collection1] main
[junit4] 2> 974776 T3777 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 974777 T3777 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 974777 T3777 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 974778 T3777 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 974778 T3777 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 974778 T3777 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 974779 T3777 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 974779 T3777 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 974779 T3777 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 974780 T3777 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 974782 T3778 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9473e7[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 974783 T3777 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 974784 T3781 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50200 collection:control_collection shard:shard1
[junit4] 2> 974784 T3750 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 974785 T3781 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 974785 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 974788 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 974789 T3783 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1565ca name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 974789 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 974790 T3781 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 974793 T3750 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 974794 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974795 T3781 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 974795 T3781 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C960 name=collection1 org.apache.solr.core.SolrCore@161c469 url=https://127.0.0.1:50200/collection1 node=127.0.0.1:50200_ C960_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50200, core=collection1, node_name=127.0.0.1:50200_}
[junit4] 2> 974795 T3781 C960 P50200 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50200/collection1/
[junit4] 2> 974795 T3781 C960 P50200 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 974796 T3781 C960 P50200 oasc.SyncStrategy.syncToMe https://127.0.0.1:50200/collection1/ has no replicas
[junit4] 2> 974796 T3781 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50200/collection1/ shard1
[junit4] 2> 974796 T3781 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 974797 T3750 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 974797 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 974800 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974801 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 974801 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 974804 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 974806 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974907 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 974907 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 974952 T3781 oasc.ZkController.register We are https://127.0.0.1:50200/collection1/ and leader is https://127.0.0.1:50200/collection1/
[junit4] 2> 974952 T3781 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50200
[junit4] 2> 974953 T3781 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 974953 T3781 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 974953 T3781 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 974954 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974955 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974955 T3781 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 974955 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 974956 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 974957 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:50200",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50200_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 974959 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 975061 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 975061 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 975072 T3750 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 975073 T3750 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 975075 T3750 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 975077 T3750 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:47996
[junit4] 2> 975078 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 975079 T3750 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 975079 T3750 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003
[junit4] 2> 975079 T3750 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/'
[junit4] 2> 975094 T3750 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/solr.xml
[junit4] 2> 975124 T3750 oasc.CoreContainer.<init> New CoreContainer 28477291
[junit4] 2> 975124 T3750 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/]
[junit4] 2> 975125 T3750 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 975126 T3750 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 975126 T3750 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 975127 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 975127 T3750 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 975127 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 975128 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 975128 T3750 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 975128 T3750 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 975130 T3750 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 975131 T3750 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 975131 T3750 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 975132 T3750 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44001/solr
[junit4] 2> 975132 T3750 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 975133 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 975135 T3794 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d46910 name:ZooKeeperConnection Watcher:127.0.0.1:44001 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 975135 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 975137 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 975139 T3796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14d02c name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 975139 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 975144 T3750 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 976147 T3750 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47996_
[junit4] 2> 976149 T3750 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47996_
[junit4] 2> 976153 T3783 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 976153 T3796 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 976153 T3774 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 976163 T3797 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 976164 T3797 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 976203 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 976203 T3797 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 976204 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 976204 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 976206 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 976207 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:47996",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47996_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 976208 T3775 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 976208 T3775 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 976212 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 976314 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 976315 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 976314 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977204 T3797 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 977205 T3797 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 977205 T3797 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 977206 T3797 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 977207 T3797 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 977207 T3797 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/collection1/'
[junit4] 2> 977208 T3797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 977208 T3797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 977209 T3797 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 977248 T3797 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 977278 T3797 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 977280 T3797 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 977288 T3797 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 977569 T3797 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 977571 T3797 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 977572 T3797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 977576 T3797 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 977593 T3797 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 977594 T3797 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 977594 T3797 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/
[junit4] 2> 977594 T3797 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ba3c3
[junit4] 2> 977595 T3797 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1
[junit4] 2> 977595 T3797 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/index/
[junit4] 2> 977595 T3797 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 977596 T3797 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/index
[junit4] 2> 977596 T3797 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=24.404296875, floorSegmentMB=0.7412109375, forceMergeDeletesPctAllowed=10.323174166504653, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15830744598373175
[junit4] 2> 977608 T3797 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 977608 T3797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 977612 T3797 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 977612 T3797 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 977612 T3797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 977612 T3797 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 977613 T3797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 977613 T3797 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 977613 T3797 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 977614 T3797 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 977614 T3797 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 977614 T3797 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 977615 T3797 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 977615 T3797 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 977616 T3797 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 977616 T3797 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 977616 T3797 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 977617 T3797 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 977620 T3797 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 977622 T3797 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 977622 T3797 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 977623 T3797 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 977624 T3797 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 977624 T3797 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 977624 T3797 oass.SolrIndexSearcher.<init> Opening Searcher@11bda6[collection1] main
[junit4] 2> 977624 T3797 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 977625 T3797 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 977626 T3797 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 977626 T3797 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 977626 T3797 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 977627 T3797 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 977627 T3797 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 977627 T3797 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 977628 T3797 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 977628 T3797 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 977632 T3798 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11bda6[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 977634 T3797 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 977635 T3801 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:47996 collection:collection1 shard:shard3
[junit4] 2> 977635 T3750 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 977636 T3801 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 977636 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 977640 T3801 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 977642 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977642 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977643 T3801 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 977643 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977643 T3801 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 977643 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C961 name=collection1 org.apache.solr.core.SolrCore@5e0366 url=https://127.0.0.1:47996/collection1 node=127.0.0.1:47996_ C961_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:47996, core=collection1, node_name=127.0.0.1:47996_}
[junit4] 2> 977644 T3801 C961 P47996 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:47996/collection1/
[junit4] 2> 977644 T3801 C961 P47996 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 977645 T3801 C961 P47996 oasc.SyncStrategy.syncToMe https://127.0.0.1:47996/collection1/ has no replicas
[junit4] 2> 977645 T3801 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:47996/collection1/ shard3
[junit4] 2> 977645 T3801 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 977646 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977649 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977656 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977757 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977757 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977758 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977800 T3801 oasc.ZkController.register We are https://127.0.0.1:47996/collection1/ and leader is https://127.0.0.1:47996/collection1/
[junit4] 2> 977801 T3801 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:47996
[junit4] 2> 977801 T3801 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 977801 T3801 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 977802 T3801 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 977803 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977804 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977804 T3801 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 977804 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977805 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 977806 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:47996",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47996_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 977808 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 977909 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977909 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977909 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 977913 T3750 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 977914 T3750 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 977916 T3750 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 977917 T3750 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:41367
[junit4] 2> 977919 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 977920 T3750 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 977920 T3750 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004
[junit4] 2> 977920 T3750 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/'
[junit4] 2> 977935 T3750 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/solr.xml
[junit4] 2> 977965 T3750 oasc.CoreContainer.<init> New CoreContainer 12412883
[junit4] 2> 977966 T3750 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/]
[junit4] 2> 977967 T3750 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 977967 T3750 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 977967 T3750 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 977968 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 977968 T3750 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 977969 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 977969 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 977969 T3750 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 977970 T3750 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 977972 T3750 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 977972 T3750 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 977972 T3750 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 977973 T3750 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44001/solr
[junit4] 2> 977973 T3750 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 977974 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 977975 T3812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11d12e9 name:ZooKeeperConnection Watcher:127.0.0.1:44001 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 977976 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 977978 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 977979 T3814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c1d85a name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 977979 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 977984 T3750 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 978986 T3750 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41367_
[junit4] 2> 978993 T3750 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41367_
[junit4] 2> 978998 T3796 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 978998 T3783 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 978999 T3814 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 978999 T3774 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 979005 T3815 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 979005 T3815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 979006 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 979006 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 979006 T3815 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 979006 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 979007 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 979008 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:41367",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41367_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 979008 T3775 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 979008 T3775 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 979010 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 979112 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 979112 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 979112 T3814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 979112 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980007 T3815 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 980007 T3815 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 980008 T3815 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 980008 T3815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 980009 T3815 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 980009 T3815 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/collection1/'
[junit4] 2> 980010 T3815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 980010 T3815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 980010 T3815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 980062 T3815 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 980095 T3815 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 980097 T3815 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 980104 T3815 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 980386 T3815 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 980389 T3815 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 980390 T3815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 980393 T3815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 980409 T3815 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 980409 T3815 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 980410 T3815 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/
[junit4] 2> 980410 T3815 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ba3c3
[junit4] 2> 980411 T3815 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2
[junit4] 2> 980411 T3815 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/index/
[junit4] 2> 980411 T3815 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 980412 T3815 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/index
[junit4] 2> 980412 T3815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=24.404296875, floorSegmentMB=0.7412109375, forceMergeDeletesPctAllowed=10.323174166504653, segmentsPerTier=44.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.15830744598373175
[junit4] 2> 980418 T3815 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 980419 T3815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 980422 T3815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 980422 T3815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 980422 T3815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 980423 T3815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 980423 T3815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 980423 T3815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 980423 T3815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 980424 T3815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 980424 T3815 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 980425 T3815 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 980425 T3815 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 980425 T3815 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 980426 T3815 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 980426 T3815 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 980427 T3815 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 980427 T3815 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 980430 T3815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 980432 T3815 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 980432 T3815 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 980433 T3815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 980434 T3815 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 980434 T3815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 980434 T3815 oass.SolrIndexSearcher.<init> Opening Searcher@129126b[collection1] main
[junit4] 2> 980435 T3815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 980436 T3815 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 980436 T3815 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 980436 T3815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 980436 T3815 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 980437 T3815 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 980437 T3815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 980437 T3815 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 980438 T3815 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 980438 T3815 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 980442 T3816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@129126b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 980444 T3815 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 980445 T3819 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:41367 collection:collection1 shard:shard2
[junit4] 2> 980445 T3750 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 980446 T3819 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 980446 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 980450 T3819 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 980453 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980453 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980453 T3819 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 980454 T3819 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 980453 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C962 name=collection1 org.apache.solr.core.SolrCore@19de3b6 url=https://127.0.0.1:41367/collection1 node=127.0.0.1:41367_ C962_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:41367, core=collection1, node_name=127.0.0.1:41367_}
[junit4] 2> 980454 T3819 C962 P41367 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:41367/collection1/
[junit4] 2> 980455 T3819 C962 P41367 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 980455 T3819 C962 P41367 oasc.SyncStrategy.syncToMe https://127.0.0.1:41367/collection1/ has no replicas
[junit4] 2> 980455 T3819 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:41367/collection1/ shard2
[junit4] 2> 980455 T3819 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 980455 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 980459 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980461 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980463 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980565 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980565 T3814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980565 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980565 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980612 T3819 oasc.ZkController.register We are https://127.0.0.1:41367/collection1/ and leader is https://127.0.0.1:41367/collection1/
[junit4] 2> 980612 T3819 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:41367
[junit4] 2> 980612 T3819 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 980612 T3819 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 980613 T3819 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 980614 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980614 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980615 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980614 T3819 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 980616 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 980617 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:41367",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41367_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 980618 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 980725 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980726 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980725 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980725 T3814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 980730 T3750 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 980731 T3750 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 980734 T3750 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 980735 T3750 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:48406
[junit4] 2> 980738 T3750 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 980738 T3750 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 980739 T3750 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005
[junit4] 2> 980740 T3750 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/'
[junit4] 2> 980758 T3750 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/solr.xml
[junit4] 2> 980791 T3750 oasc.CoreContainer.<init> New CoreContainer 10917645
[junit4] 2> 980792 T3750 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/]
[junit4] 2> 980793 T3750 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 980793 T3750 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 980793 T3750 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 980794 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 980794 T3750 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 980794 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 980795 T3750 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 980795 T3750 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 980795 T3750 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 980797 T3750 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 980798 T3750 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 980798 T3750 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 980799 T3750 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44001/solr
[junit4] 2> 980799 T3750 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 980800 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 980801 T3830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14f47b4 name:ZooKeeperConnection Watcher:127.0.0.1:44001 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 980802 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 980806 T3750 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 980807 T3832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21eace name:ZooKeeperConnection Watcher:127.0.0.1:44001/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 980807 T3750 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 980814 T3750 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 981817 T3750 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48406_
[junit4] 2> 981819 T3750 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48406_
[junit4] 2> 981824 T3796 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 981824 T3774 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 981824 T3783 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 981824 T3814 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 981824 T3832 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 981834 T3833 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 981835 T3833 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 981836 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 981836 T3833 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 981837 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 981837 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 981840 T3775 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 981841 T3775 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:48406",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:48406_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 981842 T3775 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 981842 T3775 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 981845 T3774 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 981949 T3814 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 981949 T3783 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 981949 T3774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 981949 T3832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 981949 T3796 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 982837 T3833 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 982838 T3833 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 982839 T3833 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 982840 T3833 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 982842 T3833 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 982842 T3833 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/collection1/'
[junit4] 2> 982843 T3833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/collection1/lib/.svn/' to classloader
[junit4] 2> 982844 T3833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/collection1/lib/classes/' to classloader
[junit4] 2> 982845 T3833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/collection1/lib/README' to classloader
[junit4] 2> 982903 T3833 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 982933 T3833 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 982935 T3833 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 982951 T3833 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 983225 T3833 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 983227 T3833 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 983228 T3833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 983232 T3833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 983257 T3833 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 983258 T3833 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 983258 T3833 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-005/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty3/
[junit4] 2> 983259 T3833 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2ba3c3
[junit4] 2> 983260 T3833 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty3
[junit4] 2> 983261 T3833 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty3/index/
[junit4] 2> 983261 T3833 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-620D1A4C5F907DEC-001/tempDir-001/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 983263 T3833 oasc.CachingDirectoryFactory.get return new directory
[...truncated too long message...]
estRuleMarkFailure.java:48)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
[junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 97) Thread[id=4407, name=qtp29259167-4407 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 98) Thread[id=4270, name=Thread-1837-SendThread(127.0.0.1:44001), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] 2> 99) Thread[id=3803, name=qtp8410730-3803 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 100) Thread[id=4375, name=qtp4718805-4375 Acceptor1 SslSelectChannelConnector@127.0.0.1:34075, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:102)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 101) Thread[id=4342, name=searcherExecutor-2795-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 102) Thread[id=3820, name=HashSessionScavenger-107, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> NOTE: test params are: codec=Lucene49: {id=Pulsing41(freqCutoff=9 minBlockSize=48 maxBlockSize=106), _version_=PostingsFormat(name=FSTOrdPulsing41), rnd_b=Pulsing41(freqCutoff=9 minBlockSize=48 maxBlockSize=106), a_t=PostingsFormat(name=FSTOrdPulsing41), a_si=PostingsFormat(name=Direct)}, docValues:{}, sim=DefaultSimilarity, locale=es_US, timezone=Antarctica/Vostok
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=125,free=309660064,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [FieldAnalysisRequestHandlerTest, SuggestComponentTest, BadCopyFieldTest, SolrCmdDistributorTest, TestPartialUpdateDeduplication, SortByFunctionTest, TestQuerySenderNoQuery, DocValuesMultiTest, UniqFieldsUpdateProcessorFactoryTest, AnalysisErrorHandlingTest, RankQueryTest, TestDocBasedVersionConstraints, TestFileDictionaryLookup, HdfsUnloadDistributedZkTest, TestUtils, TestFieldSortValues, ZkNodePropsTest, TestDistributedGrouping, DistributedQueryComponentCustomSortTest, CollectionsAPIDistributedZkTest, TestWordDelimiterFilterFactory, TestCollationKeyRangeQueries, TestComplexPhraseQParserPlugin, BJQParserTest, SolrCoreCheckLockOnStartupTest, TestFastOutputStream, TestFieldCollectionResource, TimeZoneUtilsTest, ChaosMonkeyNothingIsSafeTest, IndexSchemaRuntimeFieldTest, DirectSolrSpellCheckerTest, TestSchemaNameResource, DirectUpdateHandlerTest, BadIndexSchemaTest, TestFreeTextSuggestions, TestDefaultSearchFieldResource, XsltUpdateRequestHandlerTest, FileUtilsTest, TestSolrDeletionPolicy2, TestUpdate, NumericFieldsTest, MinimalSchemaTest, TestClassNameShortening, TestIndexSearcher, OpenExchangeRatesOrgProviderTest, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestSweetSpotSimilarityFactory, TestCloudInspectUtil, TestStressLucene, SuggesterWFSTTest, JSONWriterTest, TestReloadAndDeleteDocs, CursorPagingTest, HardAutoCommitTest, AddSchemaFieldsUpdateProcessorFactoryTest, StandardRequestHandlerTest, SpellingQueryConverterTest, TestHighlightDedupGrouping, TestSolrQueryParserResource, LeaderElectionIntegrationTest, PathHierarchyTokenizerFactoryTest, TestSolr4Spatial, TestSolrJ, BlockDirectoryTest, TestDistributedMissingSort, StatsComponentTest, CurrencyFieldXmlFileTest, CoreContainerCoreInitFailuresTest, SSLMigrationTest, TestZkChroot, SolrRequestParserTest, DirectSolrConnectionTest, TestFieldResource, TestPhraseSuggestions, HighlighterMaxOffsetTest, TestCopyFieldCollectionResource, SolrPluginUtilsTest, ZkSolrClientTest, DistributedSuggestComponentTest, SolrIndexConfigTest, DeleteReplicaTest, TestHashPartitioner, TestFiltering, BasicDistributedZk2Test, DistribCursorPagingTest, DistributedQueryComponentOptimizationTest, TestNonNRTOpen, SuggesterTSTTest, TestCoreDiscovery, TestNonDefinedSimilarityFactory, DocExpirationUpdateProcessorFactoryTest, TestRandomFaceting, ShardRoutingCustomTest, TestHighFrequencyDictionaryFactory, TestImplicitCoreProperties, TestRecovery, TestCollapseQParserPlugin, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=620D1A4C5F907DEC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_US -Dtests.timezone=Antarctica/Vostok -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([620D1A4C5F907DEC]:0)
[junit4] Completed on J1 in 7225.66s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1655 tests, 1 suite-level error, 1 error, 37 ignored (16 assumptions)
Total time: 177 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_20-ea-b15) - Build
# 10468 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10468/
Java: 32bit/jdk1.8.0_20-ea-b15 -client -XX:+UseConcMarkSweepGC
2 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 263 from http://127.0.0.1:36925/_m/collection1lastClient and got 257 from http://127.0.0.1:48676/_m/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 263 from http://127.0.0.1:36925/_m/collection1lastClient and got 257 from http://127.0.0.1:48676/_m/collection1
at __randomizedtesting.SeedInfo.seed([8488C7086533653:89AE0268F10C566F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
No live SolrServers available to handle this request
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
at __randomizedtesting.SeedInfo.seed([8488C7086533653:89AE0268F10C566F]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:659)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1149)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:236)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11294 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/init-core-data-001
[junit4] 2> 929163 T3603 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 929164 T3603 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_m/
[junit4] 2> 929168 T3603 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 929169 T3603 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 929169 T3604 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 929269 T3603 oasc.ZkTestServer.run start zk server on port:50306
[junit4] 2> 929271 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 929283 T3610 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ed913c name:ZooKeeperConnection Watcher:127.0.0.1:50306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 929283 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 929284 T3603 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 929297 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 929299 T3612 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8926ce name:ZooKeeperConnection Watcher:127.0.0.1:50306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 929299 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 929302 T3603 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 929312 T3603 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 929318 T3603 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 929324 T3603 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 929331 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 929333 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 929344 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 929345 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 929353 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 929354 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 929360 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 929361 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 929368 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 929373 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 929380 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 929381 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 929387 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 929389 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 929396 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 929397 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 929403 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 929405 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 929411 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 929413 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 929420 T3603 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 929421 T3603 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 929734 T3603 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 929744 T3603 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47264
[junit4] 2> 929746 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 929746 T3603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 929746 T3603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002
[junit4] 2> 929747 T3603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/'
[junit4] 2> 929768 T3603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/solr.xml
[junit4] 2> 929792 T3603 oasc.CoreContainer.<init> New CoreContainer 6326791
[junit4] 2> 929793 T3603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/]
[junit4] 2> 929794 T3603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 929795 T3603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 929795 T3603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 929796 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 929796 T3603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 929796 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 929797 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 929797 T3603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 929797 T3603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 929798 T3603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 929798 T3603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 929799 T3603 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 929799 T3603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50306/solr
[junit4] 2> 929800 T3603 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 929800 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 929802 T3623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c756f5 name:ZooKeeperConnection Watcher:127.0.0.1:50306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 929802 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 929805 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 929806 T3625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16284bb name:ZooKeeperConnection Watcher:127.0.0.1:50306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 929806 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 929808 T3603 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 929813 T3603 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 929817 T3603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 929819 T3603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 929823 T3603 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 929828 T3603 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 929832 T3603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47264__m
[junit4] 2> 929834 T3603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47264__m
[junit4] 2> 929840 T3603 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 929845 T3603 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 929855 T3603 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47264__m
[junit4] 2> 929855 T3603 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 929858 T3603 oasc.Overseer.start Overseer (id=91949945944014851-127.0.0.1:47264__m-n_0000000000) starting
[junit4] 2> 929865 T3603 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 929878 T3627 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 929879 T3603 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 929882 T3603 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 929884 T3603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 929887 T3626 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 929890 T3628 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 929890 T3628 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 929892 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 929892 T3628 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 929893 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 929894 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:47264/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47264__m",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 929894 T3626 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 929894 T3626 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 929897 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 929898 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 930893 T3628 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 930893 T3628 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 930894 T3628 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 930894 T3628 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 930895 T3628 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 930896 T3628 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/collection1/'
[junit4] 2> 930896 T3628 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 930897 T3628 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 930897 T3628 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 930954 T3628 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 930976 T3628 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 930977 T3628 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 931026 T3628 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 931165 T3628 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 931167 T3628 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 931169 T3628 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 931173 T3628 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 931207 T3628 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 931207 T3628 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 931208 T3628 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/
[junit4] 2> 931209 T3628 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10d84da
[junit4] 2> 931210 T3628 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data
[junit4] 2> 931211 T3628 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/index/
[junit4] 2> 931212 T3628 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 931212 T3628 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/index
[junit4] 2> 931213 T3628 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41854129625589753]
[junit4] 2> 931217 T3628 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 931217 T3628 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 931222 T3628 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 931223 T3628 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 931223 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 931223 T3628 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 931224 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 931224 T3628 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 931225 T3628 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 931225 T3628 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 931225 T3628 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 931226 T3628 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 931227 T3628 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 931227 T3628 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 931227 T3628 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 931228 T3628 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 931228 T3628 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 931229 T3628 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 931231 T3628 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 931233 T3628 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 931233 T3628 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 931234 T3628 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=52.6171875, floorSegmentMB=0.9375, forceMergeDeletesPctAllowed=16.626623555765985, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5998830654502972
[junit4] 2> 931235 T3628 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 931236 T3628 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 931236 T3628 oass.SolrIndexSearcher.<init> Opening Searcher@efc47[collection1] main
[junit4] 2> 931236 T3628 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 931238 T3628 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 931238 T3628 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 931238 T3628 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 931239 T3628 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 931239 T3628 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 931240 T3628 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 931240 T3628 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 931240 T3628 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 931240 T3628 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 931245 T3629 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@efc47[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 931247 T3628 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 931248 T3632 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47264/_m collection:control_collection shard:shard1
[junit4] 2> 931248 T3603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 931249 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 931249 T3632 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 931250 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 931251 T3634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ab2ddb name:ZooKeeperConnection Watcher:127.0.0.1:50306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 931252 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 931254 T3603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 931256 T3632 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 931256 T3603 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 931259 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931259 T3632 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 931259 T3632 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C5154 name=collection1 org.apache.solr.core.SolrCore@183ed2b url=http://127.0.0.1:47264/_m/collection1 node=127.0.0.1:47264__m C5154_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:47264/_m, core=collection1, node_name=127.0.0.1:47264__m}
[junit4] 2> 931260 T3632 C5154 P47264 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47264/_m/collection1/
[junit4] 2> 931260 T3632 C5154 P47264 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 931261 T3632 C5154 P47264 oasc.SyncStrategy.syncToMe http://127.0.0.1:47264/_m/collection1/ has no replicas
[junit4] 2> 931261 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 931261 T3632 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47264/_m/collection1/ shard1
[junit4] 2> 931262 T3632 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 931265 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931269 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931272 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931374 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 931374 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 931421 T3632 oasc.ZkController.register We are http://127.0.0.1:47264/_m/collection1/ and leader is http://127.0.0.1:47264/_m/collection1/
[junit4] 2> 931422 T3632 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47264/_m
[junit4] 2> 931422 T3632 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 931422 T3632 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 931422 T3632 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 931424 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931424 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931424 T3632 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 931424 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931426 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 931427 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:47264/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47264__m",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 931429 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 931531 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 931532 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 931602 T3603 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 931603 T3603 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 931606 T3603 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43387
[junit4] 2> 931607 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 931608 T3603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 931609 T3603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003
[junit4] 2> 931609 T3603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/'
[junit4] 2> 931636 T3603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/solr.xml
[junit4] 2> 931649 T3603 oasc.CoreContainer.<init> New CoreContainer 10155778
[junit4] 2> 931650 T3603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/]
[junit4] 2> 931651 T3603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 931651 T3603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 931652 T3603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 931652 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 931652 T3603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 931653 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 931653 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 931653 T3603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 931654 T3603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 931655 T3603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 931656 T3603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 931656 T3603 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 931657 T3603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50306/solr
[junit4] 2> 931657 T3603 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 931658 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 931660 T3645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72eafc name:ZooKeeperConnection Watcher:127.0.0.1:50306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 931661 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 931667 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 931672 T3647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c85ba8 name:ZooKeeperConnection Watcher:127.0.0.1:50306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 931672 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 931680 T3603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 932684 T3603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43387__m
[junit4] 2> 932686 T3603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43387__m
[junit4] 2> 932691 T3634 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 932691 T3647 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 932691 T3625 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 932701 T3648 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 932701 T3648 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 932704 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 932704 T3648 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 932704 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 932705 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 932708 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 932710 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43387/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43387__m",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 932710 T3626 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 932711 T3626 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 932716 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 932819 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 932820 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 932820 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 933705 T3648 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 933706 T3648 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 933707 T3648 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 933708 T3648 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 933709 T3648 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 933710 T3648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/collection1/'
[junit4] 2> 933711 T3648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 933712 T3648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 933713 T3648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 933759 T3648 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 933771 T3648 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 933774 T3648 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 933783 T3648 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 933927 T3648 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 933929 T3648 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 933930 T3648 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 933933 T3648 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 933952 T3648 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 933952 T3648 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 933953 T3648 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/
[junit4] 2> 933953 T3648 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10d84da
[junit4] 2> 933954 T3648 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1
[junit4] 2> 933954 T3648 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/index/
[junit4] 2> 933954 T3648 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 933955 T3648 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/index
[junit4] 2> 933955 T3648 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41854129625589753]
[junit4] 2> 933959 T3648 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 933959 T3648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 933964 T3648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 933964 T3648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 933965 T3648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 933965 T3648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 933965 T3648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 933965 T3648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 933966 T3648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 933966 T3648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 933966 T3648 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 933967 T3648 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 933968 T3648 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 933968 T3648 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 933969 T3648 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 933969 T3648 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 933970 T3648 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 933971 T3648 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 933972 T3648 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 933974 T3648 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 933974 T3648 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 933975 T3648 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=52.6171875, floorSegmentMB=0.9375, forceMergeDeletesPctAllowed=16.626623555765985, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5998830654502972
[junit4] 2> 933976 T3648 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 933977 T3648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 933977 T3648 oass.SolrIndexSearcher.<init> Opening Searcher@62b4d[collection1] main
[junit4] 2> 933977 T3648 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 933978 T3648 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 933978 T3648 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 933979 T3648 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 933979 T3648 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 933979 T3648 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 933980 T3648 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 933980 T3648 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 933981 T3648 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 933981 T3648 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 933986 T3649 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62b4d[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 933989 T3648 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 933990 T3652 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43387/_m collection:collection1 shard:shard2
[junit4] 2> 933991 T3603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 933991 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 933991 T3652 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 933998 T3652 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 934000 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934000 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934000 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934001 T3652 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 934001 T3652 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 934002 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C5155 name=collection1 org.apache.solr.core.SolrCore@10a5d7a url=http://127.0.0.1:43387/_m/collection1 node=127.0.0.1:43387__m C5155_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43387/_m, core=collection1, node_name=127.0.0.1:43387__m}
[junit4] 2> 934002 T3652 C5155 P43387 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43387/_m/collection1/
[junit4] 2> 934002 T3652 C5155 P43387 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 934003 T3652 C5155 P43387 oasc.SyncStrategy.syncToMe http://127.0.0.1:43387/_m/collection1/ has no replicas
[junit4] 2> 934003 T3652 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43387/_m/collection1/ shard2
[junit4] 2> 934003 T3652 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 934004 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934008 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934013 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934117 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934117 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934117 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934160 T3652 oasc.ZkController.register We are http://127.0.0.1:43387/_m/collection1/ and leader is http://127.0.0.1:43387/_m/collection1/
[junit4] 2> 934160 T3652 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43387/_m
[junit4] 2> 934160 T3652 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 934161 T3652 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 934161 T3652 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 934163 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934163 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934163 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934163 T3652 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 934164 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 934165 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:43387/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43387__m",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 934167 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 934269 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934269 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934269 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 934278 T3603 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 934279 T3603 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 934281 T3603 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35688
[junit4] 2> 934283 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 934283 T3603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 934283 T3603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004
[junit4] 2> 934284 T3603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/'
[junit4] 2> 934311 T3603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/solr.xml
[junit4] 2> 934334 T3603 oasc.CoreContainer.<init> New CoreContainer 2283123
[junit4] 2> 934335 T3603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/]
[junit4] 2> 934337 T3603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 934338 T3603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 934338 T3603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 934339 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 934339 T3603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 934340 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 934340 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 934341 T3603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 934341 T3603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 934344 T3603 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 934344 T3603 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 934345 T3603 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 934345 T3603 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50306/solr
[junit4] 2> 934346 T3603 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 934347 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 934348 T3663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d93b49 name:ZooKeeperConnection Watcher:127.0.0.1:50306 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 934349 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 934351 T3603 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 934353 T3665 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c6799 name:ZooKeeperConnection Watcher:127.0.0.1:50306/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 934353 T3603 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 934359 T3603 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 935363 T3603 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35688__m
[junit4] 2> 935365 T3603 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35688__m
[junit4] 2> 935369 T3634 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 935370 T3625 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 935370 T3647 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 935370 T3665 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 935378 T3666 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 935379 T3666 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 935383 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 935383 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 935383 T3666 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 935384 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 935385 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 935386 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35688/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35688__m",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 935386 T3626 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 935387 T3626 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 935388 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 935490 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 935490 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 935490 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 935490 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936384 T3666 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 936385 T3666 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 936386 T3666 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 936386 T3666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 936387 T3666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 936387 T3666 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/collection1/'
[junit4] 2> 936388 T3666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 936389 T3666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 936389 T3666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 936428 T3666 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 936441 T3666 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 936443 T3666 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 936452 T3666 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 936584 T3666 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 936586 T3666 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 936587 T3666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 936590 T3666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 936625 T3666 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 936625 T3666 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 936626 T3666 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/
[junit4] 2> 936626 T3666 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@10d84da
[junit4] 2> 936627 T3666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2
[junit4] 2> 936627 T3666 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/index/
[junit4] 2> 936628 T3666 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 936628 T3666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/index
[junit4] 2> 936629 T3666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41854129625589753]
[junit4] 2> 936634 T3666 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 936635 T3666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 936641 T3666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 936642 T3666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 936642 T3666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 936642 T3666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 936643 T3666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 936643 T3666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 936643 T3666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 936644 T3666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 936644 T3666 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 936645 T3666 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 936646 T3666 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 936646 T3666 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 936647 T3666 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 936648 T3666 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 936648 T3666 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 936650 T3666 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 936652 T3666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 936653 T3666 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 936654 T3666 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 936655 T3666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=52.6171875, floorSegmentMB=0.9375, forceMergeDeletesPctAllowed=16.626623555765985, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5998830654502972
[junit4] 2> 936656 T3666 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 936656 T3666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 936656 T3666 oass.SolrIndexSearcher.<init> Opening Searcher@1802982[collection1] main
[junit4] 2> 936657 T3666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 936658 T3666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 936658 T3666 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 936658 T3666 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 936659 T3666 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 936659 T3666 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 936659 T3666 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 936660 T3666 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 936660 T3666 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 936660 T3666 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 936666 T3667 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1802982[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 936669 T3666 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 936670 T3670 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35688/_m collection:collection1 shard:shard3
[junit4] 2> 936671 T3603 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 936671 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 936671 T3670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 936676 T3670 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 936678 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936678 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936678 T3670 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 936679 T3670 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 936678 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C5156 name=collection1 org.apache.solr.core.SolrCore@e0eede url=http://127.0.0.1:35688/_m/collection1 node=127.0.0.1:35688__m C5156_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35688/_m, core=collection1, node_name=127.0.0.1:35688__m}
[junit4] 2> 936679 T3670 C5156 P35688 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35688/_m/collection1/
[junit4] 2> 936679 T3670 C5156 P35688 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 936680 T3670 C5156 P35688 oasc.SyncStrategy.syncToMe http://127.0.0.1:35688/_m/collection1/ has no replicas
[junit4] 2> 936680 T3670 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35688/_m/collection1/ shard3
[junit4] 2> 936680 T3670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 936680 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 936683 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936684 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936686 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936791 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936791 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936791 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936791 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936835 T3670 oasc.ZkController.register We are http://127.0.0.1:35688/_m/collection1/ and leader is http://127.0.0.1:35688/_m/collection1/
[junit4] 2> 936836 T3670 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35688/_m
[junit4] 2> 936836 T3670 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 936836 T3670 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 936836 T3670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 936838 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936838 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936839 T3670 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 936839 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936840 T3626 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 936840 T3626 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:35688/_m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35688__m",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 936842 T3625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 936944 T3634 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936944 T3665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936944 T3625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936944 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 936956 T3603 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 936957 T3603 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 936960 T3603 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38561
[junit4] 2> 936961 T3603 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 936962 T3603 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 936962 T3603 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-005
[junit4] 2> 936963 T3603 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-005/'
[junit4] 2> 936990 T3603 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-005/solr.xml
[junit4] 2> 937007 T3603 oasc.CoreContainer.<init> New CoreContainer 17909346
[junit4] 2> 937008 T3603 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-8488C7086533653-001/tempDir-005/]
[junit4] 2> 937008 T3603 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 937008 T3603 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 937009 T3603 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 937009 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 937010 T3603 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 937010 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 937010 T3603 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 937011 T3603 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 937011 T3603 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 937012 T3603 oasl.LogW
[...truncated too long message...]
r.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:190)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 1180612 T4727 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1180613 T4727 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1180613 T4727 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1180614 T4727 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=8488C7086533653 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_OM -Dtests.timezone=Etc/Greenwich -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 65.4s J0 | SyncSliceTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
[junit4] > at __randomizedtesting.SeedInfo.seed([8488C7086533653:89AE0268F10C566F]:0)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:659)
[junit4] > at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
[junit4] > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1149)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
[junit4] > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:236)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1180628 T4624 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8488C7086533653-001
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=ar_OM, timezone=Etc/Greenwich
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.8.0_20-ea (32-bit)/cpus=8,threads=1,free=186895952,total=440737792
[junit4] 2> NOTE: All tests run in this JVM: [TestTrie, CollectionsAPIAsyncDistributedZkTest, DistribCursorPagingTest, TestLMJelinekMercerSimilarityFactory, TestReplicationHandlerBackup, TestReloadAndDeleteDocs, TestConfigSets, UpdateParamsTest, DateMathParserTest, DocValuesTest, MultiThreadedOCPTest, LukeRequestHandlerTest, BadCopyFieldTest, AssignTest, ResponseLogComponentTest, ShardRoutingTest, LegacyHTMLStripCharFilterTest, AnalyticsQueryTest, TestFaceting, AnalysisAfterCoreReloadTest, TestSystemIdResolver, TestFunctionQuery, PeerSyncTest, TestConfig, FileBasedSpellCheckerTest, SearchHandlerTest, TestJmxMonitoredMap, TestBM25SimilarityFactory, TestLuceneMatchVersion, TestSchemaResource, SampleTest, TestNoOpRegenerator, BasicFunctionalityTest, TestLMDirichletSimilarityFactory, SpellPossibilityIteratorTest, BasicDistributedZkTest, SchemaVersionSpecificBehaviorTest, TestShortCircuitedRequests, TestComplexPhraseQParserPlugin, ResponseHeaderTest, UUIDFieldTest, TestReversedWildcardFilterFactory, MBeansHandlerTest, DistributedQueryComponentOptimizationTest, CoreAdminRequestStatusTest, DistributedTermsComponentTest, MoreLikeThisHandlerTest, JsonLoaderTest, TestSolrXmlPersistence, DistributedQueryElevationComponentTest, SystemInfoHandlerTest, HdfsRecoveryZkTest, TestMergePolicyConfig, TestSuggestSpellingConverter, RegexBoostProcessorTest, TestQueryUtils, EnumFieldTest, AtomicUpdatesTest, CachingDirectoryFactoryTest, DistributedSpellCheckComponentTest, SliceStateUpdateTest, TestElisionMultitermQuery, TestSolrQueryParser, ReturnFieldsTest, HdfsSyncSliceTest, TestSolrIndexConfig, TestCloudManagedSchemaAddFields, TestFieldCollectionResource, TestClassNameShortening, TestRandomMergePolicy, TestGroupingSearch, ExternalFileFieldSortTest, TestStressVersions, TestPerFieldSimilarity, CircularListTest, SuggesterTest, SolrIndexSplitterTest, TestFieldTypeCollectionResource, ReplicationFactorTest, FieldMutatingUpdateProcessorTest, TestExpandComponent, TriLevelCompositeIdRoutingTest, IndexSchemaTest, TestCollapseQParserPlugin, CacheHeaderTest, DistributedQueryComponentCustomSortTest, OverseerRolesTest, TestSolrQueryParserResource, PolyFieldTest, TestPostingsSolrHighlighter, TestCloudInspectUtil, TestShardHandlerFactory, SolrInfoMBeanTest, TestStandardQParsers, TestRemoteStreaming, TestExtendedDismaxParser, TestFieldResource, TestRecoveryHdfs, OverseerCollectionProcessorTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, SyncSliceTest]
[junit4] Completed on J0 in 65.46s, 1 test, 1 error <<< FAILURES!
[...truncated 728 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1655 tests, 1 error, 1 failure, 37 ignored (16 assumptions)
Total time: 83 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_20-ea-b15 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0_20-ea-b15) - Build
# 10467 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10467/
Java: 32bit/jdk1.8.0_20-ea-b15 -client -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.SyncSliceTest.testDistribSearch
Error Message:
No live SolrServers available to handle this request
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
at __randomizedtesting.SeedInfo.seed([8096EE4CFAC6C62C:17060548D99A610]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:317)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:659)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1149)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:236)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11488 lines...]
[junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/init-core-data-001
[junit4] 2> 603793 T2627 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 603794 T2627 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 603797 T2627 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 603798 T2627 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 603799 T2628 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 603899 T2627 oasc.ZkTestServer.run start zk server on port:34258
[junit4] 2> 603900 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 603973 T2634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc6f6 name:ZooKeeperConnection Watcher:127.0.0.1:34258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 603974 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 603974 T2627 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 603980 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 603981 T2636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84ecc0 name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 603982 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 603982 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 603986 T2627 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 603990 T2627 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 603993 T2627 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 603999 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 603999 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 604007 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 604007 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 604011 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 604012 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 604015 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 604016 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 604019 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 604020 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 604023 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 604024 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 604027 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 604028 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 604031 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 604032 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 604035 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 604036 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 604040 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 604040 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 604044 T2627 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 604045 T2627 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 604359 T2627 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 604363 T2627 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49071
[junit4] 2> 604363 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 604364 T2627 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 604365 T2627 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002
[junit4] 2> 604365 T2627 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/'
[junit4] 2> 604383 T2627 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/solr.xml
[junit4] 2> 604396 T2627 oasc.CoreContainer.<init> New CoreContainer 32723421
[junit4] 2> 604397 T2627 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/]
[junit4] 2> 604397 T2627 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 604398 T2627 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 604398 T2627 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 604398 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 604399 T2627 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 604399 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 604399 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 604400 T2627 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 604400 T2627 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 604401 T2627 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 604401 T2627 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 604401 T2627 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 604402 T2627 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34258/solr
[junit4] 2> 604402 T2627 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 604403 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 604405 T2647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@780c25 name:ZooKeeperConnection Watcher:127.0.0.1:34258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 604405 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 604408 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 604409 T2649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b0869 name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 604409 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 604411 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 604416 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 604421 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 604424 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 604428 T2627 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 604435 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 604437 T2627 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49071_
[junit4] 2> 604439 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49071_
[junit4] 2> 604442 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 604444 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 604448 T2627 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49071_
[junit4] 2> 604449 T2627 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 604451 T2627 oasc.Overseer.start Overseer (id=91949256544616451-127.0.0.1:49071_-n_0000000000) starting
[junit4] 2> 604454 T2627 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 604465 T2651 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 604466 T2627 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 604469 T2627 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 604471 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 604477 T2650 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 604480 T2652 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 604480 T2652 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 604482 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 604482 T2652 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 604483 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 604484 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:49071",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49071_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 604485 T2650 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 604485 T2650 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 604487 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 604488 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 605482 T2652 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 605483 T2652 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 605484 T2652 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 605484 T2652 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 605486 T2652 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 605486 T2652 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/collection1/'
[junit4] 2> 605487 T2652 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 605488 T2652 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 605488 T2652 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 605525 T2652 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 605538 T2652 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 605540 T2652 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 605560 T2652 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 605694 T2652 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 605695 T2652 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 605696 T2652 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 605710 T2652 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 605713 T2652 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 605728 T2652 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 605731 T2652 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 605734 T2652 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 605735 T2652 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 605735 T2652 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 605735 T2652 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 605736 T2652 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 605737 T2652 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 605737 T2652 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 605737 T2652 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 605738 T2652 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-002/collection1/, dataDir=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/control/data/
[junit4] 2> 605738 T2652 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@186ccdc
[junit4] 2> 605739 T2652 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/control/data
[junit4] 2> 605739 T2652 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/control/data/index/
[junit4] 2> 605739 T2652 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 605740 T2652 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/control/data/index
[junit4] 2> 605740 T2652 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=88.484375, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=22.757069569670552, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4246451526639434
[junit4] 2> 605741 T2652 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@5bcd81 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b78aa4),segFN=segments_1,generation=1}
[junit4] 2> 605741 T2652 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 605745 T2652 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 605745 T2652 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 605746 T2652 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 605746 T2652 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 605746 T2652 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 605747 T2652 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 605747 T2652 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 605748 T2652 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 605748 T2652 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 605749 T2652 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 605749 T2652 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 605750 T2652 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 605750 T2652 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 605751 T2652 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 605751 T2652 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 605752 T2652 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 605754 T2652 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 605756 T2652 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 605756 T2652 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 605757 T2652 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=2.109375, noCFSRatio=1.0]
[junit4] 2> 605758 T2652 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@5bcd81 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1b78aa4),segFN=segments_1,generation=1}
[junit4] 2> 605758 T2652 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 605759 T2652 oass.SolrIndexSearcher.<init> Opening Searcher@1038b7c[collection1] main
[junit4] 2> 605759 T2652 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 605760 T2652 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 605760 T2652 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 605761 T2652 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 605762 T2652 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 605762 T2652 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 605763 T2652 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 605763 T2652 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 605763 T2652 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 605764 T2652 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 605768 T2653 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1038b7c[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 605769 T2652 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 605770 T2656 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49071 collection:control_collection shard:shard1
[junit4] 2> 605771 T2627 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 605771 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 605771 T2656 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 605773 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 605775 T2658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1029037 name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 605776 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 605778 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 605782 T2627 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 605783 T2656 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 605785 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605786 T2656 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 605786 T2656 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C762 name=collection1 org.apache.solr.core.SolrCore@17cc9e4 url=http://127.0.0.1:49071/collection1 node=127.0.0.1:49071_ C762_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:49071, core=collection1, node_name=127.0.0.1:49071_}
[junit4] 2> 605786 T2656 C762 P49071 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49071/collection1/
[junit4] 2> 605787 T2656 C762 P49071 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 605787 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 605787 T2656 C762 P49071 oasc.SyncStrategy.syncToMe http://127.0.0.1:49071/collection1/ has no replicas
[junit4] 2> 605788 T2656 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49071/collection1/ shard1
[junit4] 2> 605788 T2656 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 605793 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605801 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605806 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605908 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 605908 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 605953 T2656 oasc.ZkController.register We are http://127.0.0.1:49071/collection1/ and leader is http://127.0.0.1:49071/collection1/
[junit4] 2> 605953 T2656 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49071
[junit4] 2> 605953 T2656 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 605954 T2656 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 605954 T2656 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 605955 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605956 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605956 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 605956 T2656 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 605957 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 605958 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:49071",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49071_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 605961 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 606063 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 606063 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 606083 T2627 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 606084 T2627 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 606086 T2627 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43607
[junit4] 2> 606087 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 606088 T2627 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 606088 T2627 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003
[junit4] 2> 606089 T2627 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/'
[junit4] 2> 606107 T2627 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/solr.xml
[junit4] 2> 606123 T2627 oasc.CoreContainer.<init> New CoreContainer 10357821
[junit4] 2> 606123 T2627 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/]
[junit4] 2> 606124 T2627 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 606125 T2627 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 606125 T2627 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 606126 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 606126 T2627 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 606127 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 606127 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 606128 T2627 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 606128 T2627 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 606129 T2627 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 606130 T2627 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 606130 T2627 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 606131 T2627 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34258/solr
[junit4] 2> 606131 T2627 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 606132 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606134 T2669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d879a9 name:ZooKeeperConnection Watcher:127.0.0.1:34258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606135 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606137 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 606138 T2671 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e2e622 name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 606138 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 606146 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 607149 T2627 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43607_
[junit4] 2> 607151 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43607_
[junit4] 2> 607154 T2671 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 607154 T2649 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 607154 T2658 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 607159 T2672 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 607160 T2672 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 607161 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607161 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607161 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607161 T2672 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 607163 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 607163 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43607",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43607_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 607164 T2650 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 607164 T2650 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 607167 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 607269 T2671 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 607269 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 607269 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608162 T2672 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 608162 T2672 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 608163 T2672 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 608163 T2672 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 608164 T2672 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 608165 T2672 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/collection1/'
[junit4] 2> 608166 T2672 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 608166 T2672 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 608167 T2672 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 608199 T2672 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 608211 T2672 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 608212 T2672 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 608224 T2672 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 608353 T2672 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 608354 T2672 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 608354 T2672 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 608361 T2672 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 608362 T2672 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 608380 T2672 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 608383 T2672 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 608385 T2672 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 608386 T2672 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 608387 T2672 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 608387 T2672 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 608388 T2672 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 608388 T2672 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 608388 T2672 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 608388 T2672 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 608389 T2672 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-003/collection1/, dataDir=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty1/
[junit4] 2> 608389 T2672 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@186ccdc
[junit4] 2> 608390 T2672 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty1
[junit4] 2> 608390 T2672 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty1/index/
[junit4] 2> 608390 T2672 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 608391 T2672 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty1/index
[junit4] 2> 608391 T2672 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=88.484375, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=22.757069569670552, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4246451526639434
[junit4] 2> 608392 T2672 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@fc24d1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cf2285),segFN=segments_1,generation=1}
[junit4] 2> 608392 T2672 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 608397 T2672 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 608398 T2672 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 608398 T2672 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 608398 T2672 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 608399 T2672 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 608399 T2672 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 608399 T2672 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 608400 T2672 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 608400 T2672 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 608401 T2672 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 608402 T2672 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 608402 T2672 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 608402 T2672 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 608403 T2672 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 608403 T2672 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 608404 T2672 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 608406 T2672 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 608408 T2672 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 608408 T2672 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 608409 T2672 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=2.109375, noCFSRatio=1.0]
[junit4] 2> 608410 T2672 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@fc24d1 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@cf2285),segFN=segments_1,generation=1}
[junit4] 2> 608410 T2672 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 608410 T2672 oass.SolrIndexSearcher.<init> Opening Searcher@119b187[collection1] main
[junit4] 2> 608410 T2672 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 608411 T2672 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 608412 T2672 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 608412 T2672 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 608412 T2672 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 608413 T2672 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 608413 T2672 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 608414 T2672 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 608414 T2672 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 608414 T2672 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 608420 T2673 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@119b187[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 608424 T2672 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 608424 T2676 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43607 collection:collection1 shard:shard1
[junit4] 2> 608425 T2627 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 608425 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 608425 T2676 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 608432 T2676 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 608434 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608434 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608435 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608435 T2676 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 608436 T2676 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C763 name=collection1 org.apache.solr.core.SolrCore@5518cd url=http://127.0.0.1:43607/collection1 node=127.0.0.1:43607_ C763_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43607, core=collection1, node_name=127.0.0.1:43607_}
[junit4] 2> 608436 T2676 C763 P43607 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43607/collection1/
[junit4] 2> 608436 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 608436 T2676 C763 P43607 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 608437 T2676 C763 P43607 oasc.SyncStrategy.syncToMe http://127.0.0.1:43607/collection1/ has no replicas
[junit4] 2> 608437 T2676 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43607/collection1/ shard1
[junit4] 2> 608438 T2676 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 608439 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608445 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608450 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608552 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608553 T2671 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608552 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608597 T2676 oasc.ZkController.register We are http://127.0.0.1:43607/collection1/ and leader is http://127.0.0.1:43607/collection1/
[junit4] 2> 608598 T2676 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43607
[junit4] 2> 608598 T2676 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 608598 T2676 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 608598 T2676 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 608600 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608600 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608601 T2676 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 608601 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608602 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 608603 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:43607",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43607_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 608605 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 608701 T2627 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 608702 T2627 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 608704 T2627 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42043
[junit4] 2> 608705 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 608706 T2627 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 608706 T2627 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004
[junit4] 2> 608706 T2627 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/'
[junit4] 2> 608707 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608707 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608707 T2671 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 608724 T2627 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/solr.xml
[junit4] 2> 608734 T2627 oasc.CoreContainer.<init> New CoreContainer 26529791
[junit4] 2> 608735 T2627 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/]
[junit4] 2> 608735 T2627 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 608736 T2627 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 608736 T2627 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 608737 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 608737 T2627 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 608737 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 608738 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 608738 T2627 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 608738 T2627 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 608739 T2627 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 608739 T2627 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 608739 T2627 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 608740 T2627 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34258/solr
[junit4] 2> 608740 T2627 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 608741 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 608743 T2687 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5d57 name:ZooKeeperConnection Watcher:127.0.0.1:34258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 608743 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 608745 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 608746 T2689 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14ebe9f name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 608746 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 608753 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 609757 T2627 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42043_
[junit4] 2> 609759 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42043_
[junit4] 2> 609762 T2649 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 609762 T2658 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 609762 T2689 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 609762 T2671 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 609767 T2690 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 609767 T2690 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 609772 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 609772 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 609773 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 609772 T2690 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 609774 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 609775 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:42043",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42043_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 609775 T2650 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 609775 T2650 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 609777 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 609879 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 609879 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 609879 T2671 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 609880 T2689 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 610774 T2690 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 610774 T2690 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 610775 T2690 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 610775 T2690 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 610776 T2690 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 610776 T2690 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/collection1/'
[junit4] 2> 610777 T2690 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 610777 T2690 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 610778 T2690 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 610812 T2690 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_10
[junit4] 2> 610824 T2690 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 610825 T2690 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 610833 T2690 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 610965 T2690 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 610966 T2690 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 610967 T2690 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 610973 T2690 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 610975 T2690 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 610991 T2690 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 610994 T2690 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 610997 T2690 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 610998 T2690 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 610998 T2690 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 610998 T2690 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 610999 T2690 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 610999 T2690 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 611000 T2690 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 611000 T2690 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 611000 T2690 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-004/collection1/, dataDir=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty2/
[junit4] 2> 611000 T2690 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@186ccdc
[junit4] 2> 611001 T2690 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty2
[junit4] 2> 611001 T2690 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty2/index/
[junit4] 2> 611002 T2690 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 611002 T2690 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty2/index
[junit4] 2> 611002 T2690 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=31, maxMergedSegmentMB=88.484375, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=22.757069569670552, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4246451526639434
[junit4] 2> 611003 T2690 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1831072 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cbd44),segFN=segments_1,generation=1}
[junit4] 2> 611004 T2690 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 611007 T2690 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 611008 T2690 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 611008 T2690 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 611008 T2690 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 611008 T2690 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 611009 T2690 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 611009 T2690 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 611009 T2690 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 611010 T2690 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 611010 T2690 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 611011 T2690 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 611011 T2690 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 611012 T2690 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 611013 T2690 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 611013 T2690 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 611014 T2690 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 611016 T2690 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 611017 T2690 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 611017 T2690 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 611018 T2690 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=17, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=2.109375, noCFSRatio=1.0]
[junit4] 2> 611019 T2690 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1831072 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4cbd44),segFN=segments_1,generation=1}
[junit4] 2> 611019 T2690 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 611019 T2690 oass.SolrIndexSearcher.<init> Opening Searcher@12d8176[collection1] main
[junit4] 2> 611019 T2690 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 611020 T2690 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 611021 T2690 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 611021 T2690 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 611021 T2690 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 611022 T2690 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 611022 T2690 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 611023 T2690 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 611023 T2690 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 611023 T2690 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 611028 T2691 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12d8176[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 611031 T2690 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 611031 T2694 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42043 collection:collection1 shard:shard1
[junit4] 2> 611032 T2627 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 611032 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 611034 T2694 oasc.ZkController.register We are http://127.0.0.1:42043/collection1/ and leader is http://127.0.0.1:43607/collection1/
[junit4] 2> 611035 T2694 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42043
[junit4] 2> 611035 T2694 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 611035 T2694 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C764 name=collection1 org.apache.solr.core.SolrCore@12ba059 url=http://127.0.0.1:42043/collection1 node=127.0.0.1:42043_ C764_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:42043, core=collection1, node_name=127.0.0.1:42043_}
[junit4] 2> 611035 T2695 C764 P42043 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 611036 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 611036 T2695 C764 P42043 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 611037 T2695 C764 P42043 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:43607/collection1/ and I am http://127.0.0.1:42043/collection1/
[junit4] 2> 611037 T2695 C764 P42043 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 611037 T2695 C764 P42043 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 611039 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611039 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611039 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611041 T2695 C764 P42043 oasc.RecoveryStrategy.sendPrepRecoveryCmd Sending prep recovery command to http://127.0.0.1:43607; WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1%3A42043_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true
[junit4] 2> 611041 T2650 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 611042 T2650 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:42043",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42043_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 611043 T2667 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 611044 T2667 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 611044 T2667 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 611045 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 611045 T2667 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:42043_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"state":"down","base_url":"http://127.0.0.1:42043","core":"collection1","node_name":"127.0.0.1:42043_"}
[junit4] 2> 611148 T2689 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611148 T2649 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611148 T2671 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611148 T2658 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 611314 T2627 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 611315 T2627 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 611317 T2627 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54136
[junit4] 2> 611319 T2627 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 611319 T2627 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 611320 T2627 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-005
[junit4] 2> 611320 T2627 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-005/'
[junit4] 2> 611337 T2627 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-005/solr.xml
[junit4] 2> 611346 T2627 oasc.CoreContainer.<init> New CoreContainer 5612984
[junit4] 2> 611347 T2627 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-005/]
[junit4] 2> 611347 T2627 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 611348 T2627 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 611348 T2627 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 611348 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 611349 T2627 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 611349 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 611349 T2627 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 611350 T2627 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 611350 T2627 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 611350 T2627 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 611351 T2627 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 611351 T2627 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 611352 T2627 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34258/solr
[junit4] 2> 611352 T2627 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 611353 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 611354 T2707 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@87bc92 name:ZooKeeperConnection Watcher:127.0.0.1:34258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 611355 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 611357 T2627 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 611358 T2709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@187a9b name:ZooKeeperConnection Watcher:127.0.0.1:34258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 611358 T2627 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 611366 T2627 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 612046 T2667 oasha.CoreAdminHandler.handleWaitForStateAction In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:42043_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"state":"recovering","base_url":"http://127.0.0.1:42043","core":"collection1","node_name":"127.0.0.1:42043_"}
[junit4] 2> 612046 T2667 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 612047 T2667 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:42043_&onlyIfLeaderActive=true&core=collection1&coreNodeName=core_node2&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1004
[junit4] 2> 612369 T2627 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54136_
[junit4] 2> 612370 T2627 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54136_
[junit4] 2> 612373 T2658 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 612373 T2649 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 612373 T2709 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 612373 T2689 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 612373 T2671 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 612378 T2710 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 612378 T2710 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 612379 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612379 T2710 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 612379 T2649 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 612380 T2649 oasc.Dist
[...truncated too long message...]
closing IndexWriter with IndexWriterCloser
[junit4] 2> 658994 T2729 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 658994 T2729 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 658995 T2729 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4/index [CachedDir<<refCount=0;path=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4/index;done=false>>]
[junit4] 2> 658995 T2729 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4/index
[junit4] 2> 658995 T2729 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4 [CachedDir<<refCount=0;path=./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4;done=false>>]
[junit4] 2> 658995 T2729 oasc.CachingDirectoryFactory.close Closing directory: ./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001/tempDir-001/jetty4
[junit4] 2> 658995 T2729 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election/91949256544616460-core_node4-n_0000000015
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:177)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:174)
[junit4] 2> at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:71)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:200)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:444)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:208)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 658996 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 658996 T2729 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 658997 T2729 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 658997 T2729 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solr.cloud.SyncSliceTest-8096EE4CFAC6C62C-001
[junit4] 2> NOTE: test params are: codec=Lucene46, sim=DefaultSimilarity, locale=es_BO, timezone=Europe/Andorra
[junit4] 2> NOTE: Linux 3.8.0-41-generic i386/Oracle Corporation 1.8.0_20-ea (32-bit)/cpus=8,threads=1,free=25023984,total=286523392
[junit4] 2> NOTE: All tests run in this JVM: [EchoParamsTest, MultiThreadedOCPTest, TestSurroundQueryParser, TestCoreContainer, ResponseLogComponentTest, OverseerRolesTest, TestSchemaResource, TestBadConfig, PrimitiveFieldTypeTest, TestNRTOpen, OverseerCollectionProcessorTest, HdfsSyncSliceTest, IndexSchemaTest, AtomicUpdatesTest, CachingDirectoryFactoryTest, ChangedSchemaMergeTest, TestRealTimeGet, QueryEqualityTest, TestShortCircuitedRequests, OverseerStatusTest, RegexBoostProcessorTest, TestCloudManagedSchemaAddFields, TestFreeTextSuggestions, HdfsRecoveryZkTest, CoreAdminRequestStatusTest, FieldMutatingUpdateProcessorTest, TestDistributedMissingSort, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy1, TestFieldResource, TestSearchPerf, TestRandomMergePolicy, PeerSyncTest, QueryParsingTest, TestFunctionQuery, DocumentBuilderTest, LeaderElectionTest, RemoteQueryErrorTest, ExternalFileFieldSortTest, BasicDistributedZkTest, AnalysisAfterCoreReloadTest, TestPerFieldSimilarity, DOMUtilTest, TestConfigSets, DistributedTermsComponentTest, TestCharFilters, AutoCommitTest, EnumFieldTest, TestNoOpRegenerator, SolrInfoMBeanTest, TestLMJelinekMercerSimilarityFactory, CSVRequestHandlerTest, TestCollapseQParserPlugin, TestReversedWildcardFilterFactory, ZkControllerTest, DistributedQueryElevationComponentTest, HdfsDirectoryTest, TestCursorMarkWithoutUniqueKey, LoggingHandlerTest, TestSolrXMLSerializer, TestInitQParser, SolrCoreTest, CopyFieldTest, TestSchemaVersionResource, FileBasedSpellCheckerTest, CircularListTest, DirectUpdateHandlerOptimizeTest, TestSimpleQParserPlugin, TestHighlightDedupGrouping, PreAnalyzedUpdateProcessorTest, TermsComponentTest, TestStressUserVersions, QueryElevationComponentTest, PolyFieldTest, ClusterStateTest, HdfsChaosMonkeySafeLeaderTest, MoreLikeThisHandlerTest, ResponseHeaderTest, SyncSliceTest]
[junit4] Completed on J0 in 55.51s, 1 test, 1 error <<< FAILURES!
[...truncated 696 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1655 tests, 1 error, 37 ignored (16 assumptions)
Total time: 75 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0_20-ea-b15 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0_05) - Build # 10466
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/10466/
Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([4875109FBE28591B]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Captured an uncaught exception in thread: Thread[id=1948, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1948, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Caused by: java.lang.AssertionError: liveDocs.count()=56 info.docCount=95 info.getDelCount()=41
at __randomizedtesting.SeedInfo.seed([4875109FBE28591B]:0)
at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:237)
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:98)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:394)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:273)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1471)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1672)
at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([4875109FBE28591B]:0)
Build Log:
[...truncated 12023 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/init-core-data-001
[junit4] 2> 419794 T1560 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 419795 T1560 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ge_/qt
[junit4] 2> 419797 T1560 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 419798 T1560 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 419799 T1561 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 419898 T1560 oasc.ZkTestServer.run start zk server on port:44708
[junit4] 2> 419899 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 420408 T1567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2229ff28 name:ZooKeeperConnection Watcher:127.0.0.1:44708 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 420408 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 420408 T1560 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 420412 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 420413 T1569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d440c name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 420413 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 420413 T1560 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 420415 T1560 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 420417 T1560 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 420418 T1560 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 420420 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 420420 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 420422 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 420423 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 420424 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 420425 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 420426 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 420426 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 420428 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 420428 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 420430 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 420430 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 420433 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 420433 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 420436 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 420436 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 420437 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 420437 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 420439 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 420439 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 420441 T1560 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 420441 T1560 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 420678 T1560 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 420681 T1560 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46995
[junit4] 2> 420681 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 420682 T1560 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 420682 T1560 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002
[junit4] 2> 420682 T1560 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/'
[junit4] 2> 420692 T1560 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/solr.xml
[junit4] 2> 420699 T1560 oasc.CoreContainer.<init> New CoreContainer 440830218
[junit4] 2> 420699 T1560 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/]
[junit4] 2> 420700 T1560 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 420700 T1560 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 420700 T1560 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 420700 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 420700 T1560 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 420701 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 420701 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 420701 T1560 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 420701 T1560 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 420702 T1560 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 420702 T1560 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 420702 T1560 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 420702 T1560 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44708/solr
[junit4] 2> 420703 T1560 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 420703 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 420704 T1580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9932e4c name:ZooKeeperConnection Watcher:127.0.0.1:44708 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 420705 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 420706 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 420707 T1582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a418eb5 name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 420707 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 420708 T1560 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 420710 T1560 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 420712 T1560 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 420713 T1560 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 420715 T1560 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 420716 T1560 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 420717 T1560 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46995_ge_%2Fqt
[junit4] 2> 420718 T1560 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46995_ge_%2Fqt
[junit4] 2> 420719 T1560 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 420720 T1560 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 420725 T1560 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46995_ge_%2Fqt
[junit4] 2> 420725 T1560 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 420727 T1560 oasc.Overseer.start Overseer (id=91948232948776963-127.0.0.1:46995_ge_%2Fqt-n_0000000000) starting
[junit4] 2> 420732 T1560 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 420738 T1584 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 420738 T1560 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 420739 T1560 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 420740 T1560 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 420741 T1583 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 420743 T1585 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 420743 T1585 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 420744 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 420744 T1585 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 420745 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 420745 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:46995/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:46995_ge_%2Fqt",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 420745 T1583 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 420745 T1583 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 420747 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 420747 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 421744 T1585 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 421744 T1585 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 421745 T1585 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 421745 T1585 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 421746 T1585 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 421746 T1585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/collection1/'
[junit4] 2> 421746 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 421747 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 421747 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 421765 T1585 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 421773 T1585 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 421774 T1585 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 421779 T1585 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 421859 T1585 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 421860 T1585 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 421861 T1585 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 421863 T1585 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 421874 T1585 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 421875 T1585 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 421875 T1585 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/
[junit4] 2> 421875 T1585 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@638b8ed6
[junit4] 2> 421875 T1585 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data
[junit4] 2> 421876 T1585 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/index/
[junit4] 2> 421876 T1585 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 421876 T1585 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/index
[junit4] 2> 421876 T1585 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=7859317, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 421880 T1585 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 421880 T1585 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 421882 T1585 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 421882 T1585 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 421882 T1585 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 421882 T1585 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 421883 T1585 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 421883 T1585 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 421883 T1585 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 421883 T1585 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 421883 T1585 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 421883 T1585 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 421884 T1585 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 421884 T1585 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 421884 T1585 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 421884 T1585 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 421885 T1585 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 421885 T1585 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 421887 T1585 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 421888 T1585 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 421888 T1585 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 421889 T1585 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 421890 T1585 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 421890 T1585 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 421890 T1585 oass.SolrIndexSearcher.<init> Opening Searcher@660b0109[collection1] main
[junit4] 2> 421890 T1585 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 421891 T1585 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 421891 T1585 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 421891 T1585 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 421891 T1585 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 421892 T1585 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 421892 T1585 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 421892 T1585 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 421892 T1585 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 421892 T1585 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 421894 T1586 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@660b0109[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 421894 T1585 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 421895 T1589 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46995/ge_/qt collection:control_collection shard:shard1
[junit4] 2> 421895 T1560 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 421896 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 421896 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 421897 T1589 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 421897 T1591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72ff84a0 name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 421897 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 421898 T1560 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 421900 T1560 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 421900 T1589 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 421903 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 421903 T1589 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 421903 T1589 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C264 name=collection1 org.apache.solr.core.SolrCore@68cfe807 url=http://127.0.0.1:46995/ge_/qt/collection1 node=127.0.0.1:46995_ge_%2Fqt C264_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:46995/ge_/qt, core=collection1, node_name=127.0.0.1:46995_ge_%2Fqt}
[junit4] 2> 421903 T1589 C264 P46995 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46995/ge_/qt/collection1/
[junit4] 2> 421903 T1589 C264 P46995 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 421903 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 421903 T1589 C264 P46995 oasc.SyncStrategy.syncToMe http://127.0.0.1:46995/ge_/qt/collection1/ has no replicas
[junit4] 2> 421904 T1589 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46995/ge_/qt/collection1/ shard1
[junit4] 2> 421904 T1589 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 421906 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 421908 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 421910 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 422012 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 422012 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 422059 T1589 oasc.ZkController.register We are http://127.0.0.1:46995/ge_/qt/collection1/ and leader is http://127.0.0.1:46995/ge_/qt/collection1/
[junit4] 2> 422059 T1589 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46995/ge_/qt
[junit4] 2> 422060 T1589 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 422060 T1589 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 422060 T1589 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 422061 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 422061 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 422061 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 422061 T1589 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 422062 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 422063 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:46995/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:46995_ge_%2Fqt",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 422064 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 422132 T1560 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 422132 T1560 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 422134 T1560 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45557
[junit4] 2> 422135 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 422135 T1560 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 422135 T1560 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003
[junit4] 2> 422135 T1560 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/'
[junit4] 2> 422146 T1560 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/solr.xml
[junit4] 2> 422153 T1560 oasc.CoreContainer.<init> New CoreContainer 115558110
[junit4] 2> 422153 T1560 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/]
[junit4] 2> 422153 T1560 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 422154 T1560 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 422154 T1560 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 422154 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 422154 T1560 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 422155 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 422155 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 422155 T1560 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 422155 T1560 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 422156 T1560 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 422156 T1560 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 422156 T1560 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 422156 T1560 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44708/solr
[junit4] 2> 422157 T1560 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 422157 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 422158 T1602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ec7b791 name:ZooKeeperConnection Watcher:127.0.0.1:44708 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 422159 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 422161 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 422161 T1604 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5212ae46 name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 422162 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 422165 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 422165 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 422166 T1560 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 423168 T1560 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45557_ge_%2Fqt
[junit4] 2> 423171 T1560 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45557_ge_%2Fqt
[junit4] 2> 423174 T1591 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 423174 T1582 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 423175 T1604 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 423181 T1605 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 423181 T1605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 423182 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 423182 T1605 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 423182 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 423183 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 423184 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 423185 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45557/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:45557_ge_%2Fqt",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 423186 T1583 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 423186 T1583 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 423194 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 423296 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 423296 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 423296 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424183 T1605 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 424183 T1605 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 424184 T1605 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 424185 T1605 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 424186 T1605 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 424186 T1605 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/collection1/'
[junit4] 2> 424188 T1605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 424188 T1605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 424189 T1605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 424211 T1605 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 424220 T1605 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 424222 T1605 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 424227 T1605 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 424304 T1605 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 424305 T1605 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 424306 T1605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 424308 T1605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 424318 T1605 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 424319 T1605 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 424319 T1605 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/
[junit4] 2> 424319 T1605 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@638b8ed6
[junit4] 2> 424319 T1605 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1
[junit4] 2> 424320 T1605 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/index/
[junit4] 2> 424320 T1605 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 424320 T1605 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/index
[junit4] 2> 424321 T1605 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=7859317, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 424325 T1605 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 424325 T1605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 424327 T1605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 424328 T1605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 424328 T1605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 424328 T1605 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 424328 T1605 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 424329 T1605 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 424329 T1605 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 424329 T1605 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 424329 T1605 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 424329 T1605 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 424330 T1605 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 424331 T1605 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 424332 T1605 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 424332 T1605 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 424332 T1605 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 424333 T1605 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 424333 T1605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 424334 T1605 oass.SolrIndexSearcher.<init> Opening Searcher@6f972916[collection1] main
[junit4] 2> 424334 T1605 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 424334 T1605 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 424335 T1605 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 424335 T1605 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 424335 T1605 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 424335 T1605 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 424336 T1605 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 424336 T1605 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 424336 T1605 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 424336 T1605 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 424337 T1606 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f972916[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 424338 T1605 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 424339 T1609 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45557/ge_/qt collection:collection1 shard:shard2
[junit4] 2> 424339 T1560 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 424340 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 424340 T1609 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 424343 T1609 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 424344 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424344 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424344 T1609 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 424345 T1609 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 424345 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C265 name=collection1 org.apache.solr.core.SolrCore@1fabef6b url=http://127.0.0.1:45557/ge_/qt/collection1 node=127.0.0.1:45557_ge_%2Fqt C265_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:45557/ge_/qt, core=collection1, node_name=127.0.0.1:45557_ge_%2Fqt}
[junit4] 2> 424345 T1609 C265 P45557 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45557/ge_/qt/collection1/
[junit4] 2> 424345 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 424345 T1609 C265 P45557 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 424345 T1609 C265 P45557 oasc.SyncStrategy.syncToMe http://127.0.0.1:45557/ge_/qt/collection1/ has no replicas
[junit4] 2> 424346 T1609 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45557/ge_/qt/collection1/ shard2
[junit4] 2> 424346 T1609 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 424346 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424348 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424349 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424454 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424454 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424454 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424499 T1609 oasc.ZkController.register We are http://127.0.0.1:45557/ge_/qt/collection1/ and leader is http://127.0.0.1:45557/ge_/qt/collection1/
[junit4] 2> 424499 T1609 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45557/ge_/qt
[junit4] 2> 424499 T1609 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 424499 T1609 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 424499 T1609 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 424500 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424501 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424501 T1609 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 424501 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424502 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 424502 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:45557/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:45557_ge_%2Fqt",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 424506 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 424573 T1560 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 424573 T1560 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 424575 T1560 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34405
[junit4] 2> 424576 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 424576 T1560 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 424576 T1560 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004
[junit4] 2> 424576 T1560 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/'
[junit4] 2> 424586 T1560 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/solr.xml
[junit4] 2> 424592 T1560 oasc.CoreContainer.<init> New CoreContainer 260023556
[junit4] 2> 424592 T1560 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/]
[junit4] 2> 424593 T1560 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 424593 T1560 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 424593 T1560 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 424593 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 424594 T1560 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 424594 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 424594 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 424594 T1560 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 424594 T1560 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 424595 T1560 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 424595 T1560 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 424595 T1560 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 424596 T1560 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44708/solr
[junit4] 2> 424596 T1560 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 424596 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 424598 T1620 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c0f917 name:ZooKeeperConnection Watcher:127.0.0.1:44708 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 424599 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 424600 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 424601 T1622 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51ec17e7 name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 424601 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 424605 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424605 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424605 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 424605 T1560 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 425607 T1560 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34405_ge_%2Fqt
[junit4] 2> 425609 T1560 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34405_ge_%2Fqt
[junit4] 2> 425611 T1604 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 425611 T1582 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 425611 T1622 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 425611 T1591 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 425614 T1623 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 425614 T1623 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 425615 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 425615 T1623 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 425615 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 425616 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 425616 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 425617 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34405/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34405_ge_%2Fqt",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 425617 T1583 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 425617 T1583 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 425618 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 425719 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 425719 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 425719 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 425719 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 426616 T1623 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 426616 T1623 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 426617 T1623 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 426617 T1623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 426619 T1623 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 426619 T1623 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/collection1/'
[junit4] 2> 426620 T1623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 426621 T1623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 426621 T1623 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 426646 T1623 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 426659 T1623 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 426661 T1623 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 426664 T1623 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 426739 T1623 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 426740 T1623 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 426740 T1623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 426743 T1623 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 426755 T1623 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 426755 T1623 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 426755 T1623 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/
[junit4] 2> 426755 T1623 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@638b8ed6
[junit4] 2> 426756 T1623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2
[junit4] 2> 426756 T1623 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/index/
[junit4] 2> 426756 T1623 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 426756 T1623 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/index
[junit4] 2> 426757 T1623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=7859317, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 426761 T1623 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 426761 T1623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 426764 T1623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 426764 T1623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 426765 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 426765 T1623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 426765 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 426765 T1623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 426766 T1623 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 426766 T1623 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 426767 T1623 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 426767 T1623 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 426768 T1623 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 426768 T1623 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 426768 T1623 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 426768 T1623 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 426769 T1623 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 426769 T1623 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 426771 T1623 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 426773 T1623 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 426773 T1623 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 426774 T1623 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 426775 T1623 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 426775 T1623 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 426775 T1623 oass.SolrIndexSearcher.<init> Opening Searcher@39a9ff5f[collection1] main
[junit4] 2> 426776 T1623 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 426777 T1623 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 426777 T1623 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 426778 T1623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 426778 T1623 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 426778 T1623 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 426779 T1623 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 426779 T1623 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 426780 T1623 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 426780 T1623 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 426782 T1624 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39a9ff5f[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 426783 T1623 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 426784 T1627 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:34405/ge_/qt collection:collection1 shard:shard3
[junit4] 2> 426784 T1560 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4] 2> 426785 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 426785 T1627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 426789 T1627 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 426790 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426790 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426790 T1627 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 426790 T1627 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C266 name=collection1 org.apache.solr.core.SolrCore@46cf07d6 url=http://127.0.0.1:34405/ge_/qt/collection1 node=127.0.0.1:34405_ge_%2Fqt C266_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:34405/ge_/qt, core=collection1, node_name=127.0.0.1:34405_ge_%2Fqt}
[junit4] 2> 426791 T1627 C266 P34405 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:34405/ge_/qt/collection1/
[junit4] 2> 426790 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426791 T1627 C266 P34405 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 426791 T1627 C266 P34405 oasc.SyncStrategy.syncToMe http://127.0.0.1:34405/ge_/qt/collection1/ has no replicas
[junit4] 2> 426791 T1627 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:34405/ge_/qt/collection1/ shard3
[junit4] 2> 426791 T1627 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 426792 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 426793 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426796 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426897 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 426897 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 426897 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 426897 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 426945 T1627 oasc.ZkController.register We are http://127.0.0.1:34405/ge_/qt/collection1/ and leader is http://127.0.0.1:34405/ge_/qt/collection1/
[junit4] 2> 426945 T1627 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:34405/ge_/qt
[junit4] 2> 426945 T1627 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 426945 T1627 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 426945 T1627 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 426946 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426947 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426947 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 426947 T1627 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 426948 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 426948 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:34405/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:34405_ge_%2Fqt",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 426950 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 427056 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 427057 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 427056 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 427056 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 427061 T1560 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 427062 T1560 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 427064 T1560 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55490
[junit4] 2> 427065 T1560 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 427066 T1560 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 427066 T1560 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005
[junit4] 2> 427067 T1560 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/'
[junit4] 2> 427082 T1560 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/solr.xml
[junit4] 2> 427097 T1560 oasc.CoreContainer.<init> New CoreContainer 1531605114
[junit4] 2> 427097 T1560 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/]
[junit4] 2> 427098 T1560 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 427098 T1560 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 427099 T1560 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 427099 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 427099 T1560 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 427099 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 427099 T1560 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 427100 T1560 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 427100 T1560 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 427100 T1560 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 427101 T1560 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 427101 T1560 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 427101 T1560 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:44708/solr
[junit4] 2> 427101 T1560 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 427102 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 427103 T1638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a1bd746 name:ZooKeeperConnection Watcher:127.0.0.1:44708 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 427104 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 427106 T1560 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 427106 T1640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@799dbce name:ZooKeeperConnection Watcher:127.0.0.1:44708/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 427107 T1560 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 427111 T1560 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 428112 T1560 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55490_ge_%2Fqt
[junit4] 2> 428113 T1560 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55490_ge_%2Fqt
[junit4] 2> 428116 T1591 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 428116 T1604 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 428116 T1582 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 428116 T1640 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 428116 T1622 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 428120 T1641 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 428120 T1641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 428121 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 428121 T1641 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 428121 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 428121 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 428122 T1583 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 428123 T1583 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:55490/ge_/qt",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:55490_ge_%2Fqt",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 428123 T1583 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 428123 T1583 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 428125 T1582 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 428228 T1591 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 428228 T1582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 428228 T1622 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 428228 T1640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 428228 T1604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 429121 T1641 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 429122 T1641 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 429122 T1641 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 429123 T1641 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 429124 T1641 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 429124 T1641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/collection1/'
[junit4] 2> 429124 T1641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/collection1/lib/.svn/' to classloader
[junit4] 2> 429125 T1641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/collection1/lib/classes/' to classloader
[junit4] 2> 429125 T1641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/solr.cloud.ChaosMonkeySafeLeaderTest-4875109FBE28591B-001/tempDir-005/collection1/lib/README' to classloader
[junit4] 2> 429145 T1641 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
[junit4] 2> 429152 T1641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 429153 T1641 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 429156 T1641 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 429235 T1641 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 429236 T1641 oass.IndexSchema.readSchema unique
[...truncated too long message...]
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 67) Thread[id=2831, name=HashSessionScavenger-79, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> 68) Thread[id=2646, name=qtp1006319612-2646 Acceptor0 SelectChannelConnector@127.0.0.1:60093, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
[junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 69) Thread[id=2731, name=coreZkRegister-1413-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 70) Thread[id=2833, name=qtp1840590025-2833 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:317)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
[junit4] 2> at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
[junit4] 2> at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
[junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:78)
[junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 71) Thread[id=2527, name=HashSessionScavenger-66, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] 2> at java.lang.Object.wait(Native Method)
[junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=fr_BE, timezone=Pacific/Majuro
[junit4] 2> NOTE: Linux 3.8.0-41-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=88,free=123083512,total=242745344
[junit4] 2> NOTE: All tests run in this JVM: [XsltUpdateRequestHandlerTest, TestCollationFieldDocValues, ConnectionManagerTest, MigrateRouteKeyTest, XmlUpdateRequestHandlerTest, JSONWriterTest, TestLazyCores, TestStressReorder, OutputWriterTest, TestManagedResourceStorage, TestAtomicUpdateErrorCases, CursorPagingTest, TestManagedResource, HdfsUnloadDistributedZkTest, TestChildDocTransformer, TestSolrXmlPersistor, NotRequiredUniqueKeyTest, CursorMarkTest, TestRandomFaceting, TestSolrDeletionPolicy2, TestSolrCoreProperties, HdfsBasicDistributedZkTest, HighlighterTest, SolrPluginUtilsTest, TestOverriddenPrefixQueryForCustomFieldType, TestCoreDiscovery, TestComponentsName, TestManagedSynonymFilterFactory, TestHashPartitioner, TestNumberUtils, TestMultiCoreConfBootstrap, TestInfoStreamLogging, LeaderElectionIntegrationTest, HttpPartitionTest, CurrencyFieldOpenExchangeTest, SolrXmlInZkTest, TestQuerySenderNoQuery, ShardRoutingCustomTest, TestDynamicFieldResource, HdfsBasicDistributedZk2Test, ParsingFieldUpdateProcessorsTest, TestSearcherReuse, TestIBSimilarityFactory, MergeStrategyTest, TestJmxIntegration, TestXIncludeConfig, SpellCheckCollatorTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeySafeLeaderTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=4875109FBE28591B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_BE -Dtests.timezone=Pacific/Majuro -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([4875109FBE28591B]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1948, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4] > Caused by: java.lang.AssertionError: liveDocs.count()=56 info.docCount=95 info.getDelCount()=41
[junit4] > at __randomizedtesting.SeedInfo.seed([4875109FBE28591B]:0)
[junit4] > at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
[junit4] > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:109)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
[junit4] > at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:237)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:98)
[junit4] > at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:394)
[junit4] > at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:273)
[junit4] > at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
[junit4] > at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1471)
[junit4] > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1672)
[junit4] > at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
[junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
[junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
[junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
[junit4] Completed on J1 in 7225.66s, 1 test, 3 errors <<< FAILURES!
[...truncated 166 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1297: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:921: There were test failures: 405 suites, 1655 tests, 2 suite-level errors, 1 error, 177 ignored (16 assumptions)
Total time: 164 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any