You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/05/23 20:12:04 UTC

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 558 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/558/

No tests ran.

Build Log:
[...truncated 11637 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/init-core-data-001
   [junit4]   2> 805340 T5762 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /vuwma/e
   [junit4]   2> 805346 T5762 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 805347 T5762 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 805348 T5763 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 805449 T5762 oasc.ZkTestServer.run start zk server on port:32437
   [junit4]   2> 805450 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 805454 T5769 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fc6874e name:ZooKeeperConnection Watcher:127.0.0.1:32437 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 805454 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 805455 T5762 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 805466 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 805468 T5771 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@311e9fdf name:ZooKeeperConnection Watcher:127.0.0.1:32437/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 805468 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 805468 T5762 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 805477 T5762 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 805479 T5762 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 805481 T5762 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 805489 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 805490 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 805500 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 805500 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 805604 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 805605 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 805607 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 805608 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 805615 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 805616 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 805618 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 805619 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 805621 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 805622 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 805629 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 805630 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 805632 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 805633 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 805635 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 805635 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 805638 T5762 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 805638 T5762 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 806664 T5762 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 806668 T5762 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32443
   [junit4]   2> 806669 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 806669 T5762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 806669 T5762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002
   [junit4]   2> 806669 T5762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/'
   [junit4]   2> 806698 T5762 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/solr.xml
   [junit4]   2> 806766 T5762 oasc.CoreContainer.<init> New CoreContainer 1079484816
   [junit4]   2> 806766 T5762 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/]
   [junit4]   2> 806767 T5762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 806768 T5762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 806768 T5762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 806768 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 806769 T5762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 806769 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 806769 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 806769 T5762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 806770 T5762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 806770 T5762 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 806770 T5762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 806771 T5762 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 806771 T5762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32437/solr
   [junit4]   2> 806771 T5762 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 806772 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 806803 T5782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@393015f0 name:ZooKeeperConnection Watcher:127.0.0.1:32437 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 806804 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 806818 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 806820 T5784 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@600fd589 name:ZooKeeperConnection Watcher:127.0.0.1:32437/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 806820 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 806822 T5762 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 806832 T5762 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 806841 T5762 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 806844 T5762 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 806854 T5762 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 806857 T5762 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 806858 T5762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32442_vuwma%2Fe
   [junit4]   2> 806859 T5762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32442_vuwma%2Fe
   [junit4]   2> 806861 T5762 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 806863 T5762 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 806873 T5762 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:32442_vuwma%2Fe
   [junit4]   2> 806873 T5762 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 806875 T5762 oasc.Overseer.start Overseer (id=91796146359959555-127.0.0.1:32442_vuwma%2Fe-n_0000000000) starting
   [junit4]   2> 806885 T5762 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 806893 T5786 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 806894 T5762 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 806895 T5762 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 806897 T5762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 806899 T5785 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 806904 T5787 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 806904 T5787 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 806905 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 806905 T5787 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 806906 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 806906 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32442/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32442_vuwma%2Fe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 806907 T5785 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 806907 T5785 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 806916 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 806916 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 807907 T5787 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 807907 T5787 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 807908 T5787 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 807908 T5787 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 807909 T5787 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 807909 T5787 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/collection1/'
   [junit4]   2> 807911 T5787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 807912 T5787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 807912 T5787 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 808006 T5787 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 808092 T5787 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 808193 T5787 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 808201 T5787 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 808734 T5787 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 808735 T5787 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 808735 T5787 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 808747 T5787 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 808750 T5787 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 808759 T5787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 808763 T5787 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 808766 T5787 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 808766 T5787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 808767 T5787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 808767 T5787 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 808767 T5787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 808767 T5787 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 808768 T5787 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 808768 T5787 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 808768 T5787 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/control/data/
   [junit4]   2> 808768 T5787 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@187e7188
   [junit4]   2> 808769 T5787 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/control/data
   [junit4]   2> 808769 T5787 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/control/data/index/
   [junit4]   2> 808769 T5787 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 808770 T5787 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/control/data/index
   [junit4]   2> 808770 T5787 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=27.5830078125, floorSegmentMB=1.884765625, forceMergeDeletesPctAllowed=28.79996567911605, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4321915225327543
   [junit4]   2> 808771 T5787 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7b53be51 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16f185aa),segFN=segments_1,generation=1}
   [junit4]   2> 808771 T5787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 808774 T5787 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 808774 T5787 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 808774 T5787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 808775 T5787 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 808775 T5787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 808775 T5787 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 808775 T5787 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 808776 T5787 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 808776 T5787 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 808776 T5787 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 808777 T5787 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 808777 T5787 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 808777 T5787 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 808777 T5787 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 808778 T5787 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 808778 T5787 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 808795 T5787 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 808801 T5787 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 808801 T5787 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 808801 T5787 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1248039314, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 808802 T5787 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7b53be51 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16f185aa),segFN=segments_1,generation=1}
   [junit4]   2> 808802 T5787 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 808802 T5787 oass.SolrIndexSearcher.<init> Opening Searcher@6c7f4172[collection1] main
   [junit4]   2> 808803 T5787 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 808803 T5787 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 808803 T5787 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 808804 T5787 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 808804 T5787 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 808804 T5787 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 808804 T5787 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 808805 T5787 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 808805 T5787 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 808805 T5787 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 808807 T5788 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6c7f4172[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 808809 T5787 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 808809 T5791 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32442/vuwma/e collection:control_collection shard:shard1
   [junit4]   2> 808810 T5762 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 808810 T5791 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 808810 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 808813 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 808828 T5794 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@711f1c41 name:ZooKeeperConnection Watcher:127.0.0.1:32437/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 808828 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 808830 T5762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 808830 T5791 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 808841 T5762 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 808842 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808842 T5791 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 808842 T5791 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C524 name=collection1 org.apache.solr.core.SolrCore@380d844e url=http://127.0.0.1:32442/vuwma/e/collection1 node=127.0.0.1:32442_vuwma%2Fe C524_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:32442/vuwma/e, core=collection1, node_name=127.0.0.1:32442_vuwma%2Fe}
   [junit4]   2> 808843 T5791 C524 P32442 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32442/vuwma/e/collection1/
   [junit4]   2> 808843 T5791 C524 P32442 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 808843 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 808843 T5791 C524 P32442 oasc.SyncStrategy.syncToMe http://127.0.0.1:32442/vuwma/e/collection1/ has no replicas
   [junit4]   2> 808843 T5791 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32442/vuwma/e/collection1/ shard1
   [junit4]   2> 808843 T5791 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 808850 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808851 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 808851 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 808871 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808871 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 808881 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 808992 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 808992 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 809024 T5791 oasc.ZkController.register We are http://127.0.0.1:32442/vuwma/e/collection1/ and leader is http://127.0.0.1:32442/vuwma/e/collection1/
   [junit4]   2> 809024 T5791 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32442/vuwma/e
   [junit4]   2> 809024 T5791 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 809024 T5791 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 809024 T5791 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 809026 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 809026 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 809026 T5791 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 809026 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 809027 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 809027 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32442/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32442_vuwma%2Fe",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 809030 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 809135 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 809135 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 809638 T5762 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 809639 T5762 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 809643 T5762 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32450
   [junit4]   2> 809643 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 809644 T5762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 809644 T5762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003
   [junit4]   2> 809644 T5762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/'
   [junit4]   2> 809680 T5762 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/solr.xml
   [junit4]   2> 809751 T5762 oasc.CoreContainer.<init> New CoreContainer 131457178
   [junit4]   2> 809751 T5762 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/]
   [junit4]   2> 809752 T5762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 809753 T5762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 809753 T5762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 809753 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 809753 T5762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 809754 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 809754 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 809754 T5762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 809755 T5762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 809755 T5762 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 809755 T5762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 809756 T5762 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 809756 T5762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32437/solr
   [junit4]   2> 809756 T5762 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 809757 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 809759 T5805 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a8ca27a name:ZooKeeperConnection Watcher:127.0.0.1:32437 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 809760 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 809804 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 809806 T5807 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d518dd4 name:ZooKeeperConnection Watcher:127.0.0.1:32437/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 809806 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 809875 T5762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 810878 T5762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32449_vuwma%2Fe
   [junit4]   2> 810880 T5762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32449_vuwma%2Fe
   [junit4]   2> 810883 T5807 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 810883 T5784 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 810883 T5794 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 810903 T5808 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 810904 T5808 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 810905 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 810905 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 810905 T5808 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 810905 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 810906 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 810907 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32449/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32449_vuwma%2Fe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 810907 T5785 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 810907 T5785 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 810916 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 810917 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 810917 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 810917 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 811907 T5808 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 811907 T5808 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 811908 T5808 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 811908 T5808 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 811909 T5808 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 811909 T5808 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/collection1/'
   [junit4]   2> 811910 T5808 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 811911 T5808 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 811912 T5808 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 811990 T5808 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 812066 T5808 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 812167 T5808 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 812178 T5808 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 812881 T5808 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 812882 T5808 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 812883 T5808 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 812900 T5808 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 812905 T5808 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 812918 T5808 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 812923 T5808 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 812927 T5808 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 812928 T5808 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 812928 T5808 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 812929 T5808 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 812929 T5808 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 812930 T5808 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 812930 T5808 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 812930 T5808 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 812931 T5808 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty1/
   [junit4]   2> 812931 T5808 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@187e7188
   [junit4]   2> 812932 T5808 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty1
   [junit4]   2> 812932 T5808 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty1/index/
   [junit4]   2> 812932 T5808 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 812933 T5808 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty1/index
   [junit4]   2> 812933 T5808 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=27.5830078125, floorSegmentMB=1.884765625, forceMergeDeletesPctAllowed=28.79996567911605, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4321915225327543
   [junit4]   2> 812934 T5808 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75ad6aef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d9c25a2),segFN=segments_1,generation=1}
   [junit4]   2> 812935 T5808 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 812940 T5808 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 812941 T5808 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 812941 T5808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 812941 T5808 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 812941 T5808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 812942 T5808 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 812942 T5808 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 812943 T5808 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 812943 T5808 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 812944 T5808 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 812944 T5808 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 812944 T5808 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 812945 T5808 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 812945 T5808 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 812946 T5808 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 812947 T5808 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 812973 T5808 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 812980 T5808 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 812980 T5808 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 812982 T5808 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1248039314, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 812982 T5808 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75ad6aef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d9c25a2),segFN=segments_1,generation=1}
   [junit4]   2> 812983 T5808 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 812983 T5808 oass.SolrIndexSearcher.<init> Opening Searcher@63e38f77[collection1] main
   [junit4]   2> 812983 T5808 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 812984 T5808 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 812985 T5808 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 812985 T5808 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 812985 T5808 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 812986 T5808 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 812986 T5808 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 812986 T5808 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 812987 T5808 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 812987 T5808 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 812992 T5809 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@63e38f77[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 812995 T5808 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 812996 T5812 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32449/vuwma/e collection:collection1 shard:shard2
   [junit4]   2> 812996 T5762 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 812997 T5812 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 812997 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 813047 T5812 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 813050 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813050 T5812 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 813050 T5812 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C525 name=collection1 org.apache.solr.core.SolrCore@3ce758f9 url=http://127.0.0.1:32449/vuwma/e/collection1 node=127.0.0.1:32449_vuwma%2Fe C525_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:32449/vuwma/e, core=collection1, node_name=127.0.0.1:32449_vuwma%2Fe}
   [junit4]   2> 813050 T5812 C525 P32449 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32449/vuwma/e/collection1/
   [junit4]   2> 813051 T5812 C525 P32449 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 813051 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 813051 T5812 C525 P32449 oasc.SyncStrategy.syncToMe http://127.0.0.1:32449/vuwma/e/collection1/ has no replicas
   [junit4]   2> 813051 T5812 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32449/vuwma/e/collection1/ shard2
   [junit4]   2> 813052 T5812 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 813054 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813055 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813055 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813055 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813059 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813060 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 813062 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813164 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813164 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813164 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813212 T5812 oasc.ZkController.register We are http://127.0.0.1:32449/vuwma/e/collection1/ and leader is http://127.0.0.1:32449/vuwma/e/collection1/
   [junit4]   2> 813212 T5812 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32449/vuwma/e
   [junit4]   2> 813212 T5812 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 813212 T5812 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 813212 T5812 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 813215 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813215 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813215 T5812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 813215 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813216 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 813217 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32449/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32449_vuwma%2Fe",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 813226 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 813328 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813328 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 813328 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 814379 T5762 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 814380 T5762 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 814382 T5762 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:32454
   [junit4]   2> 814383 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 814383 T5762 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 814383 T5762 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004
   [junit4]   2> 814384 T5762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/'
   [junit4]   2> 814423 T5762 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/solr.xml
   [junit4]   2> 814515 T5762 oasc.CoreContainer.<init> New CoreContainer 320178115
   [junit4]   2> 814515 T5762 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/]
   [junit4]   2> 814517 T5762 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 814517 T5762 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 814517 T5762 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 814518 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 814518 T5762 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 814518 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 814518 T5762 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 814519 T5762 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 814519 T5762 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 814519 T5762 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 814520 T5762 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 814520 T5762 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 814520 T5762 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:32437/solr
   [junit4]   2> 814521 T5762 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 814521 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 814525 T5824 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a841cfe name:ZooKeeperConnection Watcher:127.0.0.1:32437 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 814526 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 814528 T5762 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 814542 T5826 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74556799 name:ZooKeeperConnection Watcher:127.0.0.1:32437/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 814542 T5762 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 814558 T5762 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 815560 T5762 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32453_vuwma%2Fe
   [junit4]   2> 815588 T5762 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32453_vuwma%2Fe
   [junit4]   2> 815592 T5807 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 815592 T5794 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 815592 T5784 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 815592 T5826 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 815606 T5827 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 815607 T5827 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 815608 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 815608 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 815608 T5827 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 815608 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 815609 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 815610 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32453/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32453_vuwma%2Fe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 815610 T5785 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 815610 T5785 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 815619 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 815620 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 815620 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 815620 T5826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 815620 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 816609 T5827 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 816609 T5827 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 816610 T5827 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 816610 T5827 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 816611 T5827 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 816611 T5827 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/collection1/'
   [junit4]   2> 816613 T5827 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 816613 T5827 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 816614 T5827 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 816697 T5827 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 816776 T5827 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 816877 T5827 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 816885 T5827 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 817373 T5827 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 817374 T5827 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 817375 T5827 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 817391 T5827 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 817395 T5827 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 817408 T5827 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 817414 T5827 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 817419 T5827 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 817420 T5827 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 817420 T5827 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 817420 T5827 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 817421 T5827 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 817421 T5827 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 817422 T5827 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 817422 T5827 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 817422 T5827 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty2/
   [junit4]   2> 817422 T5827 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@187e7188
   [junit4]   2> 817424 T5827 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty2
   [junit4]   2> 817424 T5827 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty2/index/
   [junit4]   2> 817425 T5827 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 817425 T5827 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-E37F9276EC37BB4-001/tempDir-001/jetty2/index
   [junit4]   2> 817425 T5827 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=12, maxMergedSegmentMB=27.5830078125, floorSegmentMB=1.884765625, forceMergeDeletesPctAllowed=28.79996567911605, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4321915225327543
   [junit4]   2> 817427 T5827 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7e306ed7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3c90cf06),segFN=segments_1,generation=1}
   [junit4]   2> 817427 T5827 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 817432 T5827 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 817433 T5827 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 817433 T5827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 817433 T5827 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 817434 T5827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 817434 T5827 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 817434 T5827 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 817435 T5827 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 817435 T5827 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 817436 T5827 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 817436 T5827 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 817437 T5827 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 817437 T5827 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 817438 T5827 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 817438 T5827 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 817439 T5827 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 817460 T5827 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 817466 T5827 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 817467 T5827 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 817468 T5827 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1248039314, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 817468 T5827 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7e306ed7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3c90cf06),segFN=segments_1,generation=1}
   [junit4]   2> 817469 T5827 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 817469 T5827 oass.SolrIndexSearcher.<init> Opening Searcher@148da147[collection1] main
   [junit4]   2> 817469 T5827 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 817470 T5827 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 817471 T5827 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 817471 T5827 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 817471 T5827 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 817472 T5827 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 817472 T5827 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 817472 T5827 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 817473 T5827 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 817473 T5827 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 817477 T5828 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@148da147[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 817480 T5827 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 817481 T5831 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32453/vuwma/e collection:collection1 shard:shard1
   [junit4]   2> 817481 T5762 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 817481 T5831 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 817482 T5762 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 817483 T5762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817484 T5762 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 817484 T5762 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 817484 T5762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817500 T5831 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 817502 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817502 T5831 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 817502 T5831 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C526 name=collection1 org.apache.solr.core.SolrCore@226f4822 url=http://127.0.0.1:32453/vuwma/e/collection1 node=127.0.0.1:32453_vuwma%2Fe C526_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:32453/vuwma/e, core=collection1, node_name=127.0.0.1:32453_vuwma%2Fe}
   [junit4]   2> 817502 T5831 C526 P32453 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:32453/vuwma/e/collection1/
   [junit4]   2> 817502 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817503 T5831 C526 P32453 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 817503 T5831 C526 P32453 oasc.SyncStrategy.syncToMe http://127.0.0.1:32453/vuwma/e/collection1/ has no replicas
   [junit4]   2> 817503 T5831 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:32453/vuwma/e/collection1/ shard1
   [junit4]   2> 817504 T5831 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 817505 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817506 T5826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817506 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817506 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817506 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817509 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817511 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817613 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817613 T5826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817613 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817613 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817661 T5831 oasc.ZkController.register We are http://127.0.0.1:32453/vuwma/e/collection1/ and leader is http://127.0.0.1:32453/vuwma/e/collection1/
   [junit4]   2> 817661 T5831 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32453/vuwma/e
   [junit4]   2> 817661 T5831 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 817661 T5831 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 817661 T5831 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 817663 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817664 T5831 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817664 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817664 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817664 T5785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 817665 T5785 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:32453/vuwma/e",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:32453_vuwma%2Fe",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 817674 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 817776 T5794 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817776 T5826 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817776 T5784 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 817776 T5807 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 818500 T5762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 818501 T5762 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
   [junit4]   2> 818502 T5792 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32457,localport=32442], receiveBufferSize:6565888
   [junit4]   2> 818507 T5792 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32443,localport=32458], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C527 name=collection1 org.apache.solr.core.SolrCore@380d844e url=http://127.0.0.1:32442/vuwma/e/collection1 node=127.0.0.1:32442_vuwma%2Fe C527_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:32442/vuwma/e, core=collection1, node_name=127.0.0.1:32442_vuwma%2Fe, leader=true}
   [junit4]   2> 818507 T5774 C527 P32442 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 818509 T5774 C527 P32442 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 818510 T5774 C527 P32442 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 818510 T5774 C527 P32442 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 818510 T5774 C527 P32442 oasup.LogUpdateProcessor.finish [collection1] webapp=/vuwma/e path=/update params={waitSearcher=true&commit=true&version=2&wt=javabin&softCommit=false} {commit=} 0 3
   [junit4]   2> 818512 T5813 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32459,localport=32449], receiveBufferSize:6565888
   [junit4]   2> 818512 T5813 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32450,localport=32460], receiveBufferSize=6565888
   [junit4]   2> 818515 T5832 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32461,localport=32453], receiveBufferSize:6565888
   [junit4]   2> 818516 T5832 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32454,localport=32462], receiveBufferSize=6565888
   [junit4]   2>  C526_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:32453/vuwma/e, core=collection1, node_name=127.0.0.1:32453_vuwma%2Fe, leader=true}
   [junit4]   2> 818516 T5816 C526 P32453 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 818518 T5816 C526 P32453 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 818519 T5816 C526 P32453 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 818519 T5816 C526 P32453 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 818519 T5816 C526 P32453 oasup.LogUpdateProcessor.finish [collection1] webapp=/vuwma/e path=/update params={waitSearcher=true&commit=true&version=2&expungeDeletes=false&openSearcher=true&wt=javabin&softCommit=false&update.distrib=FROMLEADER&commit_end_point=true&distrib.from=http://127.0.0.1:32449/vuwma/e/collection1/} {commit=} 0 3
   [junit4]   2> 818620 T5813 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32463,localport=32449], receiveBufferSize:6565888
   [junit4]   2> 818621 T5813 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32450,localport=32464], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C528 name=collection1 org.apache.solr.core.SolrCore@3ce758f9 url=http://127.0.0.1:32449/vuwma/e/collection1 node=127.0.0.1:32449_vuwma%2Fe C528_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:32449/vuwma/e, core=collection1, node_name=127.0.0.1:32449_vuwma%2Fe, leader=true}
   [junit4]   2> 818622 T5798 C528 P32449 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 818623 T5798 C528 P32449 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 818624 T5798 C528 P32449 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 818624 T5798 C528 P32449 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 818624 T5798 C528 P32449 oasup.LogUpdateProcessor.finish [collection1] webapp=/vuwma/e path=/update params={waitSearcher=true&commit=true&version=2&expungeDeletes=false&openSearcher=true&wt=javabin&softCommit=false&update.distrib=FROMLEADER&commit_end_point=true&distrib.from=http://127.0.0.1:32449/vuwma/e/collection1/} {commit=} 0 2
   [junit4]   2> 818725 T5797 C528 P32449 oasup.LogUpdateProcessor.finish [collection1] webapp=/vuwma/e path=/update params={waitSearcher=true&commit=true&version=2&wt=javabin&softCommit=false} {commit=} 0 212
   [junit4]   2> 818726 T5762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 818728 T5813 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32465,localport=32449], receiveBufferSize:6565888
   [junit4]   2> 818729 T5813 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32450,localport=32466], receiveBufferSize=6565888
   [junit4]   2> 818730 T5799 C528 P32449 oasc.SolrCore.execute [collection1] webapp=/vuwma/e path=/select params={version=2&rows=0&distrib=false&wt=javabin&tests=checkShardConsistency&q=*:*} hits=0 status=0 QTime=1 
   [junit4]   2> 818731 T5832 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32467,localport=32453], receiveBufferSize:6565888
   [junit4]   2> 818731 T5832 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32454,localport=32468], receiveBufferSize=6565888
   [junit4]   2> 818733 T5817 C526 P32453 oasc.SolrCore.execute [collection1] webapp=/vuwma/e path=/select params={version=2&rows=0&distrib=false&wt=javabin&tests=checkShardConsistency&q=*:*} hits=0 status=0 QTime=1 
   [junit4]   2> 820735 T5832 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=32473,localport=32453], receiveBufferSize:6565888
   [junit4]   2> 820735 T5832 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=32454,localport=32474], receiveBufferSize=6565888
   [junit4]   2> 820736 T5818 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&action=CREATE&version=2&wt=javabin&maxShardsPerNode=1&name=c8n_1x2&numShards=1
   [junit4]   2> 820738 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820738 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820738 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820738 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820739 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820740 T5784 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 820740 T5786 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "opera

[...truncated too long message...]

SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:150)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:121)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]   2> 	        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
   [junit4]   2> 	        at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:618)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1040)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 	        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  289) Thread[id=11191, name=qtp1185169633-11191, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]   2> 	        at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:150)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:121)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]   2> 	        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
   [junit4]   2> 	        at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:618)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1040)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 	        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_LU, timezone=Pacific/Guam
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=290,free=218670752,total=433192960
   [junit4]   2> NOTE: All tests run in this JVM: [MBeansHandlerTest, DeleteReplicaTest, ClusterStateUpdateTest, ReturnFieldsTest, JsonLoaderTest, LegacyHTMLStripCharFilterTest, TestSimpleQParserPlugin, BinaryUpdateRequestHandlerTest, TestLMDirichletSimilarityFactory, MultiThreadedOCPTest, CoreAdminHandlerTest, TestSolrDeletionPolicy1, SliceStateTest, AsyncMigrateRouteKeyTest, CSVRequestHandlerTest, FileBasedSpellCheckerTest, TestRandomDVFaceting, CollectionsAPIAsyncDistributedZkTest, SchemaVersionSpecificBehaviorTest, TestMaxScoreQueryParser, TestStandardQParsers, TestWriterPerf, UUIDFieldTest, TestDocSet, TestBadConfig, TestRandomMergePolicy, UnloadDistributedZkTest, TestNonDefinedSimilarityFactory, SyncSliceTest, TestDynamicFieldResource, TestCharFilters, AliasIntegrationTest, TestFieldTypeCollectionResource, LukeRequestHandlerTest, DistributedExpandComponentTest, TestBM25SimilarityFactory, TestInfoStreamLogging, HttpPartitionTest, DistributedQueryElevationComponentTest, TestPostingsSolrHighlighter, TestReversedWildcardFilterFactory, QueryElevationComponentTest, HdfsLockFactoryTest, ExternalFileFieldSortTest, TermVectorComponentTest, PreAnalyzedUpdateProcessorTest, TestSchemaVersionResource, EchoParamsTest, SynonymTokenizerTest, TermsComponentTest, SimpleFacetsTest, TestSearchPerf, DistributedSpellCheckComponentTest, TestExceedMaxTermLength, TestPerFieldSimilarity, StressHdfsTest, HdfsChaosMonkeySafeLeaderTest, UpdateRequestProcessorFactoryTest, SpellPossibilityIteratorTest, DistributedTermsComponentTest, TestRecoveryHdfs, ShardSplitTest, TestDocumentBuilder, SuggesterFSTTest, TestExpandComponent, HdfsCollectionsAPIDistributedZkTest, TestShortCircuitedRequests, SoftAutoCommitTest, SampleTest, UpdateParamsTest, QueryParsingTest, TestGroupingSearch, TestSystemIdResolver, TestLRUCache, DateMathParserTest, TestFunctionQuery, TestQueryUtils, TestManagedResource, URLClassifyProcessorTest, DOMUtilTest, TestSuggestSpellingConverter, TestJmxMonitoredMap, BasicFunctionalityTest, TestImplicitCoreProperties, TestManagedSynonymFilterFactory, SearchHandlerTest, RegexBoostProcessorTest, DocValuesTest, DocumentBuilderTest, TestConfigSets, TestNonNRTOpen, LoggingHandlerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OpenCloseCoreStressTest -Dtests.seed=E37F9276EC37BB4 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=de_LU -Dtests.timezone=Pacific/Guam -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | OpenCloseCoreStressTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E37F9276EC37BB4]:0)
   [junit4] Completed on J0 in 7227.54s, 1 test, 2 errors <<< FAILURES!

[...truncated 3266 lines...]
FATAL: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
	at hudson.remoting.Request.call(Request.java:174)
	at hudson.remoting.Channel.call(Channel.java:739)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:168)
	at com.sun.proxy.$Proxy56.join(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:951)
	at hudson.Launcher$ProcStarter.join(Launcher.java:362)
	at hudson.tasks.Ant.perform(Ant.java:217)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:745)
	at hudson.model.Build$BuildExecution.build(Build.java:198)
	at hudson.model.Build$BuildExecution.doRun(Build.java:159)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown
	at hudson.remoting.Request.abort(Request.java:299)
	at hudson.remoting.Channel.terminate(Channel.java:802)
	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:951)
	at hudson.remoting.Channel$2.handle(Channel.java:475)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
Caused by: hudson.remoting.Channel$OrderlyShutdown
	... 3 more
Caused by: Command close created at
	at hudson.remoting.Command.<init>(Command.java:56)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:945)
	at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:943)
	at hudson.remoting.Channel.close(Channel.java:1026)
	at hudson.remoting.Channel.close(Channel.java:1009)
	at hudson.remoting.Channel$CloseCommand.execute(Channel.java:950)
	... 2 more



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 566 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/566/

2 tests failed.
REGRESSION:  org.apache.lucene.search.TestControlledRealTimeReopenThread.testControlledRealTimeReopenThread

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([AB3966DCC7B7CE3E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestControlledRealTimeReopenThread

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([AB3966DCC7B7CE3E]:0)




Build Log:
[...truncated 1968 lines...]
   [junit4] Suite: org.apache.lucene.search.TestControlledRealTimeReopenThread
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestControlledRealTimeReopenThread -Dtests.method=testControlledRealTimeReopenThread -Dtests.seed=AB3966DCC7B7CE3E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_DZ -Dtests.timezone=Europe/Brussels -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7200s J1 | TestControlledRealTimeReopenThread.testControlledRealTimeReopenThread <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AB3966DCC7B7CE3E]:0)
   [junit4]   1> Thread-3366: hit exc
   [junit4]   1> Thread-3365: hit exc
   [junit4]   2> java.lang.InterruptedException
   [junit4]   1> Thread-3367: hit exc
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:182)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:135)
   [junit4]   2> 	at org.apache.lucene.search.TestControlledRealTimeReopenThread.addDocument(TestControlledRealTimeReopenThread.java:153)
   [junit4]   2> 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:234)
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:182)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:135)
   [junit4]   2> 	at org.apache.lucene.search.TestControlledRealTimeReopenThread.addDocuments(TestControlledRealTimeReopenThread.java:130)
   [junit4]   2> 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:217)
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:182)
   [junit4]   2> 	at org.apache.lucene.search.ControlledRealTimeReopenThread.waitForGeneration(ControlledRealTimeReopenThread.java:135)
   [junit4]   2> 	at org.apache.lucene.search.TestControlledRealTimeReopenThread.addDocuments(TestControlledRealTimeReopenThread.java:130)
   [junit4]   2> 	at org.apache.lucene.index.ThreadedIndexingAndSearchingTestCase$1.run(ThreadedIndexingAndSearchingTestCase.java:217)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J1/./lucene.search.TestControlledRealTimeReopenThread-AB3966DCC7B7CE3E-001
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {extra19=DFR I(ne)B2, extra14=IB LL-LZ(0.3), extra0=IB SPL-L3(800.0), extra27=DFR GL3(800.0), extra8=DFR I(n)L3(800.0), extra24=DFR GB3(800.0), extra12=DFR I(ne)L1, extra15=IB LL-D1, extra25=DFR I(ne)LZ(0.3), extra33=DFR GBZ(0.3), extra26=IB SPL-DZ(0.3), extra31=DFR I(F)L3(800.0), extra28=IB SPL-L2, extra2=DFR G3(800.0), extra34=IB SPL-LZ(0.3), titleTokenized=DFR I(n)BZ(0.3), extra3=DFR I(n)B2, extra21=IB LL-D3(800.0), extra37=DFR I(ne)3(800.0), extra35=DFR I(ne)BZ(0.3), extra38=DFR I(ne)L1, extra23=BM25(k1=1.2,b=0.75), extra10=DFR I(ne)B1, extra1=DFR I(F)L1, extra7=DFR I(ne)2, extra20=DFR GL2, extra17=DFR I(n)3(800.0), extra36=DFR I(n)B3(800.0), extra9=IB LL-D2, test=DFR I(F)1, extra39=IB LL-L1, extra18=IB SPL-D1, extra22=DFR I(n)L2, extra4=DFR GL1, extra32=DFR GB2, extra30=DefaultSimilarity, extra13=IB LL-L1, extra11=DFR I(ne)3(800.0), extra6=DFR I(F)L2, extra29=DFR I(F)BZ(0.3), extra16=IB SPL-L1, extra5=DFR I(n)LZ(0.3), body=DFR I(ne)L2}, locale=ar_DZ, timezone=Europe/Brussels
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=16,threads=4,free=447363448,total=510132224
   [junit4]   2> NOTE: All tests run in this JVM: [TestIndexInput, TestTwoPhaseCommitTool, TestStoredFieldsFormat, TestSimpleSearchEquivalence, TestNoDeletionPolicy, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestBufferedIndexInput, TestCheckIndex, TestCrashCausesCorruptIndex, TestConstantScoreQuery, TestIndexWriterExceptions2, ThrowInUncaught, TestMixedDocValuesUpdates, TestNoMergeScheduler, TestNGramPhraseQuery, TestPayloadSpans, TestPositionIncrement, TestSortDocValues, TestFilteredSearch, TestWildcardRandom, TestPayloads, Nested, Nested2, TestBlockPostingsFormat2, TestPayloadsOnVectors, TestDocValuesFormat, TestPrefixFilter, NestedSetupChain, NestedTeardownChain, TestDocCount, TestHugeRamFile, TestAtomicUpdate, TestDocIdSet, TestLevenshteinAutomata, TestSurrogates, TestMockDirectoryWrapper, TestNeverDelete, Test2BDocs, Nested1, TestPackedInts, TestPerFieldPostingsFormat2, TestSimpleFSDirectory, TestNormsFormat, TestTotalHitCountCollector, TestCompressingStoredFieldsFormat, TestByteBlockPool, TestTransactions, TestFieldReuse, TestSloppyMath, TestLucene40NormsFormat, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestDateSort, TestTopScoreDocCollector, TestMultiFields, TestDateFilter, TestTermVectorsFormat, TestIndexableBinaryStringTools, TestBasics, TestWorstCaseTestBehavior, TestPrefixCodedTerms, TestTerm, TestAttributeSource, TestSnapshotDeletionPolicy, TestBroadWord, TestWildcard, TestExplanations, TestToken, TestSimilarityBase, TestSmallFloat, TestLogMergePolicy, TestBoolean2, TestAllFilesHaveChecksumFooter, TestWeakIdentityMap, TestPriorityQueue, TestForceMergeForever, TestNRTCachingDirectory, TestOperations, TestGroupFiltering, TestFSTs, TestIndexWriterUnicode, TestPrefixInBooleanQuery, TestFieldCache, TestIndexWriterMergePolicy, FuzzyTermOnShortTermsTest, TestControlledRealTimeReopenThread]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestControlledRealTimeReopenThread -Dtests.seed=AB3966DCC7B7CE3E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_DZ -Dtests.timezone=Europe/Brussels -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | TestControlledRealTimeReopenThread (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AB3966DCC7B7CE3E]:0)
   [junit4] Completed on J1 in 7223.23s, 2 tests, 2 errors <<< FAILURES!

[...truncated 62 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 413 suites, 3502 tests, 1 suite-level error, 1 error, 205 ignored (113 assumptions)

Total time: 138 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-4.x #553
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 4844882 bytes
Compression is 0.0%
Took 3 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 565 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/565/

All tests passed

Build Log:
[...truncated 14421 lines...]
   [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp/junit4-J0-20140621_125257_468.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] 2014-06-22 14:44:03
   [junit4] Full thread dump OpenJDK 64-Bit Server VM (24.60-b09 mixed mode):
   [junit4] 
   [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e6800 nid=0x846b95c00 runnable [0x00007ffff6e70000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 	at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4] 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4] 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4] 	at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "RMI RenewClean-[127.0.0.1:40915]" daemon prio=5 tid=0x00000008517df000 nid=0x83c049800 in Object.wait() [0x00007ffff2426000]
   [junit4]    java.lang.Thread.State: TIMED_WAITING (on object monitor)
   [junit4] 	at java.lang.Object.wait(Native Method)
   [junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   [junit4] 	- locked <0x000000080de40760> (a java.lang.ref.ReferenceQueue$Lock)
   [junit4] 	at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:535)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "RMI Scheduler(0)" daemon prio=5 tid=0x00000008033e2800 nid=0x84abf0c00 waiting on condition [0x00007fffec5c8000]
   [junit4]    java.lang.Thread.State: TIMED_WAITING (parking)
   [junit4] 	at sun.misc.Unsafe.park(Native Method)
   [junit4] 	- parking to wait for  <0x000000080de3e008> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   [junit4] 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
   [junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
   [junit4] 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
   [junit4] 	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "GC Daemon" daemon prio=5 tid=0x00000008033e4000 nid=0x837d98400 in Object.wait() [0x00007ffff090b000]
   [junit4]    java.lang.Thread.State: TIMED_WAITING (on object monitor)
   [junit4] 	at java.lang.Object.wait(Native Method)
   [junit4] 	at sun.misc.GC$Daemon.run(GC.java:117)
   [junit4] 	- locked <0x000000080de434a8> (a sun.misc.GC$LatencyLock)
   [junit4] 
   [junit4] "RMI Reaper" prio=5 tid=0x00000008033e3800 nid=0x83c050400 in Object.wait() [0x00007ffff3638000]
   [junit4]    java.lang.Thread.State: WAITING (on object monitor)
   [junit4] 	at java.lang.Object.wait(Native Method)
   [junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   [junit4] 	- locked <0x000000080de36d38> (a java.lang.ref.ReferenceQueue$Lock)
   [junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
   [junit4] 	at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:351)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e2000 nid=0x850ca1800 runnable [0x00007ffff5557000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 	at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4] 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4] 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4] 	at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "RMI TCP Accept-0" daemon prio=5 tid=0x00000008033e1000 nid=0x8454da800 runnable [0x00007fffddde1000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 	at java.net.PlainSocketImpl.socketAccept(Native Method)
   [junit4] 	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
   [junit4] 	at java.net.ServerSocket.implAccept(ServerSocket.java:530)
   [junit4] 	at java.net.ServerSocket.accept(ServerSocket.java:498)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
   [junit4] 	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)
   [junit4] 
   [junit4] "Service Thread" daemon prio=5 tid=0x0000000801256800 nid=0x801183800 runnable [0x0000000000000000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 
   [junit4] "C2 CompilerThread1" daemon prio=5 tid=0x0000000801255800 nid=0x801182c00 waiting on condition [0x0000000000000000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 
   [junit4] "C2 CompilerThread0" daemon prio=5 tid=0x0000000801255000 nid=0x801049c00 waiting on condition [0x0000000000000000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 
   [junit4] "Signal Dispatcher" daemon prio=5 tid=0x0000000801254000 nid=0x801049000 waiting on condition [0x0000000000000000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 
   [junit4] "Finalizer" daemon prio=5 tid=0x0000000801253800 nid=0x801047c00 in Object.wait() [0x00007ffffebed000]
   [junit4]    java.lang.Thread.State: WAITING (on object monitor)
   [junit4] 	at java.lang.Object.wait(Native Method)
   [junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   [junit4] 	- locked <0x000000080be6bee0> (a java.lang.ref.ReferenceQueue$Lock)
   [junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
   [junit4] 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
   [junit4] 
   [junit4] "Reference Handler" daemon prio=5 tid=0x0000000801252800 nid=0x801047000 in Object.wait() [0x00007ffffecee000]
   [junit4]    java.lang.Thread.State: WAITING (on object monitor)
   [junit4] 	at java.lang.Object.wait(Native Method)
   [junit4] 	at java.lang.Object.wait(Object.java:503)
   [junit4] 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
   [junit4] 	- locked <0x000000080be84ac8> (a java.lang.ref.Reference$Lock)
   [junit4] 
   [junit4] "main" prio=5 tid=0x0000000801252000 nid=0x801007c00 runnable [0x00007fffffbfd000]
   [junit4]    java.lang.Thread.State: RUNNABLE
   [junit4] 	at java.io.FileInputStream.readBytes(Native Method)
   [junit4] 	at java.io.FileInputStream.read(FileInputStream.java:272)
   [junit4] 	at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
   [junit4] 	at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
   [junit4] 	- locked <0x000000080be8cd38> (a java.io.BufferedInputStream)
   [junit4] 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
   [junit4] 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
   [junit4] 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
   [junit4] 	- locked <0x000000080bef6c00> (a java.io.InputStreamReader)
   [junit4] 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
   [junit4] 	at java.io.BufferedReader.fill(BufferedReader.java:154)
   [junit4] 	at java.io.BufferedReader.readLine(BufferedReader.java:317)
   [junit4] 	- locked <0x000000080bef6c00> (a java.io.InputStreamReader)
   [junit4] 	at java.io.BufferedReader.readLine(BufferedReader.java:382)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.StdInLineIterator.computeNext(StdInLineIterator.java:31)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.StdInLineIterator.computeNext(StdInLineIterator.java:13)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.common.collect.Iterators$5.hasNext(Iterators.java:542)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:169)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:276)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4] 
   [junit4] "VM Thread" prio=5 tid=0x00000008013d1800 nid=0x801046400 runnable 
   [junit4] 
   [junit4] "GC task thread#0 (ParallelGC)" prio=5 tid=0x00000008013c8800 nid=0x80103c000 runnable 
   [junit4] 
   [junit4] "GC task thread#1 (ParallelGC)" prio=5 tid=0x00000008013c9800 nid=0x80103cc00 runnable 
   [junit4] 
   [junit4] "GC task thread#2 (ParallelGC)" prio=5 tid=0x00000008013ca000 nid=0x80103d800 runnable 
   [junit4] 
   [junit4] "GC task thread#3 (ParallelGC)" prio=5 tid=0x00000008013ca800 nid=0x80103e400 runnable 
   [junit4] 
   [junit4] "GC task thread#4 (ParallelGC)" prio=5 tid=0x00000008013cb000 nid=0x80103f000 runnable 
   [junit4] 
   [junit4] "GC task thread#5 (ParallelGC)" prio=5 tid=0x00000008013cc000 nid=0x80103fc00 runnable 
   [junit4] 
   [junit4] "GC task thread#6 (ParallelGC)" prio=5 tid=0x00000008013cc800 nid=0x801040800 runnable 
   [junit4] 
   [junit4] "GC task thread#7 (ParallelGC)" prio=5 tid=0x00000008013cd000 nid=0x801041400 runnable 
   [junit4] 
   [junit4] "GC task thread#8 (ParallelGC)" prio=5 tid=0x00000008013cd800 nid=0x801042000 runnable 
   [junit4] 
   [junit4] "GC task thread#9 (ParallelGC)" prio=5 tid=0x00000008013ce800 nid=0x801042c00 runnable 
   [junit4] 
   [junit4] "GC task thread#10 (ParallelGC)" prio=5 tid=0x00000008013cf000 nid=0x801043800 runnable 
   [junit4] 
   [junit4] "GC task thread#11 (ParallelGC)" prio=5 tid=0x00000008013cf800 nid=0x801044400 runnable 
   [junit4] 
   [junit4] "GC task thread#12 (ParallelGC)" prio=5 tid=0x00000008013d0000 nid=0x801045000 runnable 
   [junit4] 
   [junit4] "VM Periodic Task Thread" prio=5 tid=0x00000008013d2000 nid=0x801184800 waiting on condition 
   [junit4] 
   [junit4] JNI global references: 367
   [junit4] 
   [junit4] Heap
   [junit4]  PSYoungGen      total 166400K, used 3708K [0x0000000820e00000, 0x000000082b300000, 0x000000082b900000)
   [junit4]   eden space 163840K, 2% used [0x0000000820e00000,0x000000082119f1d8,0x000000082ae00000)
   [junit4]   from space 2560K, 0% used [0x000000082b080000,0x000000082b080000,0x000000082b300000)
   [junit4]   to   space 2560K, 0% used [0x000000082ae00000,0x000000082ae00000,0x000000082b080000)
   [junit4]  ParOldGen       total 302592K, used 130308K [0x000000080b900000, 0x000000081e080000, 0x0000000820e00000)
   [junit4]   object space 302592K, 43% used [0x000000080b900000,0x0000000813841278,0x000000081e080000)
   [junit4]  PSPermGen       total 55808K, used 55612K [0x0000000806700000, 0x0000000809d80000, 0x000000080b900000)
   [junit4]   object space 55808K, 99% used [0x0000000806700000,0x0000000809d4f160,0x0000000809d80000)
   [junit4] <<< JVM J0: EOF ----

[...truncated 2 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /usr/local/openjdk7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps -Dtests.prefix=tests -Dtests.seed=4E4398D387FD43E -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=4.10 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=2 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp -Dclover.db.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/tools/junit4/tests.policy -Dlucene.version=4.10-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.jettyConnector=Socket -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -classpath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/test-framework/lib/junit4-ant-2.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/analysis/common/lucene-analyzers-common-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/codecs/lucene-codecs-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/highlighter/lucene-highlighter-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/memory/lucene-memory-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/misc/lucene-misc-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/spatial/lucene-spatial-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/expressions/lucene-expressions-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/suggest/lucene-suggest-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/grouping/lucene-grouping-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/queries/lucene-queries-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/queryparser/lucene-queryparser-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/join/lucene-join-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/antlr-runtime-3.5.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/asm-4.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/asm-commons-4.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/commons-codec-1.9.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/commons-configuration-1.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/dom4j-1.6.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/guava-14.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/hadoop-annotations-2.2.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/hadoop-auth-2.2.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/hadoop-common-2.2.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/hadoop-hdfs-2.2.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/hppc-0.5.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/joda-time-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/log4j-1.2.17.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/org.restlet-2.1.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/protobuf-java-2.5.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/slf4j-log4j12-1.7.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/lib/spatial4j-0.4.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/commons-io-2.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/httpclient-4.3.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/httpcore-4.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/httpmime-4.3.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/noggit-0.5.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/slf4j-api-1.7.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/solrj/lib/zookeeper-3.4.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-continuation-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-deploy-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-http-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-io-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-jmx-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-security-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-server-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-servlet-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-util-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-webapp-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/jetty-xml-8.1.10.v20130312.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/test-framework/lib/randomizedtesting-runner-2.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/cglib-nodep-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/commons-collections-3.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/easymock-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/jcl-over-slf4j-1.7.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/jersey-core-1.9.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/jetty-6.1.26.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/jetty-util-6.1.26.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/test-lib/objenesis-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/analysis/icu/lucene-analyzers-icu-4.10-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/contrib/solr-analysis-extras/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/contrib/analysis-extras/lib/icu4j-53.1.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.3.0.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/usr/local/openjdk7/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.3.jar -ea:org.apache.lucene... -ea:org.apache.solr... com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp/junit4-J0-20140621_125257_468.events @/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/temp/junit4-J0-20140621_125257_468.suites
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 143. Very likely a JVM crash.  Process output piped in logs above.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1461)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:948)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:945)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: At least one slave process threw an exception, first: Forked process returned with error code: 143. Very likely a JVM crash.  Process output piped in logs above.

Total time: 1691 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-4.x #553
Archived 3 artifacts
Archive block size is 32768
Received 0 blocks and 582691490 bytes
Compression is 0.0%
Took 1 min 41 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 564 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/564/

1 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
IOException occured when talking to server at: http://127.0.0.1:64999/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:64999/collection1
	at __randomizedtesting.SeedInfo.seed([B7DB3133002727B3:363DBF2B7778478F]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:566)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:415)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:145)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.writeLine(AbstractSessionOutputBuffer.java:258)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:59)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:46)
	at org.apache.http.impl.io.AbstractMessageWriter.write(AbstractMessageWriter.java:109)
	at org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:254)
	at org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader(DefaultClientConnection.java:270)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestHeader(ManagedClientConnectionImpl.java:209)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:203)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:452)
	... 50 more




Build Log:
[...truncated 11320 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/init-core-data-001
   [junit4]   2> 27453 T32 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 27470 T32 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 27486 T32 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 27500 T33 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 27801 T32 oasc.ZkTestServer.run start zk server on port:11922
   [junit4]   2> 28160 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28369 T39 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aa57db5 name:ZooKeeperConnection Watcher:127.0.0.1:11922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28370 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28378 T32 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 28535 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 28539 T41 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ef73b32 name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 28539 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 28563 T32 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 28570 T32 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 28575 T32 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 28580 T32 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 28586 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 28588 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 28603 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 28604 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 28610 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 28611 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 28617 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 28618 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 28624 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 28625 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 28630 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 28631 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 28637 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 28638 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 28644 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 28645 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 28651 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 28652 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 28658 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 28659 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 28665 T32 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 28666 T32 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 30854 T32 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 30860 T32 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64999
   [junit4]   2> 30861 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 30862 T32 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 30863 T32 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002
   [junit4]   2> 30863 T32 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/'
   [junit4]   2> 30926 T32 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/solr.xml
   [junit4]   2> 31064 T32 oasc.CoreContainer.<init> New CoreContainer 47942279
   [junit4]   2> 31065 T32 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/]
   [junit4]   2> 31067 T32 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 31068 T32 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 31069 T32 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 31069 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 31070 T32 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 31071 T32 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 31071 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 31072 T32 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 31073 T32 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 31074 T32 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 31075 T32 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 31075 T32 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 31076 T32 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11922/solr
   [junit4]   2> 31132 T32 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 31134 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 31138 T52 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36ff26bd name:ZooKeeperConnection Watcher:127.0.0.1:11922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 31138 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 31180 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 31183 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c73ebd7 name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 31183 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 31220 T32 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 31242 T32 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 31255 T32 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 31262 T32 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 31274 T32 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 31297 T32 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 31302 T32 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64999_
   [junit4]   2> 31347 T32 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64999_
   [junit4]   2> 31406 T32 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 31412 T32 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 31480 T32 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64999_
   [junit4]   2> 31482 T32 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 31549 T32 oasc.Overseer.start Overseer (id=91876514503655427-127.0.0.1:64999_-n_0000000000) starting
   [junit4]   2> 31632 T32 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 31734 T56 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 31735 T32 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 31739 T32 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 31742 T32 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 31774 T55 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 31809 T57 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 31810 T57 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 31823 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31823 T57 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 31832 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 31834 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64999",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64999_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 31835 T55 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 31880 T55 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 31890 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 31937 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 32825 T57 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 32826 T57 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 32827 T57 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 32828 T57 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 32830 T57 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 32831 T57 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/collection1/'
   [junit4]   2> 32833 T57 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 32835 T57 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 32836 T57 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 33030 T57 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 33144 T57 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 33147 T57 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 33233 T57 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 34180 T57 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 34186 T57 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 34190 T57 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34200 T57 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 34273 T57 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 34274 T57 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 34275 T57 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-002/collection1/, dataDir=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/control/data/
   [junit4]   2> 34275 T57 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43fb81ce
   [junit4]   2> 34286 T57 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/control/data
   [junit4]   2> 34286 T57 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/control/data/index/
   [junit4]   2> 34287 T57 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 34298 T57 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/control/data/index
   [junit4]   2> 34299 T57 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.6337890625, noCFSRatio=0.8735814404442317]
   [junit4]   2> 34308 T57 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-001 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f77cb91),segFN=segments_1,generation=1}
   [junit4]   2> 34309 T57 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34319 T57 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 34443 T57 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 34444 T57 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 34445 T57 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34445 T57 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 34446 T57 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 34447 T57 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34448 T57 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 34448 T57 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 34459 T57 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 34461 T57 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 34462 T57 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 34462 T57 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 34464 T57 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 34465 T57 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 34467 T57 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 34490 T57 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 34504 T57 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 34505 T57 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 34507 T57 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@27fcd3fe
   [junit4]   2> 34520 T57 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-001 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6f77cb91),segFN=segments_1,generation=1}
   [junit4]   2> 34521 T57 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 34522 T57 oass.SolrIndexSearcher.<init> Opening Searcher@40f93dfb[collection1] main
   [junit4]   2> 34522 T57 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 34524 T57 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 34525 T57 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 34539 T57 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 34540 T57 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 34540 T57 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 34541 T57 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 34542 T57 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 34543 T57 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 34543 T57 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 34558 T58 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40f93dfb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 34567 T57 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 34569 T61 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:64999 collection:control_collection shard:shard1
   [junit4]   2> 34570 T32 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 34571 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 34608 T61 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 34616 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 34629 T63 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22a22c0e name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 34630 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 34632 T32 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 34633 T61 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 34645 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34645 T61 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 34646 T61 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 34647 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 34648 T32 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@3ec1a187 url=http://127.0.0.1:64999/collection1 node=127.0.0.1:64999_ C2_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:64999, core=collection1, node_name=127.0.0.1:64999_}
   [junit4]   2> 34654 T61 C2 P64999 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:64999/collection1/
   [junit4]   2> 34655 T61 C2 P64999 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 34655 T61 C2 P64999 oasc.SyncStrategy.syncToMe http://127.0.0.1:64999/collection1/ has no replicas
   [junit4]   2> 34656 T61 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:64999/collection1/ shard1
   [junit4]   2> 34659 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34660 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34660 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34670 T61 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 34685 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34687 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 34692 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34797 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34797 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34840 T61 oasc.ZkController.register We are http://127.0.0.1:64999/collection1/ and leader is http://127.0.0.1:64999/collection1/
   [junit4]   2> 34840 T61 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64999
   [junit4]   2> 34841 T61 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 34841 T61 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 34842 T61 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 34845 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34845 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34846 T61 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 34846 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34849 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 34851 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64999",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64999_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 34856 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 34995 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 34996 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 36084 T32 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 36085 T32 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 36088 T32 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35724
   [junit4]   2> 36089 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 36090 T32 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 36090 T32 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003
   [junit4]   2> 36090 T32 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/'
   [junit4]   2> 36125 T32 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/solr.xml
   [junit4]   2> 36219 T32 oasc.CoreContainer.<init> New CoreContainer 328741404
   [junit4]   2> 36220 T32 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/]
   [junit4]   2> 36221 T32 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 36222 T32 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 36222 T32 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 36223 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 36223 T32 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 36224 T32 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 36224 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 36224 T32 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 36225 T32 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 36226 T32 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 36226 T32 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 36227 T32 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 36227 T32 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11922/solr
   [junit4]   2> 36227 T32 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 36228 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 36232 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d01d650 name:ZooKeeperConnection Watcher:127.0.0.1:11922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 36232 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 36251 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 36253 T76 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4735a83a name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 36254 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 36264 T32 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 37269 T32 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35724_
   [junit4]   2> 37271 T32 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35724_
   [junit4]   2> 37275 T63 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 37275 T76 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 37275 T54 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 37302 T77 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 37302 T77 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 37304 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37304 T77 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 37305 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37305 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37307 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 37309 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35724",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35724_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 37309 T55 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 37309 T55 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 37314 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 37321 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 37322 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 37321 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 38306 T77 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 38306 T77 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 38308 T77 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 38309 T77 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 38310 T77 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 38311 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/collection1/'
   [junit4]   2> 38313 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 38314 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 38315 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 38427 T77 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 38536 T77 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 38539 T77 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 38615 T77 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 39346 T77 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 39350 T77 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 39354 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 39362 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 39425 T77 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 39426 T77 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 39426 T77 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-003/collection1/, dataDir=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty1/
   [junit4]   2> 39427 T77 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43fb81ce
   [junit4]   2> 39429 T77 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty1
   [junit4]   2> 39430 T77 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty1/index/
   [junit4]   2> 39430 T77 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 39432 T77 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty1/index
   [junit4]   2> 39433 T77 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.6337890625, noCFSRatio=0.8735814404442317]
   [junit4]   2> 39437 T77 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-002 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74b3a2ae),segFN=segments_1,generation=1}
   [junit4]   2> 39437 T77 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 39449 T77 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 39450 T77 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 39451 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 39451 T77 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 39452 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 39452 T77 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 39453 T77 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 39454 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 39455 T77 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 39458 T77 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 39460 T77 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 39460 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 39461 T77 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 39463 T77 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 39464 T77 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 39466 T77 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 39484 T77 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 39491 T77 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 39492 T77 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 39494 T77 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6ceb99c3
   [junit4]   2> 39498 T77 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-002 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@74b3a2ae),segFN=segments_1,generation=1}
   [junit4]   2> 39498 T77 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 39499 T77 oass.SolrIndexSearcher.<init> Opening Searcher@42b67efc[collection1] main
   [junit4]   2> 39499 T77 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 39501 T77 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 39502 T77 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 39503 T77 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 39503 T77 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 39503 T77 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 39505 T77 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 39505 T77 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 39505 T77 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 39506 T77 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 39521 T78 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42b67efc[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 39528 T77 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 39530 T81 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35724 collection:collection1 shard:shard2
   [junit4]   2> 39530 T32 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 39531 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 39531 T81 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 39552 T81 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 39562 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39562 T81 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 39563 T81 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@18286079 url=http://127.0.0.1:35724/collection1 node=127.0.0.1:35724_ C3_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35724, core=collection1, node_name=127.0.0.1:35724_}
   [junit4]   2> 39563 T81 C3 P35724 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35724/collection1/
   [junit4]   2> 39564 T81 C3 P35724 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 39564 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 39564 T81 C3 P35724 oasc.SyncStrategy.syncToMe http://127.0.0.1:35724/collection1/ has no replicas
   [junit4]   2> 39565 T81 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35724/collection1/ shard2
   [junit4]   2> 39565 T81 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 39578 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39580 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39580 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39580 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39595 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 39600 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39704 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39704 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39704 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39747 T81 oasc.ZkController.register We are http://127.0.0.1:35724/collection1/ and leader is http://127.0.0.1:35724/collection1/
   [junit4]   2> 39747 T81 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35724
   [junit4]   2> 39747 T81 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 39748 T81 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 39748 T81 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 39751 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39751 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39752 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39752 T81 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 39753 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 39755 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35724",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35724_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 39760 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 39866 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39866 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 39866 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 40901 T32 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 40902 T32 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 40905 T32 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:46412
   [junit4]   2> 40905 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 40906 T32 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 40906 T32 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004
   [junit4]   2> 40907 T32 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/'
   [junit4]   2> 40949 T32 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/solr.xml
   [junit4]   2> 41040 T32 oasc.CoreContainer.<init> New CoreContainer 1486808866
   [junit4]   2> 41041 T32 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/]
   [junit4]   2> 41043 T32 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 41043 T32 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 41043 T32 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 41044 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 41044 T32 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 41045 T32 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 41045 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 41045 T32 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 41046 T32 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 41047 T32 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 41047 T32 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 41047 T32 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 41048 T32 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11922/solr
   [junit4]   2> 41048 T32 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 41049 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41052 T92 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@285896a2 name:ZooKeeperConnection Watcher:127.0.0.1:11922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41053 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41056 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 41064 T94 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7824d24b name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 41065 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 41082 T32 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 42087 T32 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46412_
   [junit4]   2> 42092 T32 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46412_
   [junit4]   2> 42097 T54 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 42097 T94 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 42097 T76 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 42097 T63 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 42112 T95 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 42112 T95 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 42115 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42115 T95 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 42115 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42116 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42119 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 42120 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46412",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46412_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 42121 T55 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 42121 T55 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 42126 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 42134 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42135 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42135 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 42135 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 43117 T95 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 43117 T95 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 43118 T95 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 43119 T95 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 43120 T95 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 43121 T95 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/collection1/'
   [junit4]   2> 43123 T95 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 43123 T95 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 43124 T95 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 43194 T95 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 43256 T95 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 43259 T95 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 43304 T95 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 44039 T95 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 44044 T95 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 44048 T95 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 44057 T95 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 44129 T95 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 44130 T95 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 44130 T95 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-004/collection1/, dataDir=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty2/
   [junit4]   2> 44131 T95 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@43fb81ce
   [junit4]   2> 44133 T95 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty2
   [junit4]   2> 44133 T95 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty2/index/
   [junit4]   2> 44133 T95 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 44136 T95 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-001/jetty2/index
   [junit4]   2> 44137 T95 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.6337890625, noCFSRatio=0.8735814404442317]
   [junit4]   2> 44142 T95 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-003 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20880a03),segFN=segments_1,generation=1}
   [junit4]   2> 44142 T95 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44155 T95 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 44155 T95 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 44156 T95 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 44156 T95 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 44157 T95 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 44157 T95 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 44158 T95 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 44159 T95 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 44160 T95 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 44162 T95 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 44164 T95 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 44165 T95 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 44166 T95 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 44168 T95 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 44169 T95 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 44171 T95 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 44197 T95 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 44205 T95 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 44205 T95 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 44208 T95 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5e00ecff
   [junit4]   2> 44212 T95 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/index-MMapDirectory-003 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20880a03),segFN=segments_1,generation=1}
   [junit4]   2> 44212 T95 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 44213 T95 oass.SolrIndexSearcher.<init> Opening Searcher@2e3b125b[collection1] main
   [junit4]   2> 44214 T95 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 44216 T95 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 44216 T95 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 44217 T95 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 44218 T95 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 44218 T95 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44220 T95 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 44220 T95 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 44221 T95 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 44221 T95 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 44238 T96 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e3b125b[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 44254 T95 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 44255 T99 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46412 collection:collection1 shard:shard3
   [junit4]   2> 44256 T32 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 44257 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 44258 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 44282 T99 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 44292 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44292 T99 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 44293 T99 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@1a13853b url=http://127.0.0.1:46412/collection1 node=127.0.0.1:46412_ C4_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:46412, core=collection1, node_name=127.0.0.1:46412_}
   [junit4]   2> 44293 T99 C4 P46412 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46412/collection1/
   [junit4]   2> 44294 T99 C4 P46412 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 44294 T99 C4 P46412 oasc.SyncStrategy.syncToMe http://127.0.0.1:46412/collection1/ has no replicas
   [junit4]   2> 44295 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44295 T99 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46412/collection1/ shard3
   [junit4]   2> 44296 T99 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 44301 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44310 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44310 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44310 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44310 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44324 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44331 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44435 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44435 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44435 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44435 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44476 T99 oasc.ZkController.register We are http://127.0.0.1:46412/collection1/ and leader is http://127.0.0.1:46412/collection1/
   [junit4]   2> 44476 T99 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46412
   [junit4]   2> 44477 T99 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 44477 T99 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 44478 T99 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 44481 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44481 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44482 T99 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44482 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44484 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 44486 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:46412",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:46412_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 44490 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 44595 T76 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44595 T63 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44595 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 44595 T54 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 45641 T32 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 45642 T32 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 45646 T32 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:21116
   [junit4]   2> 45647 T32 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 45647 T32 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 45648 T32 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-005
   [junit4]   2> 45649 T32 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-005/'
   [junit4]   2> 45695 T32 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-005/solr.xml
   [junit4]   2> 45815 T32 oasc.CoreContainer.<init> New CoreContainer 1343064351
   [junit4]   2> 45816 T32 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001/tempDir-005/]
   [junit4]   2> 45818 T32 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 45819 T32 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 45819 T32 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 45820 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 45820 T32 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 45821 T32 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 45821 T32 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 45822 T32 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 45823 T32 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 45824 T32 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 45824 T32 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 45825 T32 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 45826 T32 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11922/solr
   [junit4]   2> 45826 T32 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 45828 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 45831 T110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28a57b48 name:ZooKeeperConnection Watcher:127.0.0.1:11922 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 45832 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 45854 T32 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 45856 T112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18191397 name:ZooKeeperConnection Watcher:127.0.0.1:11922/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 45857 T32 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 45869 T32 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 46875 T32 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21116_
   [junit4]   2> 46877 T32 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21116_
   [junit4]   2> 46882 T94 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 46882 T63 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 46883 T112 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 46882 T76 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 46883 T54 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 46897 T113 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 46898 T113 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 46900 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 46900 T113 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 46900 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 46901 T54 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 46904 T55 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 46906 T55 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  

[...truncated too long message...]

loud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141905 T94 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 141906 T150 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard5/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141906 T94 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 141907 T150 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard6/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141907 T94 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141908 T150 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141908 T94 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection2/leader_elect/shard5/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 141910 T94 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.FullSolrCloudDistribCmdsTest-B7DB3133002727B3-001
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=sr_BA, timezone=America/Anchorage
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=254671440,total=415694848
   [junit4]   2> NOTE: All tests run in this JVM: [SortByFunctionTest, TestSolrCoreProperties, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 114.92s, 1 test, 1 error <<< FAILURES!

[...truncated 1232 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 397 suites, 1640 tests, 1 error, 34 ignored (5 assumptions)

Total time: 251 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 563 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/563/

All tests passed

Build Log:
[...truncated 1137 lines...]
   [junit4] JVM J1: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J1-20140603_192925_964.sysout
   [junit4] >>> JVM J1: stdout (verbatim) ----
   [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0x0000000804b173a4, pid=17480, tid=35126711296
   [junit4] #
   [junit4] # JRE version: 7.0-b15
   [junit4] # Java VM: OpenJDK 64-Bit Server VM (23.21-b01 mixed mode bsd-amd64 compressed oops)
   [junit4] # Problematic frame:
   [junit4] # J  org.apache.lucene.util.RamUsageEstimator.measureObjectSize(Ljava/lang/Object;)J
   [junit4] #
   [junit4] # Core dump written. Default location: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J1/core or core.17480
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J1/hs_err_pid17480.log
   [junit4] #
   [junit4] # If you would like to submit a bug report, please visit:
   [junit4] #   http://bugreport.sun.com/bugreport/crash.jsp
   [junit4] #
   [junit4] <<< JVM J1: EOF ----

[...truncated 702 lines...]
   [junit4] ERROR: JVM J1 ended with an exception, command line: /usr/local/openjdk7/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps -Dtests.prefix=tests -Dtests.seed=5EAC22CCF726E110 -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=4.9 -Dtests.cleanthreads=perMethod -Djava.util.logging.config.file=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=2 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp -Dclover.db.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/tools/junit4/tests.policy -Dlucene.version=4.9-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.jettyConnector=Socket -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -classpath /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/test-framework/lib/randomizedtesting-runner-2.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/classes/test:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.3.0.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/usr/local/openjdk7/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.3.jar -ea:org.apache.lucene... -ea:org.apache.solr... com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J1-20140603_192925_964.events @/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J1-20140603_192925_964.suites
   [junit4] ERROR: JVM J1 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1461)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:948)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:945)
   [junit4] 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:724)

BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.

Total time: 96 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 562 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/562/

4 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1
	at __randomizedtesting.SeedInfo.seed([34B4BFCAADBE9F4C:B55231D2DAE1FF70]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:545)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:349)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:148)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10MinutesOld

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([34B4BFCAADBE9F4C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([34B4BFCAADBE9F4C]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1401663490390,generation=2,filelist=[_qxa.cfe, _qxa.cfs, _qxa.si, _r79.cfe, _r79.cfs, _r79.si, _r7o.cfe, _r7o.cfs, _r7o.si, _r7p.cfe, _r7p.cfs, _r7p.si, _r7q.cfe, _r7q.cfs, _r7q.si, _r7r.cfe, _r7r.cfs, _r7r.si, _r7u.cfe, _r7u.cfs, _r7u.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_2]}]> but was:<[{indexVersion=1401663490390,generation=2,filelist=[_qxa.cfe, _qxa.cfs, _qxa.si, _r79.cfe, _r79.cfs, _r79.si, _r7o.cfe, _r7o.cfs, _r7o.si, _r7p.cfe, _r7p.cfs, _r7p.si, _r7q.cfe, _r7q.cfs, _r7q.si, _r7r.cfe, _r7r.cfs, _r7r.si, _r7u.cfe, _r7u.cfs, _r7u.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_2]}, {indexVersion=1401663490390,generation=3,filelist=[_r7x.cfe, _r7x.cfs, _r7x.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1401663490390,generation=2,filelist=[_qxa.cfe, _qxa.cfs, _qxa.si, _r79.cfe, _r79.cfs, _r79.si, _r7o.cfe, _r7o.cfs, _r7o.si, _r7p.cfe, _r7p.cfs, _r7p.si, _r7q.cfe, _r7q.cfs, _r7q.si, _r7r.cfe, _r7r.cfs, _r7r.si, _r7u.cfe, _r7u.cfs, _r7u.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_2]}]> but was:<[{indexVersion=1401663490390,generation=2,filelist=[_qxa.cfe, _qxa.cfs, _qxa.si, _r79.cfe, _r79.cfs, _r79.si, _r7o.cfe, _r7o.cfs, _r7o.si, _r7p.cfe, _r7p.cfs, _r7p.si, _r7q.cfe, _r7q.cfs, _r7q.si, _r7r.cfe, _r7r.cfs, _r7r.si, _r7u.cfe, _r7u.cfs, _r7u.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_2]}, {indexVersion=1401663490390,generation=3,filelist=[_r7x.cfe, _r7x.cfs, _r7x.si, _rbb.cfe, _rbb.cfs, _rbb.si, _rbg.cfe, _rbg.cfs, _rbg.si, _rbh.cfe, _rbh.cfs, _rbh.si, _rbi.cfe, _rbi.cfs, _rbi.si, _rbj.cfe, _rbj.cfs, _rbj.si, _rbk.cfe, _rbk.cfs, _rbk.si, _rbl.cfe, _rbl.cfs, _rbl.si, _rbm.cfe, _rbm.cfs, _rbm.si, _rbn.cfe, _rbn.cfs, _rbn.si, _rbo.cfe, _rbo.cfs, _rbo.si, _rbp.cfe, _rbp.cfs, _rbp.si, _rbq.cfe, _rbq.cfs, _rbq.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([34B4BFCAADBE9F4C:1163A4FADDF6914F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1204)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 11284 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/init-core-data-001
   [junit4]   2> 137554 T178 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 137559 T178 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 137567 T178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 137574 T179 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 137775 T178 oasc.ZkTestServer.run start zk server on port:22905
   [junit4]   2> 138081 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 138128 T180 oazs.NIOServerCnxn.doIO WARN Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
   [junit4]   2> 139696 T184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c73d88 name:ZooKeeperConnection Watcher:127.0.0.1:22905 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139696 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139703 T178 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 139881 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 139907 T187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1092fe37 name:ZooKeeperConnection Watcher:127.0.0.1:22905/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139907 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139915 T178 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 139952 T178 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 139975 T178 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 140005 T178 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 140043 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 140044 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 140446 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 140447 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 140583 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 140584 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 140727 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 140728 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 140875 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 140876 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 140929 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 140930 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 141045 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 141046 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 141177 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 141178 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 141224 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 141225 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 141443 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 141444 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 141454 T178 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 141455 T178 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 143072 T178 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 143079 T178 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63759
   [junit4]   2> 143080 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 143080 T178 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 143081 T178 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002
   [junit4]   2> 143081 T178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/'
   [junit4]   2> 143123 T178 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/solr.xml
   [junit4]   2> 143224 T178 oasc.CoreContainer.<init> New CoreContainer 1497173052
   [junit4]   2> 143224 T178 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/]
   [junit4]   2> 143226 T178 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 143227 T178 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 143227 T178 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 143228 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 143228 T178 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 143229 T178 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 143229 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 143230 T178 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 143230 T178 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 143231 T178 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 143232 T178 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 143232 T178 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 143233 T178 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22905/solr
   [junit4]   2> 143274 T178 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 143275 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 143322 T198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41dcc47f name:ZooKeeperConnection Watcher:127.0.0.1:22905 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 143323 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 143359 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 143380 T200 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b51b399 name:ZooKeeperConnection Watcher:127.0.0.1:22905/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 143381 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 143434 T178 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 143454 T178 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 143485 T178 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 143503 T178 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 143516 T178 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 143546 T178 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 143549 T178 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63992_
   [junit4]   2> 143577 T178 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63992_
   [junit4]   2> 143594 T178 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 143630 T178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 143883 T178 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:63992_
   [junit4]   2> 143884 T178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 144020 T178 oasc.Overseer.start Overseer (id=91859344125919235-127.0.0.1:63992_-n_0000000000) starting
   [junit4]   2> 144182 T178 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 144465 T202 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 144466 T178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 144471 T178 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 144497 T178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 144531 T201 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 144556 T203 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 144556 T203 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 144578 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 144578 T203 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 144587 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 144588 T201 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63992",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63992_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 144589 T201 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 144632 T201 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 144680 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 144725 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 145580 T203 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 145580 T203 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 145582 T203 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 145582 T203 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 145584 T203 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 145585 T203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/collection1/'
   [junit4]   2> 145586 T203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 145587 T203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 145588 T203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 145737 T203 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 145808 T203 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 145811 T203 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 145823 T203 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 146521 T203 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 146522 T203 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 146523 T203 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 146538 T203 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 146543 T203 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 146569 T203 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 146576 T203 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 146582 T203 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 146584 T203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 146585 T203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 146585 T203 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 146587 T203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 146587 T203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 146587 T203 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 146588 T203 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 146588 T203 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-002/collection1/, dataDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/control/data/
   [junit4]   2> 146589 T203 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@177730c8
   [junit4]   2> 146601 T203 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/control/data
   [junit4]   2> 146602 T203 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/control/data/index/
   [junit4]   2> 146602 T203 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 146603 T203 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/control/data/index
   [junit4]   2> 146603 T203 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@64493a45
   [junit4]   2> 146605 T203 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33c4ea91 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@390bb73),segFN=segments_1,generation=1}
   [junit4]   2> 146605 T203 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 146615 T203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 146616 T203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 146617 T203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 146617 T203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 146618 T203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 146619 T203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 146619 T203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 146620 T203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 146620 T203 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 146622 T203 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 146623 T203 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 146624 T203 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 146624 T203 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 146626 T203 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 146626 T203 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 146628 T203 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 146650 T203 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 146658 T203 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 146659 T203 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 146660 T203 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=78.0400390625, floorSegmentMB=0.2568359375, forceMergeDeletesPctAllowed=25.220266225928427, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6623180696044442
   [junit4]   2> 146661 T203 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@33c4ea91 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@390bb73),segFN=segments_1,generation=1}
   [junit4]   2> 146662 T203 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 146662 T203 oass.SolrIndexSearcher.<init> Opening Searcher@4a9ed9f8[collection1] main
   [junit4]   2> 146663 T203 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 146665 T203 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 146665 T203 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 146681 T203 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 146681 T203 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 146682 T203 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 146683 T203 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 146683 T203 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 146684 T203 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 146684 T203 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 146694 T204 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a9ed9f8[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 146702 T203 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 146703 T207 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63992 collection:control_collection shard:shard1
   [junit4]   2> 146704 T178 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 146704 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 146732 T207 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 146793 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 146934 T210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d68c8e7 name:ZooKeeperConnection Watcher:127.0.0.1:22905/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 146935 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 146937 T178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 147154 T207 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 147258 T178 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 147308 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 147309 T207 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 147309 T207 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C40 name=collection1 org.apache.solr.core.SolrCore@aa11928 url=http://127.0.0.1:63992/collection1 node=127.0.0.1:63992_ C40_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:63992, core=collection1, node_name=127.0.0.1:63992_}
   [junit4]   2> 147310 T207 C40 P63992 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63992/collection1/
   [junit4]   2> 147310 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 147311 T207 C40 P63992 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 147311 T207 C40 P63992 oasc.SyncStrategy.syncToMe http://127.0.0.1:63992/collection1/ has no replicas
   [junit4]   2> 147311 T207 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63992/collection1/ shard1
   [junit4]   2> 147326 T207 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 147470 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 147537 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 147537 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 147938 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 148204 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 148386 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 148386 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 148471 T178 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 148472 T178 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 148475 T178 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42044
   [junit4]   2> 148476 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 148476 T178 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 148477 T178 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003
   [junit4]   2> 148477 T178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/'
   [junit4]   2> 148499 T207 oasc.ZkController.register We are http://127.0.0.1:63992/collection1/ and leader is http://127.0.0.1:63992/collection1/
   [junit4]   2> 148499 T207 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63992
   [junit4]   2> 148499 T207 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 148500 T207 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 148500 T207 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 148520 T178 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/solr.xml
   [junit4]   2> 148614 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 148614 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 148614 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 148615 T207 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 148617 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 148618 T201 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63992",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63992_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 148623 T178 oasc.CoreContainer.<init> New CoreContainer 38831533
   [junit4]   2> 148623 T178 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/]
   [junit4]   2> 148625 T178 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 148626 T178 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 148626 T178 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 148627 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 148627 T178 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 148628 T178 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 148628 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 148629 T178 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 148629 T178 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 148630 T178 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 148631 T178 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 148631 T178 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 148632 T178 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22905/solr
   [junit4]   2> 148632 T178 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 148633 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 148834 T221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7478220a name:ZooKeeperConnection Watcher:127.0.0.1:22905 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 148835 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 148969 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 149058 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 149076 T223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f6b6954 name:ZooKeeperConnection Watcher:127.0.0.1:22905/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 149077 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 149277 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 149277 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 149379 T178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 150386 T178 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24683_
   [junit4]   2> 150442 T178 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24683_
   [junit4]   2> 150453 T200 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 150453 T210 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 150453 T223 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 150523 T224 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 150523 T224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 150542 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 150542 T224 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 150542 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 150543 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 150545 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 150547 T201 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:24683",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:24683_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 150547 T201 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 150547 T201 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 150597 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 150733 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 150733 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 150733 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 151544 T224 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 151545 T224 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 151547 T224 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 151547 T224 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 151549 T224 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 151550 T224 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/collection1/'
   [junit4]   2> 151551 T224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 151552 T224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 151553 T224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 151642 T224 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 151715 T224 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 151717 T224 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 151728 T224 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 152411 T224 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 152411 T224 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 152412 T224 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 152430 T224 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 152435 T224 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 152460 T224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 152466 T224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 152472 T224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 152474 T224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 152474 T224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 152475 T224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 152476 T224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 152477 T224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 152477 T224 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 152477 T224 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 152478 T224 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-003/collection1/, dataDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty1/
   [junit4]   2> 152478 T224 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@177730c8
   [junit4]   2> 152480 T224 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty1
   [junit4]   2> 152480 T224 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty1/index/
   [junit4]   2> 152480 T224 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 152481 T224 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty1/index
   [junit4]   2> 152481 T224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1c99c1ba
   [junit4]   2> 152483 T224 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@69d47f66 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@327da901),segFN=segments_1,generation=1}
   [junit4]   2> 152483 T224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 152495 T224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 152495 T224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 152496 T224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 152496 T224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 152496 T224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 152497 T224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 152497 T224 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 152498 T224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 152498 T224 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 152500 T224 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 152502 T224 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 152503 T224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 152503 T224 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 152505 T224 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 152506 T224 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 152508 T224 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 152528 T224 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 152535 T224 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 152535 T224 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 152537 T224 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=78.0400390625, floorSegmentMB=0.2568359375, forceMergeDeletesPctAllowed=25.220266225928427, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6623180696044442
   [junit4]   2> 152538 T224 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@69d47f66 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@327da901),segFN=segments_1,generation=1}
   [junit4]   2> 152539 T224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 152539 T224 oass.SolrIndexSearcher.<init> Opening Searcher@305ab838[collection1] main
   [junit4]   2> 152540 T224 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 152542 T224 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 152542 T224 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 152543 T224 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 152543 T224 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 152544 T224 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 152545 T224 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 152545 T224 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 152545 T224 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 152546 T224 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 152561 T225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@305ab838[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 152571 T224 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 152572 T228 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:24683 collection:collection1 shard:shard1
   [junit4]   2> 152573 T178 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 152574 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 152574 T228 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 152718 T228 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 152759 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152759 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152759 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152759 T228 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 152760 T228 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C41 name=collection1 org.apache.solr.core.SolrCore@2a995a52 url=http://127.0.0.1:24683/collection1 node=127.0.0.1:24683_ C41_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:24683, core=collection1, node_name=127.0.0.1:24683_}
   [junit4]   2> 152760 T228 C41 P24683 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:24683/collection1/
   [junit4]   2> 152761 T228 C41 P24683 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 152761 T228 C41 P24683 oasc.SyncStrategy.syncToMe http://127.0.0.1:24683/collection1/ has no replicas
   [junit4]   2> 152761 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 152762 T228 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:24683/collection1/ shard1
   [junit4]   2> 152762 T228 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 152834 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152845 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 152845 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 152845 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 152950 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 152953 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 153003 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 153150 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153151 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153151 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153156 T228 oasc.ZkController.register We are http://127.0.0.1:24683/collection1/ and leader is http://127.0.0.1:24683/collection1/
   [junit4]   2> 153156 T228 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:24683
   [junit4]   2> 153156 T228 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 153156 T228 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 153157 T228 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 153216 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 153216 T228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 153216 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 153218 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 153219 T201 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:24683",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:24683_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 153280 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 153396 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153396 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153396 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 153590 T178 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 153591 T178 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 153593 T178 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61880
   [junit4]   2> 153594 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 153594 T178 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 153595 T178 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004
   [junit4]   2> 153595 T178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/'
   [junit4]   2> 153639 T178 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/solr.xml
   [junit4]   2> 153724 T178 oasc.CoreContainer.<init> New CoreContainer 885544864
   [junit4]   2> 153724 T178 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/]
   [junit4]   2> 153726 T178 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 153726 T178 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 153726 T178 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 153727 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 153727 T178 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 153727 T178 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 153727 T178 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 153728 T178 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 153728 T178 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 153729 T178 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 153729 T178 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 153729 T178 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 153730 T178 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22905/solr
   [junit4]   2> 153730 T178 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 153731 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 153740 T240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4cf0e472 name:ZooKeeperConnection Watcher:127.0.0.1:22905 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 153741 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 153803 T178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 153841 T242 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ad82934 name:ZooKeeperConnection Watcher:127.0.0.1:22905/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 153841 T178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 153988 T178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 154994 T178 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38907_
   [junit4]   2> 154997 T178 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38907_
   [junit4]   2> 155024 T200 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 155024 T223 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 155024 T210 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 155024 T242 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 155058 T243 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 155058 T243 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 155077 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 155077 T243 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 155077 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 155078 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 155080 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 155082 T201 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38907",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38907_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 155082 T201 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 155082 T201 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 155342 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 155380 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 155380 T242 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 155380 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 155380 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 156079 T243 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 156079 T243 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 156080 T243 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 156081 T243 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 156082 T243 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 156083 T243 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/collection1/'
   [junit4]   2> 156084 T243 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 156085 T243 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 156086 T243 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 156163 T243 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 156245 T243 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 156247 T243 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 156259 T243 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 156926 T243 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 156927 T243 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 156928 T243 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 156946 T243 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 156950 T243 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 156974 T243 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 156980 T243 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 156985 T243 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 156987 T243 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 156987 T243 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 156988 T243 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 156989 T243 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 156990 T243 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 156990 T243 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 156990 T243 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 156991 T243 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-004/collection1/, dataDir=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty2/
   [junit4]   2> 156991 T243 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@177730c8
   [junit4]   2> 156993 T243 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty2
   [junit4]   2> 156993 T243 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty2/index/
   [junit4]   2> 156994 T243 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 156994 T243 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.HttpPartitionTest-34B4BFCAADBE9F4C-001/tempDir-001/jetty2/index
   [junit4]   2> 156995 T243 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@775498f4
   [junit4]   2> 156996 T243 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@70f3dc37 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5305f860),segFN=segments_1,generation=1}
   [junit4]   2> 156996 T243 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 157008 T243 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 157009 T243 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 157009 T243 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 157009 T243 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 157010 T243 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 157010 T243 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 157011 T243 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 157011 T243 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 157012 T243 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 157014 T243 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 157016 T243 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 157016 T243 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 157017 T243 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 157019 T243 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 157019 T243 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 157021 T243 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 157040 T243 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 157046 T243 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 157046 T243 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 157049 T243 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=78.0400390625, floorSegmentMB=0.2568359375, forceMergeDeletesPctAllowed=25.220266225928427, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6623180696044442
   [junit4]   2> 157050 T243 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@70f3dc37 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5305f860),segFN=segments_1,generation=1}
   [junit4]   2> 157050 T243 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 157051 T243 oass.SolrIndexSearcher.<init> Opening Searcher@3c958935[collection1] main
   [junit4]   2> 157051 T243 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 157053 T243 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 157053 T243 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 157054 T243 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 157055 T243 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 157055 T243 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 157056 T243 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 157057 T243 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 157057 T243 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 157057 T243 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 157073 T244 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c958935[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 157081 T243 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 157082 T247 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38907 collection:collection1 shard:shard2
   [junit4]   2> 157083 T178 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
   [junit4]   2> 157084 T178 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 157084 T247 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 157086 T178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 157088 T178 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 157089 T178 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 157103 T178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 157206 T247 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 157271 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 157272 T247 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 157272 T247 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C42 name=collection1 org.apache.solr.core.SolrCore@7ea8cf04 url=http://127.0.0.1:38907/collection1 node=127.0.0.1:38907_ C42_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:38907, core=collection1, node_name=127.0.0.1:38907_}
   [junit4]   2> 157272 T247 C42 P38907 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38907/collection1/
   [junit4]   2> 157273 T247 C42 P38907 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 157274 T247 C42 P38907 oasc.SyncStrategy.syncToMe http://127.0.0.1:38907/collection1/ has no replicas
   [junit4]   2> 157274 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 157274 T247 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38907/collection1/ shard2
   [junit4]   2> 157275 T247 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 157460 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 157476 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157476 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157476 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157476 T242 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157511 T201 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 157531 T200 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 157672 T200 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157672 T223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157672 T242 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157672 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 157712 T247 oasc.ZkController.register We are http://127.0.0.1:38907/collection1/ and leader is http://127.0.0.1:3890

[...truncated too long message...]

 2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  288) Thread[id=26429, name=qtp1199347612-26429, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]   2> 	        at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:150)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:121)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]   2> 	        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
   [junit4]   2> 	        at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:618)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1040)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 	        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  289) Thread[id=26478, name=qtp1199347612-26478, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]   2> 	        at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:150)
   [junit4]   2> 	        at java.net.SocketInputStream.read(SocketInputStream.java:121)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]   2> 	        at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]   2> 	        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]   2> 	        at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]   2> 	        at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:391)
   [junit4]   2> 	        at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:141)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:227)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:618)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.fill(HttpParser.java:1040)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:280)
   [junit4]   2> 	        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 	        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 	        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 	        at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:670)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {text=PostingsFormat(name=MockRandom), id=PostingsFormat(name=Lucene41WithOrds)}, docValues:{}, sim=DefaultSimilarity, locale=sr_ME, timezone=Indian/Comoro
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=290,free=158689616,total=489488384
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrXml, TestDynamicFieldCollectionResource, TestSearcherReuse, TestRecovery, TestPartialUpdateDeduplication, RequestHandlersTest, TestAnalyzeInfixSuggestions, DebugComponentTest, TestExceedMaxTermLength, SpellCheckComponentTest, TestIndexSearcher, PathHierarchyTokenizerFactoryTest, HdfsWriteToMultipleCollectionsTest, TestSerializedLuceneMatchVersion, CurrencyFieldOpenExchangeTest, TestSweetSpotSimilarityFactory, HttpPartitionTest, TestCollationFieldDocValues, PrimUtilsTest, AlternateDirectoryTest, TestSimpleQParserPlugin, CollectionsAPIAsyncDistributedZkTest, TermVectorComponentDistributedTest, TestSchemaResource, TestFoldingMultitermQuery, MinimalSchemaTest, TestSolrXmlPersistence, ShardRoutingCustomTest, TestManagedResourceStorage, HdfsSyncSliceTest, TestDistributedGrouping, TestMaxScoreQueryParser, TestNumberUtils, TestFiltering, CursorPagingTest, TestRangeQuery, CoreAdminCreateDiscoverTest, DocValuesMultiTest, TestCSVLoader, CollectionsAPIDistributedZkTest, StatsComponentTest, TestDistributedMissingSort, SolrIndexConfigTest, DocumentAnalysisRequestHandlerTest, TestRandomMergePolicy, TestCSVResponseWriter, TestAnalyzedSuggestions, PreAnalyzedFieldTest, TestCollationField, HardAutoCommitTest, DirectSolrSpellCheckerTest, SortByFunctionTest, TestUpdate, ConvertedLegacyTest, TestMiniSolrCloudCluster, TestFastOutputStream, SuggesterTSTTest, DeleteShardTest, TestFieldTypeResource, TestCollationKeyRangeQueries, UniqFieldsUpdateProcessorFactoryTest, DistributedExpandComponentTest, SolrPluginUtilsTest, HdfsLockFactoryTest, SolrRequestParserTest, TestManagedSynonymFilterFactory, FullSolrCloudDistribCmdsTest, OpenExchangeRatesOrgProviderTest, TestInfoStreamLogging, TestHashPartitioner, DocValuesMissingTest, TestCoreDiscovery, TestDynamicFieldResource, DistanceFunctionTest, TestSolr4Spatial, TestComponentsName, TestNRTOpen, TestDFRSimilarityFactory, TestQueryTypes, DisMaxRequestHandlerTest, TestUtils, TestQuerySenderListener, TestSolrCoreProperties, TestShardHandlerFactory, PreAnalyzedUpdateProcessorTest, DeleteReplicaTest, FieldAnalysisRequestHandlerTest, XmlUpdateRequestHandlerTest, TestAtomicUpdateErrorCases, DateFieldTest, ShardSplitTest, TestSolrDeletionPolicy2, TestZkChroot, TestLazyCores, TestFastWriter, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OpenCloseCoreStressTest -Dtests.seed=34B4BFCAADBE9F4C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=sr_ME -Dtests.timezone=Indian/Comoro -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | OpenCloseCoreStressTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([34B4BFCAADBE9F4C]:0)
   [junit4] Completed on J1 in 7227.83s, 1 test, 2 errors <<< FAILURES!

[...truncated 13 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 396 suites, 1635 tests, 1 suite-level error, 2 errors, 1 failure, 32 ignored (3 assumptions)

Total time: 318 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 561 - Still Failing

Posted by Michael McCandless <lu...@mikemccandless.com>.
I opened https://issues.apache.org/jira/browse/LUCENE-5724

Mike McCandless

http://blog.mikemccandless.com


On Sun, Jun 1, 2014 at 4:56 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Now why on earth is this test using RAMDirectory ....
>
> I'll dig.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sun, Jun 1, 2014 at 4:28 AM, Apache Jenkins Server
> <je...@builds.apache.org> wrote:
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/561/
>>
>> 1 tests failed.
>> REGRESSION:  org.apache.lucene.index.Test2BPostings.test
>>
>> Error Message:
>> Java heap space
>>
>> Stack Trace:
>> java.lang.OutOfMemoryError: Java heap space
>>         at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
>>         at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
>>         at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
>>         at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
>>         at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
>>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
>>         at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
>>         at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
>>         at org.apache.lucene.store.Directory.copy(Directory.java:195)
>>         at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
>>         at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
>>         at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
>>         at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
>>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
>>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>>         at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
>>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
>>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
>>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
>>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
>>         at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 865 lines...]
>>    [junit4] Suite: org.apache.lucene.index.Test2BPostings
>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test2BPostings -Dtests.method=test -Dtests.seed=33378E77AE43B10B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=sl -Dtests.timezone=America/Argentina/ComodRivadavia -Dtests.file.encoding=UTF-8
>>    [junit4] ERROR    699s J0 | Test2BPostings.test <<<
>>    [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
>>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
>>    [junit4]    >        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
>>    [junit4]    >        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
>>    [junit4]    >        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
>>    [junit4]    >        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
>>    [junit4]    >        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
>>    [junit4]    >        at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
>>    [junit4]    >        at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
>>    [junit4]    >        at org.apache.lucene.store.Directory.copy(Directory.java:195)
>>    [junit4]    >        at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
>>    [junit4]    >        at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
>>    [junit4]    >        at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
>>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
>>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
>>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
>>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
>>    [junit4]    >        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
>>    [junit4]    >        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
>>    [junit4]    >        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
>>    [junit4]    >        at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
>>    [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J0/./lucene.index.Test2BPostings-33378E77AE43B10B-001
>>    [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=sl, timezone=America/Argentina/ComodRivadavia
>>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=456127264,total=468320256
>>    [junit4]   2> NOTE: All tests run in this JVM: [TestTransactionRollback, TestPhrasePrefixQuery, TestPrefixInBooleanQuery, TestMultiTermQueryRewrites, TestVirtualMethod, TestFilterIterator, TestDeterminizeLexicon, TestLucene45DocValuesFormat, TestMultiLevelSkipList, TestCustomNorms, Test2BSortedDocValues, TestDocTermOrds, TestTermRangeFilter, TestSpanFirstQuery, TestRegexpQuery, TestCachingWrapperFilter, TestAllFilesHaveCodecHeader, TestParallelTermEnum, TestShardSearching, TestReuseDocsEnum, TestTermsEnum, TestSortRandom, TestBooleanMinShouldMatch, TestArrayUtil, TestLucene41StoredFieldsFormat, TestMmapDirectory, TestRollback, TestTermsEnum2, TestMockAnalyzer, TestBroadWord, TestSloppyPhraseQuery2, TestSizeBoundedForceMerge, TestIndexWriterOnDiskFull, TestFieldCacheTermsFilter, TestStressIndexing2, Test2BTerms, TestFastCompressionMode, TestTryDelete, TestFieldReuse, TestSort, TestPayloads, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestCustomSearcherSort, TestSameTokenSamePosition, TestIndexableBinaryStringTools, TestTermdocPerf, TestDeterminism, TestIndexWriterExceptions2, Test2BBinaryDocValues, TestDirectoryReader, TestTimSorter, TestSimpleExplanationsOfNonMatches, TestOmitTf, TestDocValuesWithThreads, TestVersion, TestMultiTermConstantScore, TestPostingsFormat, TestStressIndexing, TestSpansAdvanced2, TestCompiledAutomaton, TestByteSlices, TestPositiveScoresOnlyCollector, TestEliasFanoSequence, TestComplexExplanationsOfNonMatches, TestBytesRefHash, TestPagedBytes, TestSimpleExplanations, TestBinaryDocument, TestFileSwitchDirectory, TestSentinelIntSet, TestRollingBuffer, TestUTF32ToUTF8, FuzzyTermOnShortTermsTest, TestPhraseQuery, Test2BPostings]
>>    [junit4] Completed on J0 in 699.04s, 1 test, 1 error <<< FAILURES!
>>
>> [...truncated 855 lines...]
>>    [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J0-20140601_065554_386.sysout
>>    [junit4] >>> JVM J0: stdout (verbatim) ----
>>    [junit4] java.lang.OutOfMemoryError: Java heap space
>>    [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps/java_pid5079.hprof ...
>>    [junit4] Heap dump file created [425665048 bytes in 0.706 secs]
>>    [junit4] <<< JVM J0: EOF ----
>>
>> [...truncated 9 lines...]
>> BUILD FAILED
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
>> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 405 suites, 3219 tests, 1 error, 78 ignored (68 assumptions)
>>
>> Total time: 92 minutes 9 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 561 - Still Failing

Posted by Michael McCandless <lu...@mikemccandless.com>.
Now why on earth is this test using RAMDirectory ....

I'll dig.

Mike McCandless

http://blog.mikemccandless.com


On Sun, Jun 1, 2014 at 4:28 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/561/
>
> 1 tests failed.
> REGRESSION:  org.apache.lucene.index.Test2BPostings.test
>
> Error Message:
> Java heap space
>
> Stack Trace:
> java.lang.OutOfMemoryError: Java heap space
>         at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
>         at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
>         at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
>         at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
>         at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
>         at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
>         at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
>         at org.apache.lucene.store.Directory.copy(Directory.java:195)
>         at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
>         at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
>         at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
>         at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>         at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
>         at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>
>
>
>
> Build Log:
> [...truncated 865 lines...]
>    [junit4] Suite: org.apache.lucene.index.Test2BPostings
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test2BPostings -Dtests.method=test -Dtests.seed=33378E77AE43B10B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=sl -Dtests.timezone=America/Argentina/ComodRivadavia -Dtests.file.encoding=UTF-8
>    [junit4] ERROR    699s J0 | Test2BPostings.test <<<
>    [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
>    [junit4]    >        at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
>    [junit4]    >        at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
>    [junit4]    >        at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
>    [junit4]    >        at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
>    [junit4]    >        at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
>    [junit4]    >        at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
>    [junit4]    >        at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
>    [junit4]    >        at org.apache.lucene.store.Directory.copy(Directory.java:195)
>    [junit4]    >        at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
>    [junit4]    >        at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
>    [junit4]    >        at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
>    [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J0/./lucene.index.Test2BPostings-33378E77AE43B10B-001
>    [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=sl, timezone=America/Argentina/ComodRivadavia
>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=456127264,total=468320256
>    [junit4]   2> NOTE: All tests run in this JVM: [TestTransactionRollback, TestPhrasePrefixQuery, TestPrefixInBooleanQuery, TestMultiTermQueryRewrites, TestVirtualMethod, TestFilterIterator, TestDeterminizeLexicon, TestLucene45DocValuesFormat, TestMultiLevelSkipList, TestCustomNorms, Test2BSortedDocValues, TestDocTermOrds, TestTermRangeFilter, TestSpanFirstQuery, TestRegexpQuery, TestCachingWrapperFilter, TestAllFilesHaveCodecHeader, TestParallelTermEnum, TestShardSearching, TestReuseDocsEnum, TestTermsEnum, TestSortRandom, TestBooleanMinShouldMatch, TestArrayUtil, TestLucene41StoredFieldsFormat, TestMmapDirectory, TestRollback, TestTermsEnum2, TestMockAnalyzer, TestBroadWord, TestSloppyPhraseQuery2, TestSizeBoundedForceMerge, TestIndexWriterOnDiskFull, TestFieldCacheTermsFilter, TestStressIndexing2, Test2BTerms, TestFastCompressionMode, TestTryDelete, TestFieldReuse, TestSort, TestPayloads, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestCustomSearcherSort, TestSameTokenSamePosition, TestIndexableBinaryStringTools, TestTermdocPerf, TestDeterminism, TestIndexWriterExceptions2, Test2BBinaryDocValues, TestDirectoryReader, TestTimSorter, TestSimpleExplanationsOfNonMatches, TestOmitTf, TestDocValuesWithThreads, TestVersion, TestMultiTermConstantScore, TestPostingsFormat, TestStressIndexing, TestSpansAdvanced2, TestCompiledAutomaton, TestByteSlices, TestPositiveScoresOnlyCollector, TestEliasFanoSequence, TestComplexExplanationsOfNonMatches, TestBytesRefHash, TestPagedBytes, TestSimpleExplanations, TestBinaryDocument, TestFileSwitchDirectory, TestSentinelIntSet, TestRollingBuffer, TestUTF32ToUTF8, FuzzyTermOnShortTermsTest, TestPhraseQuery, Test2BPostings]
>    [junit4] Completed on J0 in 699.04s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 855 lines...]
>    [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J0-20140601_065554_386.sysout
>    [junit4] >>> JVM J0: stdout (verbatim) ----
>    [junit4] java.lang.OutOfMemoryError: Java heap space
>    [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps/java_pid5079.hprof ...
>    [junit4] Heap dump file created [425665048 bytes in 0.706 secs]
>    [junit4] <<< JVM J0: EOF ----
>
> [...truncated 9 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 405 suites, 3219 tests, 1 error, 78 ignored (68 assumptions)
>
> Total time: 92 minutes 9 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 561 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/561/

1 tests failed.
REGRESSION:  org.apache.lucene.index.Test2BPostings.test

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
	at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
	at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
	at org.apache.lucene.store.Directory.copy(Directory.java:195)
	at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
	at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
	at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)




Build Log:
[...truncated 865 lines...]
   [junit4] Suite: org.apache.lucene.index.Test2BPostings
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test2BPostings -Dtests.method=test -Dtests.seed=33378E77AE43B10B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=sl -Dtests.timezone=America/Argentina/ComodRivadavia -Dtests.file.encoding=UTF-8
   [junit4] ERROR    699s J0 | Test2BPostings.test <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([33378E77AE43B10B:BB63B1AD00BFDCF3]:0)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:77)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:50)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:172)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:158)
   [junit4]    > 	at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:131)
   [junit4]    > 	at org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput.writeBytes(CompoundFileWriter.java:365)
   [junit4]    > 	at org.apache.lucene.store.DataOutput.copyBytes(DataOutput.java:255)
   [junit4]    > 	at org.apache.lucene.store.Directory.copy(Directory.java:195)
   [junit4]    > 	at org.apache.lucene.store.MockDirectoryWrapper.copy(MockDirectoryWrapper.java:954)
   [junit4]    > 	at org.apache.lucene.store.TrackingDirectoryWrapper.copy(TrackingDirectoryWrapper.java:50)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4673)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:485)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:452)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:374)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:466)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1498)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1194)
   [junit4]    > 	at org.apache.lucene.index.Test2BPostings.test(Test2BPostings.java:77)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/J0/./lucene.index.Test2BPostings-33378E77AE43B10B-001
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=sl, timezone=America/Argentina/ComodRivadavia
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=456127264,total=468320256
   [junit4]   2> NOTE: All tests run in this JVM: [TestTransactionRollback, TestPhrasePrefixQuery, TestPrefixInBooleanQuery, TestMultiTermQueryRewrites, TestVirtualMethod, TestFilterIterator, TestDeterminizeLexicon, TestLucene45DocValuesFormat, TestMultiLevelSkipList, TestCustomNorms, Test2BSortedDocValues, TestDocTermOrds, TestTermRangeFilter, TestSpanFirstQuery, TestRegexpQuery, TestCachingWrapperFilter, TestAllFilesHaveCodecHeader, TestParallelTermEnum, TestShardSearching, TestReuseDocsEnum, TestTermsEnum, TestSortRandom, TestBooleanMinShouldMatch, TestArrayUtil, TestLucene41StoredFieldsFormat, TestMmapDirectory, TestRollback, TestTermsEnum2, TestMockAnalyzer, TestBroadWord, TestSloppyPhraseQuery2, TestSizeBoundedForceMerge, TestIndexWriterOnDiskFull, TestFieldCacheTermsFilter, TestStressIndexing2, Test2BTerms, TestFastCompressionMode, TestTryDelete, TestFieldReuse, TestSort, TestPayloads, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestCustomSearcherSort, TestSameTokenSamePosition, TestIndexableBinaryStringTools, TestTermdocPerf, TestDeterminism, TestIndexWriterExceptions2, Test2BBinaryDocValues, TestDirectoryReader, TestTimSorter, TestSimpleExplanationsOfNonMatches, TestOmitTf, TestDocValuesWithThreads, TestVersion, TestMultiTermConstantScore, TestPostingsFormat, TestStressIndexing, TestSpansAdvanced2, TestCompiledAutomaton, TestByteSlices, TestPositiveScoresOnlyCollector, TestEliasFanoSequence, TestComplexExplanationsOfNonMatches, TestBytesRefHash, TestPagedBytes, TestSimpleExplanations, TestBinaryDocument, TestFileSwitchDirectory, TestSentinelIntSet, TestRollingBuffer, TestUTF32ToUTF8, FuzzyTermOnShortTermsTest, TestPhraseQuery, Test2BPostings]
   [junit4] Completed on J0 in 699.04s, 1 test, 1 error <<< FAILURES!

[...truncated 855 lines...]
   [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/core/test/temp/junit4-J0-20140601_065554_386.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/heapdumps/java_pid5079.hprof ...
   [junit4] Heap dump file created [425665048 bytes in 0.706 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 405 suites, 3219 tests, 1 error, 78 ignored (68 assumptions)

Total time: 92 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 560 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/560/

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard2 is not consistent.  Got 330 from https://127.0.0.1:26600/n/we/collection1lastClient and got 325 from https://127.0.0.1:16488/n/we/collection1

Stack Trace:
java.lang.AssertionError: shard2 is not consistent.  Got 330 from https://127.0.0.1:26600/n/we/collection1lastClient and got 325 from https://127.0.0.1:16488/n/we/collection1
	at __randomizedtesting.SeedInfo.seed([98CBB41A9456CBB3:192D3A02E309AB8F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1118)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([98CBB41A9456CBB3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([98CBB41A9456CBB3]:0)




Build Log:
[...truncated 11776 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/init-core-data-001
   [junit4]   2> 3142046 T20436 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 3142047 T20436 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /n/we
   [junit4]   2> 3142054 T20436 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 3142054 T20436 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3142055 T20437 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3142156 T20436 oasc.ZkTestServer.run start zk server on port:37458
   [junit4]   2> 3142156 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3142185 T20443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@417cab66 name:ZooKeeperConnection Watcher:127.0.0.1:37458 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3142185 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3142185 T20436 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3142190 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3142192 T20445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69fca8e2 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3142192 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3142192 T20436 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3142195 T20436 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3142198 T20436 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3142200 T20436 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3142211 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3142211 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3142225 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 3142225 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3142330 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3142330 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3142333 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3142334 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3142343 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3142344 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3142346 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3142347 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3142349 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3142350 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3142353 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3142354 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3142356 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3142357 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3142365 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3142366 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3142368 T20436 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3142369 T20436 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3142372 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3142381 T20447 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@757b0844 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3142382 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3143570 T20436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3143574 T20436 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3143576 T20436 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:65501
   [junit4]   2> 3143578 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3143578 T20436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3143578 T20436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002
   [junit4]   2> 3143579 T20436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/'
   [junit4]   2> 3143606 T20436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/solr.xml
   [junit4]   2> 3143675 T20436 oasc.CoreContainer.<init> New CoreContainer 1271674526
   [junit4]   2> 3143675 T20436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/]
   [junit4]   2> 3143677 T20436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3143677 T20436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3143677 T20436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3143677 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3143678 T20436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3143678 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3143678 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3143678 T20436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3143679 T20436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3143682 T20436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3143682 T20436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3143682 T20436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3143683 T20436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37458/solr
   [junit4]   2> 3143683 T20436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3143683 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3143686 T20458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d5e9c18 name:ZooKeeperConnection Watcher:127.0.0.1:37458 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3143687 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3143700 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3143701 T20460 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@224839b4 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3143702 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3143703 T20436 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3143706 T20436 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3143709 T20436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3143712 T20436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3143715 T20436 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3143718 T20436 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3143719 T20436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65501_n%2Fwe
   [junit4]   2> 3143721 T20436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65501_n%2Fwe
   [junit4]   2> 3143723 T20436 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3143725 T20436 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3143736 T20436 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:65501_n%2Fwe
   [junit4]   2> 3143736 T20436 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3143738 T20436 oasc.Overseer.start Overseer (id=91847557880086532-127.0.0.1:65501_n%2Fwe-n_0000000000) starting
   [junit4]   2> 3143748 T20436 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3143763 T20462 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3143764 T20436 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3143765 T20436 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3143766 T20436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3143768 T20461 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3143773 T20463 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 3143773 T20463 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3143774 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3143774 T20463 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3143775 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3143775 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65501/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65501_n%2Fwe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3143775 T20461 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 3143776 T20461 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3143777 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3143778 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3144775 T20463 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3144775 T20463 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 3144776 T20463 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3144776 T20463 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3144777 T20463 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3144777 T20463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/collection1/'
   [junit4]   2> 3144778 T20463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 3144779 T20463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 3144780 T20463 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 3144863 T20463 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 3144941 T20463 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3145043 T20463 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3145054 T20463 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3145657 T20463 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3145662 T20463 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3145664 T20463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3145677 T20463 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3145696 T20463 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3145696 T20463 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3145696 T20463 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/
   [junit4]   2> 3145697 T20463 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3de2f652
   [junit4]   2> 3145698 T20463 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data
   [junit4]   2> 3145700 T20463 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/index/
   [junit4]   2> 3145701 T20463 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3145702 T20463 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/index
   [junit4]   2> 3145702 T20463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@464e5ef9
   [junit4]   2> 3145708 T20463 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3145708 T20463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3145712 T20463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3145713 T20463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3145713 T20463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3145713 T20463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3145714 T20463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3145714 T20463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3145714 T20463 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3145715 T20463 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3145715 T20463 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3145715 T20463 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3145716 T20463 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3145716 T20463 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3145717 T20463 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3145717 T20463 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3145718 T20463 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3145718 T20463 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3145736 T20463 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3145740 T20463 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3145741 T20463 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3145742 T20463 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=11.6865234375, floorSegmentMB=1.7421875, forceMergeDeletesPctAllowed=5.921752442431799, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2868122748611447
   [junit4]   2> 3145743 T20463 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3145744 T20463 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3145744 T20463 oass.SolrIndexSearcher.<init> Opening Searcher@4a6a77f3[collection1] main
   [junit4]   2> 3145744 T20463 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3145745 T20463 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3145745 T20463 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3145746 T20463 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3145746 T20463 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3145746 T20463 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3145747 T20463 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3145747 T20463 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3145747 T20463 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 3145748 T20463 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3145751 T20464 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a6a77f3[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3145753 T20463 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3145754 T20467 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:65501/n/we collection:control_collection shard:shard1
   [junit4]   2> 3145754 T20436 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 3145755 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3145755 T20467 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3145760 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3145773 T20469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@514e82d5 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3145773 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3145779 T20436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3145787 T20467 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3145789 T20436 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3145790 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145790 T20467 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3145790 T20467 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2209 name=collection1 org.apache.solr.core.SolrCore@14bcf2bb url=https://127.0.0.1:65501/n/we/collection1 node=127.0.0.1:65501_n%2Fwe C2209_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:65501/n/we, core=collection1, node_name=127.0.0.1:65501_n%2Fwe}
   [junit4]   2> 3145790 T20467 C2209 P65501 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:65501/n/we/collection1/
   [junit4]   2> 3145791 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3145791 T20467 C2209 P65501 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3145791 T20467 C2209 P65501 oasc.SyncStrategy.syncToMe https://127.0.0.1:65501/n/we/collection1/ has no replicas
   [junit4]   2> 3145791 T20467 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:65501/n/we/collection1/ shard1
   [junit4]   2> 3145792 T20467 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3145801 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145812 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3145812 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3145828 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3145831 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145934 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3145934 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3145981 T20467 oasc.ZkController.register We are https://127.0.0.1:65501/n/we/collection1/ and leader is https://127.0.0.1:65501/n/we/collection1/
   [junit4]   2> 3145981 T20467 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:65501/n/we
   [junit4]   2> 3145981 T20467 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3145981 T20467 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 3145981 T20467 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3145983 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145983 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145983 T20467 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3145983 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3145985 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3145985 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:65501/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65501_n%2Fwe",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3145994 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3146097 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3146097 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3146831 T20436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 3146832 T20436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3146836 T20436 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3146837 T20436 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:39637
   [junit4]   2> 3146839 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3146839 T20436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3146839 T20436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003
   [junit4]   2> 3146840 T20436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/'
   [junit4]   2> 3146868 T20436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/solr.xml
   [junit4]   2> 3146936 T20436 oasc.CoreContainer.<init> New CoreContainer 1046203395
   [junit4]   2> 3146937 T20436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/]
   [junit4]   2> 3146938 T20436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3146938 T20436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3146939 T20436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3146939 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3146939 T20436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3146939 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3146939 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3146940 T20436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3146940 T20436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3146943 T20436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3146943 T20436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3146944 T20436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3146944 T20436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37458/solr
   [junit4]   2> 3146944 T20436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3146945 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3146973 T20480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@433fbb72 name:ZooKeeperConnection Watcher:127.0.0.1:37458 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3146973 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3146988 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3146989 T20482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33fc9299 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3146990 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3147015 T20436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3148018 T20436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39637_n%2Fwe
   [junit4]   2> 3148020 T20436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39637_n%2Fwe
   [junit4]   2> 3148022 T20460 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3148022 T20482 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3148022 T20469 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 3148043 T20483 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3148043 T20483 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3148044 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3148044 T20483 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3148045 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3148045 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3148046 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3148047 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39637/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39637_n%2Fwe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3148047 T20461 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 3148047 T20461 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 3148050 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3148059 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3148059 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3148059 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3149046 T20483 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3149046 T20483 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3149047 T20483 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3149047 T20483 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3149048 T20483 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3149048 T20483 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/collection1/'
   [junit4]   2> 3149050 T20483 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 3149051 T20483 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 3149052 T20483 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 3149128 T20483 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 3149204 T20483 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3149305 T20483 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3149315 T20483 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3149925 T20483 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3149930 T20483 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3149932 T20483 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3149937 T20483 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3149956 T20483 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3149956 T20483 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3149957 T20483 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/
   [junit4]   2> 3149957 T20483 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3de2f652
   [junit4]   2> 3149958 T20483 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1
   [junit4]   2> 3149959 T20483 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/index/
   [junit4]   2> 3149960 T20483 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 3149962 T20483 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/index
   [junit4]   2> 3149963 T20483 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2d26acd5
   [junit4]   2> 3149967 T20483 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 3149968 T20483 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3149973 T20483 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3149973 T20483 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3149974 T20483 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3149974 T20483 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3149974 T20483 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3149975 T20483 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3149975 T20483 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3149976 T20483 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3149976 T20483 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3149977 T20483 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3149977 T20483 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3149977 T20483 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3149978 T20483 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3149978 T20483 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3149979 T20483 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3149980 T20483 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3149998 T20483 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3150002 T20483 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3150002 T20483 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3150004 T20483 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=11.6865234375, floorSegmentMB=1.7421875, forceMergeDeletesPctAllowed=5.921752442431799, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2868122748611447
   [junit4]   2> 3150005 T20483 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 3150005 T20483 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3150006 T20483 oass.SolrIndexSearcher.<init> Opening Searcher@a335a41[collection1] main
   [junit4]   2> 3150006 T20483 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3150007 T20483 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3150007 T20483 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3150008 T20483 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3150008 T20483 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3150008 T20483 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3150009 T20483 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3150009 T20483 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3150010 T20483 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 3150010 T20483 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3150014 T20484 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a335a41[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3150017 T20483 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3150018 T20487 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:39637/n/we collection:collection1 shard:shard3
   [junit4]   2> 3150018 T20436 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 3150019 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3150019 T20487 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 3150109 T20487 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 3150140 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150140 T20487 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3150140 T20487 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2210 name=collection1 org.apache.solr.core.SolrCore@7ba1ffee url=https://127.0.0.1:39637/n/we/collection1 node=127.0.0.1:39637_n%2Fwe C2210_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:39637/n/we, core=collection1, node_name=127.0.0.1:39637_n%2Fwe}
   [junit4]   2> 3150140 T20487 C2210 P39637 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:39637/n/we/collection1/
   [junit4]   2> 3150141 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3150141 T20487 C2210 P39637 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3150141 T20487 C2210 P39637 oasc.SyncStrategy.syncToMe https://127.0.0.1:39637/n/we/collection1/ has no replicas
   [junit4]   2> 3150142 T20487 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:39637/n/we/collection1/ shard3
   [junit4]   2> 3150142 T20487 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 3150164 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150165 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150165 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150165 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150184 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3150187 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150294 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150294 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150294 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150337 T20487 oasc.ZkController.register We are https://127.0.0.1:39637/n/we/collection1/ and leader is https://127.0.0.1:39637/n/we/collection1/
   [junit4]   2> 3150337 T20487 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:39637/n/we
   [junit4]   2> 3150337 T20487 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3150337 T20487 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 3150338 T20487 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3150339 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150340 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150340 T20487 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3150340 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150341 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3150342 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:39637/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:39637_n%2Fwe",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3150352 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3150457 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150457 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3150457 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3151005 T20436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 3151006 T20436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3151010 T20436 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3151011 T20436 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31063
   [junit4]   2> 3151013 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3151013 T20436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3151014 T20436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004
   [junit4]   2> 3151014 T20436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/'
   [junit4]   2> 3151042 T20436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/solr.xml
   [junit4]   2> 3151110 T20436 oasc.CoreContainer.<init> New CoreContainer 882252243
   [junit4]   2> 3151111 T20436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/]
   [junit4]   2> 3151112 T20436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3151112 T20436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3151113 T20436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3151113 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3151113 T20436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3151113 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3151114 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3151114 T20436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3151114 T20436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3151117 T20436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3151118 T20436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3151118 T20436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3151118 T20436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37458/solr
   [junit4]   2> 3151118 T20436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3151119 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3151127 T20498 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a161cb8 name:ZooKeeperConnection Watcher:127.0.0.1:37458 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3151128 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3151131 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3151138 T20500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76fb710 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3151138 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3151154 T20436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3152157 T20436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31063_n%2Fwe
   [junit4]   2> 3152166 T20436 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31063_n%2Fwe
   [junit4]   2> 3152169 T20460 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3152169 T20469 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3152169 T20482 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3152169 T20500 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 3152190 T20501 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3152190 T20501 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3152191 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3152192 T20501 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3152192 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3152192 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3152193 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3152193 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31063/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:31063_n%2Fwe",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3152194 T20461 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3152194 T20461 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3152197 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3152206 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3152206 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3152206 T20500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3152206 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3153193 T20501 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3153193 T20501 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3153194 T20501 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3153194 T20501 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3153194 T20501 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3153195 T20501 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/collection1/'
   [junit4]   2> 3153196 T20501 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 3153197 T20501 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 3153197 T20501 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 3153273 T20501 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 3153347 T20501 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3153448 T20501 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3153458 T20501 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3154058 T20501 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3154062 T20501 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3154064 T20501 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3154069 T20501 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3154087 T20501 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3154087 T20501 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3154087 T20501 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/
   [junit4]   2> 3154088 T20501 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3de2f652
   [junit4]   2> 3154089 T20501 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2
   [junit4]   2> 3154090 T20501 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/index/
   [junit4]   2> 3154091 T20501 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 3154093 T20501 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/index
   [junit4]   2> 3154093 T20501 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@72dcc48e
   [junit4]   2> 3154098 T20501 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 3154098 T20501 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3154103 T20501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3154104 T20501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3154104 T20501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3154104 T20501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3154105 T20501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3154105 T20501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3154105 T20501 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3154106 T20501 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3154106 T20501 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3154107 T20501 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3154107 T20501 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3154108 T20501 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3154108 T20501 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3154109 T20501 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3154109 T20501 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3154110 T20501 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3154128 T20501 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3154132 T20501 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3154133 T20501 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3154134 T20501 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=11.6865234375, floorSegmentMB=1.7421875, forceMergeDeletesPctAllowed=5.921752442431799, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2868122748611447
   [junit4]   2> 3154135 T20501 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 3154136 T20501 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3154136 T20501 oass.SolrIndexSearcher.<init> Opening Searcher@6995d179[collection1] main
   [junit4]   2> 3154136 T20501 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3154137 T20501 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3154137 T20501 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3154138 T20501 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3154138 T20501 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3154138 T20501 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3154139 T20501 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3154139 T20501 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3154139 T20501 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 3154140 T20501 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3154144 T20502 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6995d179[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 3154147 T20501 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3154148 T20505 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:31063/n/we collection:collection1 shard:shard1
   [junit4]   2> 3154148 T20436 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
   [junit4]   2> 3154148 T20505 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3154149 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3154165 T20505 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3154174 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154174 T20505 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3154174 T20505 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2211 name=collection1 org.apache.solr.core.SolrCore@78cdcad2 url=https://127.0.0.1:31063/n/we/collection1 node=127.0.0.1:31063_n%2Fwe C2211_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:31063/n/we, core=collection1, node_name=127.0.0.1:31063_n%2Fwe}
   [junit4]   2> 3154175 T20505 C2211 P31063 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:31063/n/we/collection1/
   [junit4]   2> 3154175 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3154175 T20505 C2211 P31063 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3154175 T20505 C2211 P31063 oasc.SyncStrategy.syncToMe https://127.0.0.1:31063/n/we/collection1/ has no replicas
   [junit4]   2> 3154176 T20505 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:31063/n/we/collection1/ shard1
   [junit4]   2> 3154176 T20505 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3154182 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154183 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154183 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154183 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154183 T20500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154200 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3154203 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154306 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154306 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154306 T20500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154306 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154353 T20505 oasc.ZkController.register We are https://127.0.0.1:31063/n/we/collection1/ and leader is https://127.0.0.1:31063/n/we/collection1/
   [junit4]   2> 3154353 T20505 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:31063/n/we
   [junit4]   2> 3154353 T20505 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3154353 T20505 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 3154353 T20505 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3154355 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154355 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154355 T20505 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3154355 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154356 T20461 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3154356 T20461 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31063/n/we",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:31063_n%2Fwe",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 3154364 T20460 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3154467 T20460 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154467 T20469 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154467 T20482 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3154467 T20500 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3155389 T20436 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 3155390 T20436 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3155396 T20436 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 3155398 T20436 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:39948
   [junit4]   2> 3155400 T20436 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3155401 T20436 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3155401 T20436 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-005
   [junit4]   2> 3155402 T20436 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-005/'
   [junit4]   2> 3155445 T20436 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-005/solr.xml
   [junit4]   2> 3155561 T20436 oasc.CoreContainer.<init> New CoreContainer 1987435229
   [junit4]   2> 3155562 T20436 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-98CBB41A9456CBB3-001/tempDir-005/]
   [junit4]   2> 3155564 T20436 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3155564 T20436 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3155565 T20436 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3155565 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3155565 T20436 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3155566 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3155566 T20436 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3155567 T20436 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3155567 T20436 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3155571 T20436 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3155572 T20436 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3155572 T20436 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3155573 T20436 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:37458/solr
   [junit4]   2> 3155573 T20436 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3155574 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3155603 T20516 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3eb6be72 name:ZooKeeperConnection Watcher:127.0.0.1:37458 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3155603 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3155615 T20436 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3155617 T20518 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@339ab3e0 name:ZooKeeperConnection Watcher:127.0.0.1:37458/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3155617 T20436 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3155639 T20436 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3156641 T20436 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39948_n%2Fwe
   [junit4]   2> 3156643 T20436 oascc.SolrZkClient.makePath makePath: /live_n

[...truncated too long message...]

concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 	  35) Thread[id=1293, name=TEST-CollectionsAPIDistributedZkTest.testDistribSearch-seed#[98CBB41A9456CBB3], state=BLOCKED, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]   2> 	        at org.apache.solr.update.DefaultSolrCoreState.cancelRecovery(DefaultSolrCoreState.java:339)
   [junit4]   2> 	        at org.apache.solr.core.CoreContainer.cancelCoreRecoveries(CoreContainer.java:432)
   [junit4]   2> 	        at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:370)
   [junit4]   2> 	        at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:200)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:223)
   [junit4]   2> 	        at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:560)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1454)
   [junit4]   2> 	        at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:286)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:231)
   [junit4]   2> 	        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1435)
   [junit4]   2> 	        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.tearDown(CollectionsAPIDistributedZkTest.java:1319)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=es_DO, timezone=Europe/Zagreb
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=37,free=68909752,total=318439424
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, TestReload, TriLevelCompositeIdRoutingTest, TestCloudManagedSchema, AsyncMigrateRouteKeyTest, SolrPluginUtilsTest, TestMaxScoreQueryParser, TestPartialUpdateDeduplication, TestMiniSolrCloudCluster, TestAtomicUpdateErrorCases, TestRandomMergePolicy, DirectSolrSpellCheckerTest, TestShardHandlerFactory, TestManagedResource, StatsComponentTest, SolrCoreCheckLockOnStartupTest, OutputWriterTest, TestFastLRUCache, TestInfoStreamLogging, TestCollationField, DistributedDebugComponentTest, DateFieldTest, CollectionsAPIDistributedZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.seed=98CBB41A9456CBB3 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_DO -Dtests.timezone=Europe/Zagreb -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | CollectionsAPIDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([98CBB41A9456CBB3]:0)
   [junit4] Completed on J1 in 7223.90s, 1 test, 2 errors <<< FAILURES!

[...truncated 415 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 396 suites, 1640 tests, 1 suite-level error, 1 error, 1 failure, 502 ignored (1 assumption)

Total time: 252 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 559 - Still Failing

Posted by Robert Muir <rc...@gmail.com>.
I think its the Chuck Norris object: and he can do this even without G1GC

On Tue, May 27, 2014 at 11:09 AM, Dawid Weiss
<da...@cs.put.poznan.pl> wrote:
>> java.lang.ArrayStoreException: <unknown>
>>         at __randomizedtesting.SeedInfo.seed([97ABA0C8320EBCCB:E5E785C7836E0AB8]:0)
>>         at org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.add(RamUsageEstimator.java:674)
>
> Oh boy... what the hell is this? :)
>
> D.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 559 - Still Failing

Posted by Dawid Weiss <da...@cs.put.poznan.pl>.
> java.lang.ArrayStoreException: <unknown>
>         at __randomizedtesting.SeedInfo.seed([97ABA0C8320EBCCB:E5E785C7836E0AB8]:0)
>         at org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.add(RamUsageEstimator.java:674)

Oh boy... what the hell is this? :)

D.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 559 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/559/

1 tests failed.
REGRESSION:  org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom

Error Message:
<unknown>

Stack Trace:
java.lang.ArrayStoreException: <unknown>
	at __randomizedtesting.SeedInfo.seed([97ABA0C8320EBCCB:E5E785C7836E0AB8]:0)
	at org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.add(RamUsageEstimator.java:674)
	at org.apache.lucene.util.RamUsageEstimator.measureObjectSize(RamUsageEstimator.java:420)
	at org.apache.lucene.util.RamUsageEstimator.sizeOf(RamUsageEstimator.java:333)
	at org.apache.lucene.search.FieldCache$CacheEntry.estimateSize(FieldCache.java:739)
	at org.apache.lucene.util.FieldCacheSanityChecker.check(FieldCacheSanityChecker.java:104)
	at org.apache.lucene.util.FieldCacheSanityChecker.checkSanity(FieldCacheSanityChecker.java:87)
	at org.apache.lucene.util.LuceneTestCase.assertSaneFieldCaches(LuceneTestCase.java:781)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:55)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 8692 lines...]
   [junit4] Suite: org.apache.lucene.search.grouping.GroupFacetCollectorTest
   [junit4]   2> *** BEGIN testRandom(org.apache.lucene.search.grouping.GroupFacetCollectorTest): FieldCache ***
   [junit4]   2> 'ParallelAtomicReader(FieldFilterAtomicReader(reader=SlowCompositeReaderWrapper(ParallelCompositeReader(ParallelCompositeReader(ParallelAtomicReader(_0(4.9):c817) ParallelAtomicReader(_1(4.9):C473)))), fields=[facet]), FieldFilterAtomicReader(reader=SlowCompositeReaderWrapper(ParallelCompositeReader(ParallelCompositeReader(ParallelAtomicReader(_0(4.9):c817) ParallelAtomicReader(_1(4.9):C473)))), fields=![facet]))'=>'facet',class org.apache.lucene.index.DocTermOrds,null=>org.apache.lucene.index.DocTermOrds#736721523
   [junit4]   2> 'ParallelAtomicReader(FieldFilterAtomicReader(reader=SlowCompositeReaderWrapper(ParallelCompositeReader(ParallelCompositeReader(ParallelAtomicReader(_0(4.9):c817) ParallelAtomicReader(_1(4.9):C473)))), fields=[facet]), FieldFilterAtomicReader(reader=SlowCompositeReaderWrapper(ParallelCompositeReader(ParallelCompositeReader(ParallelAtomicReader(_0(4.9):c817) ParallelAtomicReader(_1(4.9):C473)))), fields=![facet]))'=>'group',class org.apache.lucene.index.SortedDocValues,0.5=>org.apache.lucene.search.FieldCacheImpl$SortedDocValuesImpl#1290203164
   [junit4]   2> *** END testRandom(org.apache.lucene.search.grouping.GroupFacetCollectorTest): FieldCache ***
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=GroupFacetCollectorTest -Dtests.method=testRandom -Dtests.seed=97ABA0C8320EBCCB -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ms_MY -Dtests.timezone=Europe/Uzhgorod -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   15.1s J1 | GroupFacetCollectorTest.testRandom <<<
   [junit4]    > Throwable #1: java.lang.ArrayStoreException: <unknown>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([97ABA0C8320EBCCB:E5E785C7836E0AB8]:0)
   [junit4]    > 	at org.apache.lucene.util.RamUsageEstimator$IdentityHashSet.add(RamUsageEstimator.java:674)
   [junit4]    > 	at org.apache.lucene.util.RamUsageEstimator.measureObjectSize(RamUsageEstimator.java:420)
   [junit4]    > 	at org.apache.lucene.util.RamUsageEstimator.sizeOf(RamUsageEstimator.java:333)
   [junit4]    > 	at org.apache.lucene.search.FieldCache$CacheEntry.estimateSize(FieldCache.java:739)
   [junit4]    > 	at org.apache.lucene.util.FieldCacheSanityChecker.check(FieldCacheSanityChecker.java:104)
   [junit4]    > 	at org.apache.lucene.util.FieldCacheSanityChecker.checkSanity(FieldCacheSanityChecker.java:87)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.assertSaneFieldCaches(LuceneTestCase.java:781)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build/grouping/test/J1/./lucene.search.grouping.GroupFacetCollectorTest-97ABA0C8320EBCCB-001
   [junit4]   2> NOTE: test params are: codec=Lucene46: {group=PostingsFormat(name=MockRandom), content=Pulsing41(freqCutoff=18 minBlockSize=86 maxBlockSize=174), hotel=PostingsFormat(name=FSTOrdPulsing41), duration=PostingsFormat(name=FSTOrdPulsing41), facet=PostingsFormat(name=MockRandom), x=PostingsFormat(name=FSTOrdPulsing41), airport=Pulsing41(freqCutoff=18 minBlockSize=86 maxBlockSize=174)}, docValues:{group_dv=DocValuesFormat(name=Memory), hotel_dv=DocValuesFormat(name=Disk), duration_dv=DocValuesFormat(name=Disk), airport_dv=DocValuesFormat(name=Memory), facet_dv=DocValuesFormat(name=Memory)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {content=DFR I(F)B3(800.0)}, locale=ms_MY, timezone=Europe/Uzhgorod
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=131987976,total=184942592
   [junit4]   2> NOTE: All tests run in this JVM: [AllGroupsCollectorTest, GroupFacetCollectorTest]
   [junit4] Completed on J1 in 15.90s, 3 tests, 1 error <<< FAILURES!

[...truncated 10 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:539: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:2017: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/module-build.xml:60: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1297: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:921: There were test failures: 6 suites, 12 tests, 1 error

Total time: 119 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure