You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/01/06 10:12:54 UTC

[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.6.0) - Build # 1167 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1167/
Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:485)         at java.util.TimerThread.mainLoop(Timer.java:483)         at java.util.TimerThread.run(Timer.java:462)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        at java.util.TimerThread.run(Timer.java:462)
	at __randomizedtesting.SeedInfo.seed([F0307FD98AEF0997]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]         at java.lang.Object.wait(Native Method)         at java.lang.Object.wait(Object.java:485)         at java.util.TimerThread.mainLoop(Timer.java:483)         at java.util.TimerThread.run(Timer.java:462)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        at java.util.TimerThread.run(Timer.java:462)
	at __randomizedtesting.SeedInfo.seed([F0307FD98AEF0997]:0)




Build Log:
[...truncated 9617 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 856021 T2673 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 856026 T2673 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1388999353934
   [junit4]   2> 856027 T2673 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 856028 T2674 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 856129 T2673 oasc.ZkTestServer.run start zk server on port:53068
   [junit4]   2> 856131 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856138 T2680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55dd686a name:ZooKeeperConnection Watcher:127.0.0.1:53068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856138 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856139 T2673 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 856156 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856160 T2682 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a3449bb name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856161 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856161 T2673 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 856172 T2673 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 856181 T2673 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 856189 T2673 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 856200 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 856200 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 856216 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 856217 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 856228 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 856228 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 856238 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 856239 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 856249 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 856250 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 856263 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 856263 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 856278 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 856279 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 856290 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 856290 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 856301 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 856302 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 856313 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 856313 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 856335 T2673 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 856337 T2673 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 856648 T2673 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 856652 T2673 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53071
   [junit4]   2> 856654 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 856655 T2673 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 856655 T2673 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260
   [junit4]   2> 856656 T2673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/'
   [junit4]   2> 856686 T2673 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/solr.xml
   [junit4]   2> 856748 T2673 oasc.CoreContainer.<init> New CoreContainer 823223491
   [junit4]   2> 856749 T2673 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/]
   [junit4]   2> 856751 T2673 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 856751 T2673 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 856751 T2673 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 856752 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 856753 T2673 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 856753 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 856753 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 856754 T2673 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 856754 T2673 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 856776 T2673 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 856776 T2673 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 856777 T2673 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 856777 T2673 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53068/solr
   [junit4]   2> 856778 T2673 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 856779 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856785 T2693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cee575f name:ZooKeeperConnection Watcher:127.0.0.1:53068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856786 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856795 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856798 T2695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42840b43 name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856799 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856805 T2673 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 856818 T2673 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 856830 T2673 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 856837 T2673 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53071_
   [junit4]   2> 856840 T2673 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53071_
   [junit4]   2> 856852 T2673 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 856861 T2673 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 856876 T2673 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 856883 T2673 oasc.Overseer.start Overseer (id=91029461659811843-127.0.0.1:53071_-n_0000000000) starting
   [junit4]   2> 856895 T2673 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 856910 T2697 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 856912 T2673 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 856922 T2673 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 856929 T2673 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 856942 T2696 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 856945 T2698 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 856945 T2698 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 856948 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 856949 T2698 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 856954 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 856956 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53071",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53071_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 856956 T2696 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 856957 T2696 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 856965 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 856968 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 857950 T2698 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 857950 T2698 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/collection1
   [junit4]   2> 857951 T2698 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 857953 T2698 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 857954 T2698 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 857958 T2698 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/collection1/'
   [junit4]   2> 857959 T2698 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/collection1/lib/classes/' to classloader
   [junit4]   2> 857960 T2698 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/collection1/lib/README' to classloader
   [junit4]   2> 858041 T2698 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 858090 T2698 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 858095 T2698 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 858103 T2698 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 858613 T2698 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 858614 T2698 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 858615 T2698 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 858629 T2698 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 858634 T2698 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 858692 T2698 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 858700 T2698 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 858707 T2698 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 858712 T2698 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 858713 T2698 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 858713 T2698 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 858718 T2698 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 858718 T2698 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 858718 T2698 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 858719 T2698 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1388999354260/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/control/data/
   [junit4]   2> 858719 T2698 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ff243df
   [junit4]   2> 858721 T2698 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/control/data
   [junit4]   2> 858721 T2698 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/control/data/index/
   [junit4]   2> 858721 T2698 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 858722 T2698 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/control/data/index
   [junit4]   2> 858722 T2698 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.03515625, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=10.602316259348633, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32001400600413704
   [junit4]   2> 858723 T2698 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4563a650 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cd408b7),segFN=segments_1,generation=1}
   [junit4]   2> 858723 T2698 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 858728 T2698 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 858728 T2698 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 858729 T2698 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 858729 T2698 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 858729 T2698 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 858729 T2698 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 858730 T2698 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 858730 T2698 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 858731 T2698 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 858731 T2698 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 858732 T2698 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 858732 T2698 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 858732 T2698 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 858733 T2698 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 858733 T2698 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 858734 T2698 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 858740 T2698 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 858744 T2698 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 858744 T2698 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 858745 T2698 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=55.650390625, floorSegmentMB=2.083984375, forceMergeDeletesPctAllowed=28.872202941960445, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5341143654781632
   [junit4]   2> 858746 T2698 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@4563a650 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5cd408b7),segFN=segments_1,generation=1}
   [junit4]   2> 858747 T2698 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 858747 T2698 oass.SolrIndexSearcher.<init> Opening Searcher@7422bef1 main
   [junit4]   2> 858753 T2699 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7422bef1 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 858756 T2698 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 858757 T2673 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 858758 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 858757 T2702 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53071 collection:control_collection shard:shard1
   [junit4]   2> 858760 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 858766 T2704 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c40449c name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 858766 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 858767 T2702 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 858775 T2673 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 858795 T2673 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 858810 T2702 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 858819 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 858827 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 858828 T2702 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 858828 T2702 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1818 name=collection1 org.apache.solr.core.SolrCore@6957ea0e url=http://127.0.0.1:53071/collection1 node=127.0.0.1:53071_ C1818_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:53071, core=collection1, node_name=127.0.0.1:53071_}
   [junit4]   2> 858829 T2702 C1818 P53071 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53071/collection1/
   [junit4]   2> 858829 T2702 C1818 P53071 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 858829 T2702 C1818 P53071 oasc.SyncStrategy.syncToMe http://127.0.0.1:53071/collection1/ has no replicas
   [junit4]   2> 858830 T2702 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53071/collection1/ shard1
   [junit4]   2> 858830 T2702 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 858852 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 858856 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 858859 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 858882 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 858891 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 858999 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 859000 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 859032 T2702 oasc.ZkController.register We are http://127.0.0.1:53071/collection1/ and leader is http://127.0.0.1:53071/collection1/
   [junit4]   2> 859032 T2702 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53071
   [junit4]   2> 859033 T2702 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 859033 T2702 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 859034 T2702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 859037 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 859037 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 859038 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 859044 T2702 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 859049 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 859051 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53071",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53071_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 859059 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 859139 T2673 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 859143 T2673 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53075
   [junit4]   2> 859146 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 859147 T2673 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 859148 T2673 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703
   [junit4]   2> 859148 T2673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/'
   [junit4]   2> 859166 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 859167 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 859187 T2673 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/solr.xml
   [junit4]   2> 859255 T2673 oasc.CoreContainer.<init> New CoreContainer 1588543918
   [junit4]   2> 859256 T2673 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/]
   [junit4]   2> 859257 T2673 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 859258 T2673 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 859258 T2673 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 859258 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 859258 T2673 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 859259 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 859259 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 859260 T2673 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 859260 T2673 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 859282 T2673 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 859282 T2673 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 859283 T2673 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 859283 T2673 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53068/solr
   [junit4]   2> 859284 T2673 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 859286 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 859290 T2715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44013712 name:ZooKeeperConnection Watcher:127.0.0.1:53068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 859292 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 859305 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 859309 T2717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1920e375 name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 859310 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 859324 T2673 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 860337 T2673 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53075_
   [junit4]   2> 860341 T2673 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53075_
   [junit4]   2> 860357 T2704 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 860358 T2695 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 860359 T2717 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 860384 T2718 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 860384 T2718 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 860387 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 860387 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 860388 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 860387 T2718 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 860398 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 860401 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53075",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53075_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 860401 T2696 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 860402 T2696 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 860411 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 860521 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 860521 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 860522 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 861392 T2718 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 861393 T2718 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/collection1
   [junit4]   2> 861393 T2718 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 861399 T2718 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 861399 T2718 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 861408 T2718 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/collection1/'
   [junit4]   2> 861409 T2718 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/collection1/lib/classes/' to classloader
   [junit4]   2> 861410 T2718 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/collection1/lib/README' to classloader
   [junit4]   2> 861491 T2718 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 861566 T2718 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 861571 T2718 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 861581 T2718 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 862239 T2718 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 862240 T2718 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 862241 T2718 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 862249 T2718 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 862253 T2718 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 862299 T2718 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 862320 T2718 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 862327 T2718 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 862332 T2718 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 862332 T2718 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 862333 T2718 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 862337 T2718 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 862338 T2718 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 862338 T2718 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 862339 T2718 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1388999356703/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty1/
   [junit4]   2> 862339 T2718 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ff243df
   [junit4]   2> 862340 T2718 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty1
   [junit4]   2> 862340 T2718 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty1/index/
   [junit4]   2> 862341 T2718 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 862341 T2718 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty1/index
   [junit4]   2> 862341 T2718 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.03515625, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=10.602316259348633, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32001400600413704
   [junit4]   2> 862342 T2718 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5baf594f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4463db73),segFN=segments_1,generation=1}
   [junit4]   2> 862343 T2718 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 862352 T2718 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 862352 T2718 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 862353 T2718 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 862353 T2718 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 862354 T2718 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 862354 T2718 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 862355 T2718 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 862355 T2718 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 862355 T2718 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 862356 T2718 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 862357 T2718 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 862357 T2718 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 862358 T2718 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 862359 T2718 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 862359 T2718 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 862360 T2718 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 862367 T2718 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 862371 T2718 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 862372 T2718 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 862373 T2718 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=55.650390625, floorSegmentMB=2.083984375, forceMergeDeletesPctAllowed=28.872202941960445, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5341143654781632
   [junit4]   2> 862373 T2718 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5baf594f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4463db73),segFN=segments_1,generation=1}
   [junit4]   2> 862374 T2718 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 862374 T2718 oass.SolrIndexSearcher.<init> Opening Searcher@1df3b58e main
   [junit4]   2> 862381 T2719 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1df3b58e main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 862384 T2718 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 862386 T2673 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 862386 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 862386 T2722 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53075 collection:collection1 shard:shard1
   [junit4]   2> 862391 T2722 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 862419 T2722 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 862426 T2722 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 862427 T2722 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1819 name=collection1 org.apache.solr.core.SolrCore@7192a096 url=http://127.0.0.1:53075/collection1 node=127.0.0.1:53075_ C1819_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53075, core=collection1, node_name=127.0.0.1:53075_}
   [junit4]   2> 862427 T2722 C1819 P53075 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53075/collection1/
   [junit4]   2> 862427 T2722 C1819 P53075 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 862428 T2722 C1819 P53075 oasc.SyncStrategy.syncToMe http://127.0.0.1:53075/collection1/ has no replicas
   [junit4]   2> 862428 T2722 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53075/collection1/ shard1
   [junit4]   2> 862429 T2722 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 862437 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862437 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862437 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862458 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 862469 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862482 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862484 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862484 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862505 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 862518 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862642 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862642 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862643 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862686 T2722 oasc.ZkController.register We are http://127.0.0.1:53075/collection1/ and leader is http://127.0.0.1:53075/collection1/
   [junit4]   2> 862686 T2722 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53075
   [junit4]   2> 862687 T2722 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 862687 T2722 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 862687 T2722 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 862690 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862690 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862691 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862693 T2722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 862701 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 862704 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53075",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53075_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 862715 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 862793 T2673 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 862801 T2673 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53078
   [junit4]   2> 862802 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 862803 T2673 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 862803 T2673 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294
   [junit4]   2> 862804 T2673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/'
   [junit4]   2> 862822 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862822 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862824 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 862854 T2673 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/solr.xml
   [junit4]   2> 862935 T2673 oasc.CoreContainer.<init> New CoreContainer 239698831
   [junit4]   2> 862936 T2673 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/]
   [junit4]   2> 862937 T2673 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 862939 T2673 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 862939 T2673 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 862940 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 862940 T2673 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 862941 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 862941 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 862942 T2673 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 862942 T2673 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 862968 T2673 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 862969 T2673 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 862970 T2673 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 862971 T2673 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53068/solr
   [junit4]   2> 862971 T2673 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 862973 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 863633 T2733 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c5c289c name:ZooKeeperConnection Watcher:127.0.0.1:53068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 863634 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 863646 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 863650 T2735 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17edf844 name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 863650 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 863667 T2673 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 864682 T2673 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53078_
   [junit4]   2> 864686 T2673 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53078_
   [junit4]   2> 864703 T2717 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 864706 T2695 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 864707 T2704 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 864709 T2735 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 864735 T2736 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 864735 T2736 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 864739 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 864739 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 864740 T2736 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 864740 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 864747 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 864749 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53078",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53078_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 864750 T2696 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 864750 T2696 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 864762 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 864766 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 864766 T2735 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 864767 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 864766 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 865742 T2736 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 865742 T2736 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/collection1
   [junit4]   2> 865742 T2736 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 865745 T2736 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 865745 T2736 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 865749 T2736 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/collection1/'
   [junit4]   2> 865751 T2736 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/collection1/lib/classes/' to classloader
   [junit4]   2> 865752 T2736 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/collection1/lib/README' to classloader
   [junit4]   2> 865821 T2736 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 865898 T2736 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 865906 T2736 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 865920 T2736 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 866609 T2736 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 866611 T2736 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 866612 T2736 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 866623 T2736 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 866628 T2736 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 866678 T2736 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 866701 T2736 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 866715 T2736 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 866723 T2736 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 866724 T2736 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 866724 T2736 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 866729 T2736 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 866729 T2736 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 866730 T2736 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 866730 T2736 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1388999360294/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty2/
   [junit4]   2> 866731 T2736 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ff243df
   [junit4]   2> 866732 T2736 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty2
   [junit4]   2> 866732 T2736 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty2/index/
   [junit4]   2> 866733 T2736 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 866733 T2736 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty2/index
   [junit4]   2> 866733 T2736 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=10.03515625, floorSegmentMB=1.466796875, forceMergeDeletesPctAllowed=10.602316259348633, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.32001400600413704
   [junit4]   2> 866735 T2736 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c84a268 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1386e5b4),segFN=segments_1,generation=1}
   [junit4]   2> 866735 T2736 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 866744 T2736 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 866745 T2736 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 866745 T2736 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 866746 T2736 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 866746 T2736 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 866747 T2736 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 866747 T2736 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 866748 T2736 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 866748 T2736 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 866749 T2736 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 866751 T2736 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 866751 T2736 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 866752 T2736 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 866753 T2736 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 866753 T2736 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 866754 T2736 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 866762 T2736 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 866767 T2736 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 866768 T2736 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 866769 T2736 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=55.650390625, floorSegmentMB=2.083984375, forceMergeDeletesPctAllowed=28.872202941960445, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5341143654781632
   [junit4]   2> 866770 T2736 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@2c84a268 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1386e5b4),segFN=segments_1,generation=1}
   [junit4]   2> 866770 T2736 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 866771 T2736 oass.SolrIndexSearcher.<init> Opening Searcher@467eb8eb main
   [junit4]   2> 866780 T2737 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@467eb8eb main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 866785 T2736 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 866787 T2673 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 866788 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 866788 T2740 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53078 collection:collection1 shard:shard2
   [junit4]   2> 866793 T2740 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 866820 T2740 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 866828 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 866832 T2740 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 866832 T2740 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1820 name=collection1 org.apache.solr.core.SolrCore@68c8af42 url=http://127.0.0.1:53078/collection1 node=127.0.0.1:53078_ C1820_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:53078, core=collection1, node_name=127.0.0.1:53078_}
   [junit4]   2> 866833 T2740 C1820 P53078 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53078/collection1/
   [junit4]   2> 866834 T2740 C1820 P53078 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 866834 T2740 C1820 P53078 oasc.SyncStrategy.syncToMe http://127.0.0.1:53078/collection1/ has no replicas
   [junit4]   2> 866834 T2740 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53078/collection1/ shard2
   [junit4]   2> 866835 T2740 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 866840 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 866852 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 866856 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 866857 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 866857 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 866868 T2735 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 866891 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 866898 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 866913 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867026 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867028 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867029 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867038 T2735 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867055 T2740 oasc.ZkController.register We are http://127.0.0.1:53078/collection1/ and leader is http://127.0.0.1:53078/collection1/
   [junit4]   2> 867055 T2740 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53078
   [junit4]   2> 867056 T2740 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 867057 T2740 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 867057 T2740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 867062 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867063 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867072 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 867074 T2740 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 867077 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53078",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53078_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 867090 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867203 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867203 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867203 T2735 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867207 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 867238 T2673 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 867244 T2673 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53081
   [junit4]   2> 867249 T2673 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 867249 T2673 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 867249 T2673 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696
   [junit4]   2> 867250 T2673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/'
   [junit4]   2> 867282 T2673 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/solr.xml
   [junit4]   2> 867355 T2673 oasc.CoreContainer.<init> New CoreContainer 622309918
   [junit4]   2> 867355 T2673 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/]
   [junit4]   2> 867357 T2673 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 867357 T2673 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 867357 T2673 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 867358 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 867359 T2673 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 867359 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 867359 T2673 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 867360 T2673 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 867360 T2673 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 867383 T2673 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 867383 T2673 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 867384 T2673 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 867384 T2673 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53068/solr
   [junit4]   2> 867384 T2673 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 867386 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 867391 T2751 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36aee298 name:ZooKeeperConnection Watcher:127.0.0.1:53068 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 867393 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 867404 T2673 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 867408 T2753 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15deb678 name:ZooKeeperConnection Watcher:127.0.0.1:53068/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 867409 T2673 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 867426 T2673 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 868440 T2673 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53081_
   [junit4]   2> 868443 T2673 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53081_
   [junit4]   2> 868457 T2704 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 868461 T2695 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 868463 T2735 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 868464 T2717 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 868465 T2753 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 868487 T2754 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 868487 T2754 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 868491 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 868491 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 868491 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 868492 T2754 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 868498 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 868500 T2696 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53081",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53081_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 868500 T2696 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 868500 T2696 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 868510 T2695 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 868622 T2704 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 868623 T2735 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 868623 T2717 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 868624 T2753 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 868625 T2695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 869495 T2754 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 869496 T2754 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/collection1
   [junit4]   2> 869496 T2754 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 869499 T2754 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 869500 T2754 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 869503 T2754 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/collection1/'
   [junit4]   2> 869505 T2754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/collection1/lib/classes/' to classloader
   [junit4]   2> 869505 T2754 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/collection1/lib/README' to classloader
   [junit4]   2> 869596 T2754 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
   [junit4]   2> 869670 T2754 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 869676 T2754 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 869689 T2754 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 870421 T2754 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 870423 T2754 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 870424 T2754 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 870435 T2754 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 870439 T2754 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 870501 T2754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 870527 T2754 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 870535 T2754 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 870540 T2754 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 870540 T2754 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 870540 T2754 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 870545 T2754 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 870546 T2754 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 870547 T2754 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 870547 T2754 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1388999364696/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1388999353933/jetty3/
   [junit4]   2> 870547 T2754 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7f

[...truncated too long message...]

rrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 121059 T2672 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 141119 T2672 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]   2> 	   1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:485)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:462)
   [junit4]   2> 141119 T2672 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
   [junit4]   2> 	   1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 144130 T2672 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
   [junit4]   2> 	   1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:485)
   [junit4]   2> 	        at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]   2> 	        at java.util.TimerThread.run(Timer.java:462)
   [junit4]   2> NOTE: test params are: codec=Lucene46: {foo_b=PostingsFormat(name=Memory doPackFST= true), n_f1=PostingsFormat(name=SimpleText), cat=PostingsFormat(name=TestBloomFilteredLucene41Postings), foo_d=PostingsFormat(name=Lucene41WithOrds), foo_f=PostingsFormat(name=Memory doPackFST= true), n_tl1=PostingsFormat(name=SimpleText), n_d1=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=PostingsFormat(name=Lucene41WithOrds), intDefault=PostingsFormat(name=SimpleText), n_td1=PostingsFormat(name=SimpleText), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), id=PostingsFormat(name=Lucene41WithOrds), range_facet_sl=PostingsFormat(name=Lucene41WithOrds), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), oddField_s=PostingsFormat(name=Lucene41WithOrds), sequence_i=PostingsFormat(name=Lucene41WithOrds), name=PostingsFormat(name=Lucene41WithOrds), foo_i=PostingsFormat(name=SimpleText), regex_dup_B_s=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_tf1=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_dt1=PostingsFormat(name=SimpleText), genre_s=PostingsFormat(name=Lucene41WithOrds), author_t=PostingsFormat(name=SimpleText), range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_ti1=PostingsFormat(name=Lucene41WithOrds), text=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=SimpleText), val_i=PostingsFormat(name=Lucene41WithOrds), SubjectTerms_mfacet=PostingsFormat(name=SimpleText), series_t=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_tdt1=PostingsFormat(name=SimpleText), regex_dup_A_s=PostingsFormat(name=Memory doPackFST= true), price=PostingsFormat(name=Memory doPackFST= true), other_tl1=PostingsFormat(name=SimpleText), n_l1=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=SimpleText), inStock=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=DefaultSimilarity, locale=ro_RO, timezone=Europe/Sarajevo
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Apple Inc. 1.6.0_65 (64-bit)/cpus=2,threads=1,free=200657368,total=438423552
   [junit4]   2> NOTE: All tests run in this JVM: [TestCopyFieldCollectionResource, SynonymTokenizerTest, TestCharFilters, SolrCoreTest, FileUtilsTest, SuggesterTest, TestIBSimilarityFactory, TestUtils, BasicFunctionalityTest, TestDistributedMissingSort, HighlighterConfigTest, DocValuesTest, TestReload, BasicZkTest, TestSolrCoreProperties, UpdateRequestProcessorFactoryTest, TestDistributedGrouping, DistributedSuggestComponentTest, TestRTGBase, SolrCmdDistributorTest, ZkSolrClientTest, TestSolrIndexConfig, TestDocSet, ScriptEngineTest, TestShardHandlerFactory, QueryEqualityTest, SpellCheckCollatorTest, TestSystemIdResolver, OpenCloseCoreStressTest, DirectSolrSpellCheckerTest, DOMUtilTest, TestSolrXMLSerializer, TestClassNameShortening, TestDistributedSearch, SolrIndexConfigTest, ChangedSchemaMergeTest, WordBreakSolrSpellCheckerTest, TestSolrQueryParserResource, ExternalFileFieldSortTest, TestCollapseQParserPlugin, RecoveryZkTest, TestSolrXml, IndexSchemaTest, TestLMDirichletSimilarityFactory, TestQueryTypes, ClusterStateUpdateTest, TestSolrXmlPersistence, TestHighFrequencyDictionaryFactory, TriLevelCompositeIdRoutingTest, TestXIncludeConfig, TestStressLucene, SuggesterTSTTest, TestAnalyzeInfixSuggestions, TestFieldTypeResource, TestIndexSearcher, DateFieldTest, SolrIndexSplitterTest, URLClassifyProcessorTest, TestFoldingMultitermQuery, SolrInfoMBeanTest, TestFieldCollectionResource, TestBinaryResponseWriter, TestWordDelimiterFilterFactory, BasicDistributedZk2Test, TestDFRSimilarityFactory, TestElisionMultitermQuery, TestHashPartitioner, RegexBoostProcessorTest, SyncSliceTest, TestPerFieldSimilarity, TestDefaultSimilarityFactory, TestSolrQueryParser, MinimalSchemaTest, BlockDirectoryTest, UUIDFieldTest, FastVectorHighlighterTest, TestNoOpRegenerator, TestRangeQuery, EchoParamsTest, DistanceFunctionTest, TestExtendedDismaxParser, PluginInfoTest, IndexBasedSpellCheckerTest, TestDynamicFieldResource, TestDynamicFieldCollectionResource, DirectUpdateHandlerTest, HardAutoCommitTest, OpenExchangeRatesOrgProviderTest, ZkControllerTest, TestBinaryField, NumericFieldsTest, NoCacheHeaderTest, ResponseLogComponentTest, TestStressVersions, TestUniqueKeyFieldResource, TestSchemaSimilarityResource, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=F0307FD98AEF0997 -Dtests.slow=true -Dtests.locale=ro_RO -Dtests.timezone=Europe/Sarajevo -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:485)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:462)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F0307FD98AEF0997]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=3016, name=Timer-7, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.lang.Object.wait(Native Method)
   [junit4]    >         at java.lang.Object.wait(Object.java:485)
   [junit4]    >         at java.util.TimerThread.mainLoop(Timer.java:483)
   [junit4]    >         at java.util.TimerThread.run(Timer.java:462)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F0307FD98AEF0997]:0)
   [junit4] Completed in 144.24s, 1 test, 2 errors <<< FAILURES!

[...truncated 750 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:459: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:439: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1307: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:940: There were test failures: 348 suites, 1522 tests, 2 suite-level errors, 1066 ignored (4 assumptions)

Total time: 52 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure