You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/07/24 01:31:16 UTC

[JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1437 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1437/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at java.net.PlainSocketImpl.socketConnect(Native Method)         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)         at java.net.Socket.connect(Socket.java:579)         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([662672A8316E3676]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at java.net.PlainSocketImpl.socketConnect(Native Method)         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)         at java.net.Socket.connect(Socket.java:579)         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:722)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
	at __randomizedtesting.SeedInfo.seed([662672A8316E3676]:0)




Build Log:
[...truncated 9689 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> 685870 T2365 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /o/b
   [junit4]   2> 685877 T2365 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374621690049
   [junit4]   2> 685878 T2365 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 685879 T2366 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 685980 T2365 oasc.ZkTestServer.run start zk server on port:11504
   [junit4]   2> 685981 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 685985 T2372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50ff2712 name:ZooKeeperConnection Watcher:127.0.0.1:11504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 685985 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 685986 T2365 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 686005 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 686007 T2374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6caa5e7a name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 686007 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 686008 T2365 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 686017 T2365 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 686020 T2365 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 686023 T2365 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 686031 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 686032 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 686045 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 686046 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 686169 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 686170 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 686174 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 686174 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 686183 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 686184 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 686187 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 686188 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 686191 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 686192 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 686195 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 686195 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 686198 T2365 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 686199 T2365 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 686528 T2365 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 686535 T2365 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11507
   [junit4]   2> 686535 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 686536 T2365 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 686537 T2365 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380
   [junit4]   2> 686537 T2365 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/'
   [junit4]   2> 686586 T2365 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/solr.xml
   [junit4]   2> 686697 T2365 oasc.CoreContainer.<init> New CoreContainer 876467191
   [junit4]   2> 686698 T2365 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/]
   [junit4]   2> 686700 T2365 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 686701 T2365 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 686701 T2365 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 686702 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 686702 T2365 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 686703 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 686703 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 686704 T2365 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 686705 T2365 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 686705 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 686725 T2365 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11504/solr
   [junit4]   2> 686725 T2365 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 686727 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 686738 T2385 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fdcf649 name:ZooKeeperConnection Watcher:127.0.0.1:11504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 686738 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 686761 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 686775 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 686777 T2387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cdb791a name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 686778 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 686797 T2365 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 686890 T2365 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 686893 T2365 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 686949 T2365 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11507_o%2Fb
   [junit4]   2> 686985 T2365 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11507_o%2Fb
   [junit4]   2> 687015 T2365 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 687078 T2365 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 687088 T2365 oasc.Overseer.start Overseer (id=90087207079510019-127.0.0.1:11507_o%2Fb-n_0000000000) starting
   [junit4]   2> 687103 T2365 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 687110 T2389 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 687111 T2365 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 687114 T2365 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 687121 T2365 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 687124 T2388 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 687130 T2390 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 687131 T2390 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 687132 T2390 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 688629 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 688630 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11507_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11507/o/b"}
   [junit4]   2> 688630 T2388 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
   [junit4]   2> 688630 T2388 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 688650 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 689135 T2390 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 689135 T2390 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/collection1
   [junit4]   2> 689135 T2390 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 689136 T2390 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 689136 T2390 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 689137 T2390 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/collection1/'
   [junit4]   2> 689138 T2390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/collection1/lib/classes/' to classloader
   [junit4]   2> 689139 T2390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/collection1/lib/README' to classloader
   [junit4]   2> 689184 T2390 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 689232 T2390 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 689335 T2390 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 689344 T2390 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 690013 T2390 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 690013 T2390 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 690014 T2390 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 690024 T2390 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 690028 T2390 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 690040 T2390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 690044 T2390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 690048 T2390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 690049 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 690049 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 690049 T2390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 690050 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 690050 T2390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 690051 T2390 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 690051 T2390 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374621690380/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/control/data/
   [junit4]   2> 690051 T2390 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24a06fb1
   [junit4]   2> 690052 T2390 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/control/data
   [junit4]   2> 690052 T2390 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/control/data/index/
   [junit4]   2> 690052 T2390 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 690053 T2390 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/control/data/index
   [junit4]   2> 690056 T2390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7302d80 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2570436),segFN=segments_1,generation=1}
   [junit4]   2> 690056 T2390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 690060 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 690060 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 690061 T2390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 690061 T2390 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 690062 T2390 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 690062 T2390 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 690062 T2390 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 690063 T2390 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 690063 T2390 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 690076 T2390 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 690081 T2390 oass.SolrIndexSearcher.<init> Opening Searcher@56436e68 main
   [junit4]   2> 690082 T2390 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 690083 T2390 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 690085 T2391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56436e68 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 690086 T2390 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 690087 T2390 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11507/o/b collection:control_collection shard:shard1
   [junit4]   2> 690087 T2390 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 690099 T2390 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 690119 T2390 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 690121 T2390 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 690122 T2390 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 690122 T2390 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11507/o/b/collection1/
   [junit4]   2> 690122 T2390 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 690122 T2390 oasc.SyncStrategy.syncToMe http://127.0.0.1:11507/o/b/collection1/ has no replicas
   [junit4]   2> 690123 T2390 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11507/o/b/collection1/ shard1
   [junit4]   2> 690123 T2390 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 690154 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 690160 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 690180 T2390 oasc.ZkController.register We are http://127.0.0.1:11507/o/b/collection1/ and leader is http://127.0.0.1:11507/o/b/collection1/
   [junit4]   2> 690180 T2390 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11507/o/b
   [junit4]   2> 690180 T2390 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 690180 T2390 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 690181 T2390 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 690182 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 690183 T2365 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 690184 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 690185 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 690195 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 690197 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 690199 T2394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5688f080 name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 690199 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 690200 T2365 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 690202 T2365 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 690455 T2365 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 690458 T2365 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11511
   [junit4]   2> 690459 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 690460 T2365 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 690460 T2365 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374
   [junit4]   2> 690461 T2365 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/'
   [junit4]   2> 690498 T2365 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/solr.xml
   [junit4]   2> 690582 T2365 oasc.CoreContainer.<init> New CoreContainer 1128781519
   [junit4]   2> 690583 T2365 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/]
   [junit4]   2> 690585 T2365 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 690585 T2365 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 690586 T2365 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 690586 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 690587 T2365 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 690587 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 690588 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 690588 T2365 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 690589 T2365 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 690589 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 690604 T2365 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11504/solr
   [junit4]   2> 690605 T2365 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 690606 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 690608 T2405 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d079622 name:ZooKeeperConnection Watcher:127.0.0.1:11504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 690609 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 690623 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 690631 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 690633 T2407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f7f7109 name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 690634 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 690644 T2365 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 691646 T2365 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11511_o%2Fb
   [junit4]   2> 691649 T2365 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11511_o%2Fb
   [junit4]   2> 691653 T2387 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 691653 T2407 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 691653 T2394 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 691653 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 691663 T2408 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 691664 T2408 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 691667 T2408 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 691668 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 691669 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11507_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11507/o/b"}
   [junit4]   2> 691747 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11511_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11511/o/b"}
   [junit4]   2> 691748 T2388 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
   [junit4]   2> 691748 T2388 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 691802 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 691802 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 691802 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 692669 T2408 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 692669 T2408 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/collection1
   [junit4]   2> 692669 T2408 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 692670 T2408 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 692671 T2408 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 692672 T2408 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/collection1/'
   [junit4]   2> 692673 T2408 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/collection1/lib/README' to classloader
   [junit4]   2> 692674 T2408 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/collection1/lib/classes/' to classloader
   [junit4]   2> 692726 T2408 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 692782 T2408 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 692884 T2408 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 692893 T2408 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 693417 T2408 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 693418 T2408 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 693419 T2408 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 693436 T2408 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 693441 T2408 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 693456 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 693461 T2408 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 693466 T2408 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 693467 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 693468 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 693468 T2408 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 693469 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 693469 T2408 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 693470 T2408 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 693470 T2408 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374621694374/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty1/
   [junit4]   2> 693471 T2408 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24a06fb1
   [junit4]   2> 693472 T2408 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty1
   [junit4]   2> 693472 T2408 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty1/index/
   [junit4]   2> 693473 T2408 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 693473 T2408 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty1/index
   [junit4]   2> 693477 T2408 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7b834c79 lockFactory=org.apache.lucene.store.NativeFSLockFactory@517fd46d),segFN=segments_1,generation=1}
   [junit4]   2> 693478 T2408 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 693483 T2408 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 693484 T2408 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 693485 T2408 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 693486 T2408 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 693486 T2408 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 693487 T2408 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 693487 T2408 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 693488 T2408 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 693488 T2408 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 693507 T2408 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 693515 T2408 oass.SolrIndexSearcher.<init> Opening Searcher@334319e6 main
   [junit4]   2> 693516 T2408 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 693517 T2408 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 693521 T2409 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@334319e6 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 693524 T2408 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 693524 T2408 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11511/o/b collection:collection1 shard:shard1
   [junit4]   2> 693524 T2408 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 693542 T2408 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 693576 T2408 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 693580 T2408 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 693580 T2408 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 693581 T2408 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11511/o/b/collection1/
   [junit4]   2> 693581 T2408 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 693581 T2408 oasc.SyncStrategy.syncToMe http://127.0.0.1:11511/o/b/collection1/ has no replicas
   [junit4]   2> 693581 T2408 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11511/o/b/collection1/ shard1
   [junit4]   2> 693582 T2408 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 694818 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 694842 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 694842 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 694842 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 694877 T2408 oasc.ZkController.register We are http://127.0.0.1:11511/o/b/collection1/ and leader is http://127.0.0.1:11511/o/b/collection1/
   [junit4]   2> 694877 T2408 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11511/o/b
   [junit4]   2> 694877 T2408 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 694877 T2408 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 694878 T2408 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 694880 T2408 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 694881 T2365 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 694881 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 694882 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 695144 T2365 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 695147 T2365 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11514
   [junit4]   2> 695148 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 695149 T2365 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 695149 T2365 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062
   [junit4]   2> 695150 T2365 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/'
   [junit4]   2> 695187 T2365 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/solr.xml
   [junit4]   2> 695274 T2365 oasc.CoreContainer.<init> New CoreContainer 2127990342
   [junit4]   2> 695275 T2365 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/]
   [junit4]   2> 695277 T2365 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 695277 T2365 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 695278 T2365 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 695278 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 695279 T2365 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 695279 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 695280 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 695280 T2365 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 695280 T2365 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 695281 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 695295 T2365 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11504/solr
   [junit4]   2> 695296 T2365 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 695297 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695300 T2421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f568e11 name:ZooKeeperConnection Watcher:127.0.0.1:11504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695300 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695312 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 695324 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695325 T2423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@82aa25f name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695326 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695335 T2365 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 696339 T2365 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11514_o%2Fb
   [junit4]   2> 696352 T2365 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11514_o%2Fb
   [junit4]   2> 696353 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 696356 T2394 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 696356 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11511_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11511/o/b"}
   [junit4]   2> 696356 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696356 T2407 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 696356 T2423 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 696357 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696375 T2424 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 696375 T2424 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 696377 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696377 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696377 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696378 T2424 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 696386 T2387 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 696386 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 696387 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697887 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 697889 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11514_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11514/o/b"}
   [junit4]   2> 697889 T2388 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 697890 T2388 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 697907 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697907 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697907 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 697907 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 698381 T2424 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 698381 T2424 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/collection1
   [junit4]   2> 698382 T2424 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 698383 T2424 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 698383 T2424 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 698384 T2424 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/collection1/'
   [junit4]   2> 698385 T2424 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/collection1/lib/README' to classloader
   [junit4]   2> 698386 T2424 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/collection1/lib/classes/' to classloader
   [junit4]   2> 698443 T2424 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 698504 T2424 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 698607 T2424 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 698619 T2424 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 699307 T2424 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 699308 T2424 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 699309 T2424 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 699322 T2424 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 699327 T2424 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 699343 T2424 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699349 T2424 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 699354 T2424 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699355 T2424 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699356 T2424 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699356 T2424 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 699357 T2424 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 699358 T2424 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 699358 T2424 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 699359 T2424 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374621699062/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty2/
   [junit4]   2> 699359 T2424 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24a06fb1
   [junit4]   2> 699360 T2424 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty2
   [junit4]   2> 699360 T2424 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty2/index/
   [junit4]   2> 699361 T2424 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 699361 T2424 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty2/index
   [junit4]   2> 699365 T2424 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6d718923 lockFactory=org.apache.lucene.store.NativeFSLockFactory@817f5a2),segFN=segments_1,generation=1}
   [junit4]   2> 699366 T2424 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 699372 T2424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 699372 T2424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 699373 T2424 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 699374 T2424 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 699374 T2424 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 699375 T2424 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 699375 T2424 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 699376 T2424 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 699376 T2424 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 699393 T2424 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 699401 T2424 oass.SolrIndexSearcher.<init> Opening Searcher@7a16bbb4 main
   [junit4]   2> 699403 T2424 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 699403 T2424 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 699408 T2425 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a16bbb4 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 699410 T2424 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 699411 T2424 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11514/o/b collection:collection1 shard:shard2
   [junit4]   2> 699411 T2424 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 699430 T2424 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 699455 T2424 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 699466 T2424 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 699467 T2424 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 699467 T2424 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11514/o/b/collection1/
   [junit4]   2> 699467 T2424 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 699468 T2424 oasc.SyncStrategy.syncToMe http://127.0.0.1:11514/o/b/collection1/ has no replicas
   [junit4]   2> 699468 T2424 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11514/o/b/collection1/ shard2
   [junit4]   2> 699468 T2424 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 700913 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700935 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 700935 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 700935 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 700935 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 700967 T2424 oasc.ZkController.register We are http://127.0.0.1:11514/o/b/collection1/ and leader is http://127.0.0.1:11514/o/b/collection1/
   [junit4]   2> 700967 T2424 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11514/o/b
   [junit4]   2> 700967 T2424 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 700968 T2424 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 700968 T2424 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 700970 T2424 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 700971 T2365 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 700972 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 700972 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 701259 T2365 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 701263 T2365 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11517
   [junit4]   2> 701264 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 701264 T2365 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 701265 T2365 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154
   [junit4]   2> 701266 T2365 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/'
   [junit4]   2> 701306 T2365 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/solr.xml
   [junit4]   2> 701398 T2365 oasc.CoreContainer.<init> New CoreContainer 2007632579
   [junit4]   2> 701399 T2365 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/]
   [junit4]   2> 701401 T2365 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 701401 T2365 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 701402 T2365 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 701403 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 701403 T2365 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 701404 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 701404 T2365 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 701405 T2365 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 701405 T2365 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 701406 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 701423 T2365 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11504/solr
   [junit4]   2> 701423 T2365 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 701424 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 701427 T2437 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a2e6210 name:ZooKeeperConnection Watcher:127.0.0.1:11504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 701428 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 701444 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 701456 T2365 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 701458 T2439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70066c70 name:ZooKeeperConnection Watcher:127.0.0.1:11504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 701458 T2365 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 701470 T2365 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 702446 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 702447 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11514_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11514/o/b"}
   [junit4]   2> 702468 T2439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702468 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702468 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702468 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702468 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702474 T2365 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11517_o%2Fb
   [junit4]   2> 702476 T2365 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11517_o%2Fb
   [junit4]   2> 702478 T2439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702478 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 702479 T2387 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 702479 T2407 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 702479 T2394 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 702480 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 702479 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 702480 T2439 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 702480 T2423 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 702480 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 702494 T2440 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 702495 T2440 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 702496 T2440 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 703976 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 703977 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11517_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11517/o/b"}
   [junit4]   2> 703977 T2388 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 703978 T2388 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 704008 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 704008 T2439 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 704008 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 704008 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 704008 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 704500 T2440 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 704500 T2440 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/collection1
   [junit4]   2> 704500 T2440 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 704501 T2440 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 704502 T2440 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 704503 T2440 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/collection1/'
   [junit4]   2> 704504 T2440 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/collection1/lib/classes/' to classloader
   [junit4]   2> 704505 T2440 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/collection1/lib/README' to classloader
   [junit4]   2> 704565 T2440 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 704629 T2440 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 704732 T2440 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 704744 T2440 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 705393 T2440 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 705394 T2440 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 705394 T2440 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 705403 T2440 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 705406 T2440 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 705418 T2440 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 705422 T2440 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 705426 T2440 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 705427 T2440 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 705427 T2440 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 705427 T2440 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 705428 T2440 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 705429 T2440 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 705429 T2440 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 705429 T2440 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374621705154/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty3/
   [junit4]   2> 705429 T2440 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@24a06fb1
   [junit4]   2> 705430 T2440 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty3
   [junit4]   2> 705431 T2440 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty3/index/
   [junit4]   2> 705431 T2440 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 705431 T2440 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374621690048/jetty3/index
   [junit4]   2> 705434 T2440 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@27b7c337 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c413e63),segFN=segments_1,generation=1}
   [junit4]   2> 705434 T2440 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 705438 T2440 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 705438 T2440 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 705439 T2440 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 705440 T2440 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 705440 T2440 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 705440 T2440 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 705441 T2440 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 705441 T2440 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 705441 T2440 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 705452 T2440 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 705457 T2440 oass.SolrIndexSearcher.<init> Opening Searcher@72205fdc main
   [junit4]   2> 705458 T2440 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 705458 T2440 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 705462 T2441 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72205fdc main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 705463 T2440 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 705464 T2440 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11517/o/b collection:collection1 shard:shard1
   [junit4]   2> 705464 T2440 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
   [junit4]   2> 705478 T2440 oasc.ZkController.register We are http://127.0.0.1:11517/o/b/collection1/ and leader is http://127.0.0.1:11511/o/b/collection1/
   [junit4]   2> 705479 T2440 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11517/o/b
   [junit4]   2> 705479 T2440 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 705479 T2440 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C588 name=collection1 org.apache.solr.core.SolrCore@7673826a url=http://127.0.0.1:11517/o/b/collection1 node=127.0.0.1:11517_o%2Fb C588_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:11517_o%2Fb, base_url=http://127.0.0.1:11517/o/b}
   [junit4]   2> 705479 T2442 C588 P11517 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 705480 T2440 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 705480 T2442 C588 P11517 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 705481 T2442 C588 P11517 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 705481 T2442 C588 P11517 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 705481 T2365 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 705482 T2365 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 705482 T2365 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 705482 T2442 C588 P11517 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 705500 T2397 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 705520 T2388 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 705521 T2388 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:11517_o%2Fb",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11517/o/b"}
   [junit4]   2> 705536 T2387 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 705536 T2394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 705536 T2423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 705536 T2407 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... 

[...truncated too long message...]

cksize=128), intDefault=MockFixedIntBlock(blockSize=1014), n_td1=PostingsFormat(name=SimpleText), timestamp=MockFixedIntBlock(blockSize=1014), id=Lucene41(blocksize=128), range_facet_sl=Lucene41(blocksize=128), range_facet_si=MockFixedIntBlock(blockSize=1014), oddField_s=Lucene41(blocksize=128), sequence_i=Lucene41(blocksize=128), name=Lucene41(blocksize=128), foo_i=PostingsFormat(name=SimpleText), regex_dup_B_s=MockFixedIntBlock(blockSize=1014), multiDefault=MockFixedIntBlock(blockSize=1014), n_tf1=MockFixedIntBlock(blockSize=1014), n_dt1=PostingsFormat(name=SimpleText), genre_s=Lucene41(blocksize=128), author_t=MockFixedIntBlock(blockSize=1014), range_facet_l=MockFixedIntBlock(blockSize=1014), n_ti1=Lucene41(blocksize=128), text=PostingsFormat(name=NestedPulsing), _version_=PostingsFormat(name=SimpleText), val_i=Lucene41(blocksize=128), SubjectTerms_mfacet=PostingsFormat(name=SimpleText), series_t=MockFixedIntBlock(blockSize=1014), a_t=MockFixedIntBlock(blockSize=1014), n_tdt1=PostingsFormat(name=SimpleText), regex_dup_A_s=PostingsFormat(name=NestedPulsing), price=PostingsFormat(name=NestedPulsing), other_tl1=MockFixedIntBlock(blockSize=1014), n_l1=MockFixedIntBlock(blockSize=1014), a_si=PostingsFormat(name=SimpleText), inStock=PostingsFormat(name=NestedPulsing)}, docValues:{}, sim=DefaultSimilarity, locale=vi_VN, timezone=Pacific/Tongatapu
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=177315104,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [TestLMDirichletSimilarityFactory, TestReplicationHandler, BlockCacheTest, SpellCheckCollatorTest, SearchHandlerTest, SolrCoreTest, TestRandomDVFaceting, PluginInfoTest, DocumentAnalysisRequestHandlerTest, TestLuceneMatchVersion, TestSolrQueryParserResource, TestStressLucene, RegexBoostProcessorTest, SolrIndexSplitterTest, StressHdfsTest, TestPropInjectDefaults, FileBasedSpellCheckerTest, TestHashPartitioner, TestRTGBase, TestDynamicFieldCollectionResource, TestGroupingSearch, TestZkChroot, TestPluginEnable, DirectUpdateHandlerOptimizeTest, TestLFUCache, TestIndexingPerformance, PreAnalyzedUpdateProcessorTest, TestFastOutputStream, TestSuggestSpellingConverter, CollectionsAPIDistributedZkTest, TestUniqueKeyFieldResource, ReturnFieldsTest, ZkCLITest, TestTrie, DistributedTermsComponentTest, EchoParamsTest, DirectUpdateHandlerTest, TestLazyCores, HighlighterTest, TestFuzzyAnalyzedSuggestions, TestFunctionQuery, TestRealTimeGet, SpellingQueryConverterTest, MoreLikeThisHandlerTest, DirectSolrConnectionTest, TestUpdate, TestSolrQueryParserDefaultOperatorResource, TestFoldingMultitermQuery, HdfsLockFactoryTest, TestCopyFieldCollectionResource, TestSurroundQueryParser, PingRequestHandlerTest, TestFieldTypeResource, TestDefaultSimilarityFactory, AssignTest, LeaderElectionIntegrationTest, BadIndexSchemaTest, TestPHPSerializedResponseWriter, UUIDFieldTest, HdfsBasicDistributedZkTest, TestPhraseSuggestions, LoggingHandlerTest, DateFieldTest, LeaderElectionTest, OverseerCollectionProcessorTest, TestDynamicFieldResource, TestSolrXml, TestJmxMonitoredMap, XsltUpdateRequestHandlerTest, HdfsRecoveryZkTest, IndexSchemaRuntimeFieldTest, TestConfig, TestCSVResponseWriter, UpdateParamsTest, TestLMJelinekMercerSimilarityFactory, TestValueSourceCache, TermVectorComponentTest, DistributedQueryElevationComponentTest, BasicDistributedZkTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=662672A8316E3676 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi_VN -Dtests.timezone=Pacific/Tongatapu -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   [junit4]    >    1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]    >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
   [junit4]    >         at java.net.Socket.connect(Socket.java:579)
   [junit4]    >         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    >         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    >         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    >         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:722)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([662672A8316E3676]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=2844, name=recoveryCmdExecutor-1465-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
   [junit4]    >         at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]    >         at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]    >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
   [junit4]    >         at java.net.Socket.connect(Socket.java:579)
   [junit4]    >         at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    >         at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    >         at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    >         at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    >         at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    >         at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    >         at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]    >         at java.lang.Thread.run(Thread.java:722)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([662672A8316E3676]:0)
   [junit4] Completed on J0 in 419.09s, 1 test, 2 errors <<< FAILURES!

[...truncated 487 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:395: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:375: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1250: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:893: There were test failures: 319 suites, 1345 tests, 2 suite-level errors, 605 ignored

Total time: 44 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure