You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/11/16 15:29:02 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2171 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2171/
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at __randomizedtesting.SeedInfo.seed([BBCE0E7280CAB87B]:0)
at java.lang.Thread.getStackTrace(Thread.java:1495)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:556)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1049)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:947)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:939)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:253)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:467)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:758)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:627)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:575)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:256)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:248)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6ac92e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6dbe2b55)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
Build Log:
[...truncated 9401 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 14595 T15 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 14609 T15 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1384609934072
[junit4] 2> 14624 T15 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 14637 T16 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 14938 T15 oasc.ZkTestServer.run start zk server on port:46548
[junit4] 2> 15449 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 15660 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c0c74fe name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 15661 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 15669 T15 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 15825 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 15828 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@643f58bb name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 15829 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 15837 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 15844 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 15851 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 15856 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 15863 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 15864 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 15882 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 15883 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 15891 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 15892 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 15898 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 15899 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 15905 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 15906 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 15912 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 15913 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 15920 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 15921 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 15927 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 15928 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 15937 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 15938 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 15945 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 15946 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 15952 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 15953 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 17221 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 17655 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16959
[junit4] 2> 17703 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 17704 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 17704 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421
[junit4] 2> 17705 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/'
[junit4] 2> 17734 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/solr.xml
[junit4] 2> 17823 T15 oasc.CoreContainer.<init> New CoreContainer 709323028
[junit4] 2> 17824 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/]
[junit4] 2> 17913 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 17913 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 17914 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 17914 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 17915 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 17915 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 17916 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 17916 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 17917 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 17926 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 17927 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 17927 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 17928 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
[junit4] 2> 17968 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 17969 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 17974 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4af6fd54 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 17975 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 18000 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 18002 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ce9927a name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 18002 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 18018 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 18039 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 18065 T15 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 18069 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16959_
[junit4] 2> 18079 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16959_
[junit4] 2> 18085 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 18122 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 18126 T15 oasc.Overseer.start Overseer (id=90741796671389699-127.0.0.1:16959_-n_0000000000) starting
[junit4] 2> 18168 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 18190 T40 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 18190 T15 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 18194 T15 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 18197 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 18235 T39 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 18251 T41 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 18251 T41 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 18254 T41 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 19747 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 19749 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:16959",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:16959_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 19750 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 19803 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 19833 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 20256 T41 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 20256 T41 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1
[junit4] 2> 20257 T41 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 20258 T41 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 20258 T41 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 20261 T41 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/'
[junit4] 2> 20262 T41 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/lib/classes/' to classloader
[junit4] 2> 20263 T41 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/lib/README' to classloader
[junit4] 2> 20365 T41 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 20437 T41 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 20440 T41 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 20528 T41 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 20687 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.ByteField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 20713 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.ShortField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 20754 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 20775 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 20796 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 20817 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
[junit4] 2> 21748 T41 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 21749 T41 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 21806 T41 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 21839 T41 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 21843 T41 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 22013 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 22028 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 22034 T41 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 22040 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 22040 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 22041 T41 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 22043 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 22043 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 22043 T41 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 22048 T41 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/
[junit4] 2> 22053 T41 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
[junit4] 2> 22072 T41 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data
[junit4] 2> 22072 T41 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index/
[junit4] 2> 22072 T41 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 22094 T41 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index
[junit4] 2> 22096 T41 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 22100 T41 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7f7c3056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476ff9e0),segFN=segments_1,generation=1}
[junit4] 2> 22101 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 22106 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 22210 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 22220 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 22220 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 22221 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 22221 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 22345 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 22346 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 22346 T41 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 22353 T41 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 22354 T41 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 22354 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 22355 T41 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 22356 T41 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 22357 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 22383 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 22390 T41 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 22429 T41 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 22430 T41 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 22431 T41 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 22483 T41 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7f7c3056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476ff9e0),segFN=segments_1,generation=1}
[junit4] 2> 22484 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 22485 T41 oass.SolrIndexSearcher.<init> Opening Searcher@695e0163 main
[junit4] 2> 22537 T42 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@695e0163 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 22545 T41 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 22545 T41 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16959 collection:control_collection shard:shard1
[junit4] 2> 22585 T41 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 22619 T41 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 22624 T41 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 22624 T41 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 22634 T41 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16959/collection1/
[junit4] 2> 22634 T41 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 22635 T41 oasc.SyncStrategy.syncToMe http://127.0.0.1:16959/collection1/ has no replicas
[junit4] 2> 22635 T41 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16959/collection1/ shard1
[junit4] 2> 22636 T41 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 22844 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 22864 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 22901 T41 oasc.ZkController.register We are http://127.0.0.1:16959/collection1/ and leader is http://127.0.0.1:16959/collection1/
[junit4] 2> 22901 T41 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16959
[junit4] 2> 22902 T41 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 22902 T41 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 22903 T41 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 22905 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 22907 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 22908 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 23033 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 23036 T45 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@642ddc4c name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 23036 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 23039 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 23047 T15 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 23443 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 23447 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35873
[junit4] 2> 23448 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 23448 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 23449 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508
[junit4] 2> 23450 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/'
[junit4] 2> 23498 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/solr.xml
[junit4] 2> 23602 T15 oasc.CoreContainer.<init> New CoreContainer 1950714347
[junit4] 2> 23603 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/]
[junit4] 2> 23605 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 23606 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 23607 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 23607 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 23608 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 23609 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 23610 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 23611 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 23611 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 23618 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 23619 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 23620 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 23621 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
[junit4] 2> 23621 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 23623 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 23632 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161def0 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 23632 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 23641 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 23644 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188d92e name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 23644 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 23657 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 24380 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 24382 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:16959",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:16959_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 24570 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 24570 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 24570 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 24664 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35873_
[junit4] 2> 24667 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35873_
[junit4] 2> 24671 T45 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 24671 T38 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 24671 T58 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 24692 T59 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 24693 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 24695 T59 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 26078 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 26080 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35873",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35873_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 26080 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 26081 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 26096 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 26096 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 26096 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 26697 T59 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 26697 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1
[junit4] 2> 26698 T59 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 26699 T59 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 26700 T59 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 26702 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/'
[junit4] 2> 26704 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/lib/classes/' to classloader
[junit4] 2> 26705 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/lib/README' to classloader
[junit4] 2> 26800 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 26884 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 26887 T59 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 26994 T59 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 27735 T59 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 27736 T59 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 27737 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 27754 T59 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 27758 T59 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 27788 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 27795 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 27800 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 27802 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 27803 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 27803 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 27805 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 27805 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 27805 T59 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 27806 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/
[junit4] 2> 27806 T59 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
[junit4] 2> 27808 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1
[junit4] 2> 27809 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index/
[junit4] 2> 27809 T59 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 27809 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index
[junit4] 2> 27810 T59 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 27811 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@74ba86ef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31d15a18),segFN=segments_1,generation=1}
[junit4] 2> 27812 T59 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 27819 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 27819 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 27820 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 27820 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 27821 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 27821 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 27822 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 27822 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 27822 T59 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 27824 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 27826 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 27826 T59 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 27827 T59 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 27828 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 27829 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 27831 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 27842 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 27848 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 27848 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 27850 T59 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 27902 T59 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@74ba86ef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31d15a18),segFN=segments_1,generation=1}
[junit4] 2> 27903 T59 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 27903 T59 oass.SolrIndexSearcher.<init> Opening Searcher@69365360 main
[junit4] 2> 27936 T60 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69365360 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 27942 T59 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 27943 T59 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35873 collection:collection1 shard:shard1
[junit4] 2> 27945 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 27970 T59 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 27974 T59 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 27974 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 27975 T59 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35873/collection1/
[junit4] 2> 27975 T59 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 27975 T59 oasc.SyncStrategy.syncToMe http://127.0.0.1:35873/collection1/ has no replicas
[junit4] 2> 27976 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35873/collection1/ shard1
[junit4] 2> 27976 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 29107 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 29138 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 29138 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 29138 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 29159 T59 oasc.ZkController.register We are http://127.0.0.1:35873/collection1/ and leader is http://127.0.0.1:35873/collection1/
[junit4] 2> 29159 T59 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35873
[junit4] 2> 29160 T59 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 29160 T59 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 29160 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 29164 T59 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 29166 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 29167 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 29458 T15 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 29461 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:23305
[junit4] 2> 29462 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 29463 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 29464 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630
[junit4] 2> 29464 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/'
[junit4] 2> 29505 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/solr.xml
[junit4] 2> 29592 T15 oasc.CoreContainer.<init> New CoreContainer 574978041
[junit4] 2> 29593 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/]
[junit4] 2> 29596 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 29597 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 29597 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 29598 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 29599 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 29599 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 29600 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 29601 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 29601 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 29608 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 29609 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 29610 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 29611 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
[junit4] 2> 29612 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 29613 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 29617 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f3c6654 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 29618 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 29628 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 29631 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31313e0e name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 29631 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 29644 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 30649 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 30650 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23305_
[junit4] 2> 30651 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:35873",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35873_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 30678 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23305_
[junit4] 2> 30694 T45 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 30694 T74 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 30694 T58 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 30695 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 30696 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 30696 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 30706 T38 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 30707 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 30713 T75 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 30713 T75 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 30716 T75 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 32209 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 32210 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:23305",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:23305_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 32211 T39 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 32211 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 32228 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 32228 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 32228 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 32228 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 32719 T75 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 32719 T75 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1
[junit4] 2> 32719 T75 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 32721 T75 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 32721 T75 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 32722 T75 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/'
[junit4] 2> 32725 T75 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/lib/classes/' to classloader
[junit4] 2> 32726 T75 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/lib/README' to classloader
[junit4] 2> 32792 T75 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 32849 T75 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 32851 T75 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 32911 T75 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 33462 T75 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 33462 T75 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 33464 T75 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 33481 T75 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 33485 T75 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 33517 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 33523 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 33529 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 33531 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 33531 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 33532 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 33534 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 33534 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 33534 T75 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 33535 T75 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/
[junit4] 2> 33535 T75 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
[junit4] 2> 33537 T75 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2
[junit4] 2> 33537 T75 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index/
[junit4] 2> 33537 T75 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 33538 T75 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index
[junit4] 2> 33539 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 33540 T75 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1dacecf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ee964fe),segFN=segments_1,generation=1}
[junit4] 2> 33540 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 33548 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 33549 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 33551 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 33551 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 33552 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 33552 T75 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 33554 T75 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 33556 T75 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 33556 T75 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 33557 T75 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 33559 T75 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 33559 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 33561 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 33568 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 33573 T75 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 33574 T75 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 33576 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 33628 T75 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1dacecf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ee964fe),segFN=segments_1,generation=1}
[junit4] 2> 33629 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 33629 T75 oass.SolrIndexSearcher.<init> Opening Searcher@69e1e999 main
[junit4] 2> 33643 T76 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69e1e999 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 33653 T75 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 33653 T75 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:23305 collection:collection1 shard:shard1
[junit4] 2> 33665 T75 oasc.ZkController.register We are http://127.0.0.1:23305/collection1/ and leader is http://127.0.0.1:35873/collection1/
[junit4] 2> 33666 T75 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23305
[junit4] 2> 33666 T75 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 33667 T75 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6e34b350 url=http://127.0.0.1:23305/collection1 node=127.0.0.1:23305_ C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:23305, core=collection1, node_name=127.0.0.1:23305_}
[junit4] 2> 33673 T77 C1 P23305 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 33674 T77 C1 P23305 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 33674 T75 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 33674 T77 C1 P23305 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 33675 T77 C1 P23305 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 33676 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4] 2> 33676 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 33681 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 33736 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 33737 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:23305",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:23305_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 33757 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 33757 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 33757 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 33757 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 33896 T15 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 23305
[junit4] 2> 33909 T15 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=574978041
[junit4] 2> 35186 T48 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 35188 T48 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
[junit4] 2> 35189 T48 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:23305_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=13
[junit4] 2> 35265 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 35267 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:23305",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:23305_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 35270 T39 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 35288 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 35288 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 35288 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 35289 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@2ac02d83 url=http://127.0.0.1:16959/collection1 node=127.0.0.1:16959_ C2_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:16959, core=collection1, node_name=127.0.0.1:16959_, leader=true}
[junit4] 2> 35545 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1451868768091242496)]} 0 215
[junit4] 2> 35553 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1451868768091242497)]} 0 223
[junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@129a41d2 url=http://127.0.0.1:35873/collection1 node=127.0.0.1:35873_ C3_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:35873, core=collection1, node_name=127.0.0.1:35873_, leader=true}
[junit4] 2> 35612 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1451868768245383168)]} 0 18
[junit4] 2> 35615 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1451868768245383169)]} 0 18
[junit4] 2> 35675 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1451868768313540608)]} 0 5
[junit4] 2> 35677 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1451868768313540609)]} 0 6
[junit4] 2> 35682 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1451868768321929216)]} 0 3
[junit4] 2> 35684 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1451868768322977792)]} 0 4
[junit4] 2> 35700 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1451868768339755008)]} 0 4
[junit4] 2> 35704 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1451868768343949312)]} 0 5
[junit4] 2> 35708 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1451868768348143616)]} 0 5
[junit4] 2> 35711 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1451868768352337920)]} 0 3
[junit4] 2> 35814 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1451868768461389824)]} 0 2
[junit4] 2> 35817 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1451868768464535552)]} 0 1
[junit4] 2> 35819 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1451868768467681280)]} 0 1
[junit4] 2> 35822 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1451868768470827008)]} 0 1
[junit4] 2> 35828 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1451868768473972736)]} 0 4
[junit4] 2> 35830 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1451868768476069888)]} 0 4
[junit4] 2> 35835 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1451868768482361344)]} 0 3
[junit4] 2> 35837 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1451868768483409920)]} 0 4
[junit4] 2> 35904 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1451868768553664512)]} 0 4
[junit4] 2> 35907 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1451868768555761664)]} 0 6
[junit4] 2> 35912 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1451868768562053120)]} 0 4
[junit4] 2> 35915 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1451868768565198848)]} 0 5
[junit4] 2> 35982 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1451868768635453440)]} 0 4
[junit4] 2> 35985 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1451868768638599168)]} 0 5
[junit4] 2> 35990 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1451868768643842048)]} 0 4
[junit4] 2> 35993 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1451868768646987776)]} 0 5
[junit4] 2> 36099 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1451868768758136832)]} 0 5
[junit4] 2> 36101 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1451868768760233984)]} 0 5
[junit4] 2> 36108 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1451868768767574016)]} 0 5
[junit4] 2> 36110 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1451868768768622592)]} 0 5
[junit4] 2> 36133 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1451868768793788416)]} 0 4
[junit4] 2> 36136 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1451868768795885568)]} 0 6
[junit4] 2> 36143 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1451868768804274176)]} 0 5
[junit4] 2> 36145 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1451868768805322752)]} 0 6
[junit4] 2> 36224 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1451868768890257408)]} 0 3
[junit4] 2> 36226 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1451868768891305984)]} 0 4
[junit4] 2> 36231 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1451868768897597440)]} 0 3
[junit4] 2> 36233 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1451868768898646016)]} 0 4
[junit4] 2> 36241 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1451868768908083200)]} 0 3
[junit4] 2> 36245 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1451868768911228928)]} 0 4
[junit4] 2> 36247 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1451868768914374656)]} 0 3
[junit4] 2> 36252 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1451868768919617536)]} 0 3
[junit4] 2> 36279 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1451868768946880512)]} 0 4
[junit4] 2> 36283 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1451868768952123392)]} 0 4
[junit4] 2> 36286 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1451868768955269120)]} 0 3
[junit4] 2> 36291 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1451868768959463424)]} 0 4
[junit4] 2> 36305 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1451868768974143488)]} 0 5
[junit4] 2> 36309 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1451868768979386368)]} 0 3
[junit4] 2> 36314 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1451868768983580672)]} 0 5
[junit4] 2> 36316 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1451868768985677824)]} 0 4
[junit4] 2> 36342 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1451868769011892224)]} 0 5
[junit4] 2> 36344 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1451868769013989376)]} 0 6
[junit4] 2> 36349 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1451868769021329408)]} 0 4
[junit4] 2> 36352 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1451868769023426560)]} 0 4
[junit4] 2> 36396 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1451868769068515328)]} 0 5
[junit4] 2> 36398 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1451868769071661056)]} 0 5
[junit4] 2> 36404 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1451868769077952512)]} 0 5
[junit4] 2> 36405 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1451868769080049664)]} 0 4
[junit4] 2> 36468 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1451868769148207104)]} 0 1
[junit4] 2> 36469 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1451868769149255680)]} 0 2
[junit4] 2> 36473 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1451868769153449984)]} 0 2
[junit4] 2> 36474 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1451868769154498560)]} 0 1
[junit4] 2> 36481 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1451868769159741441)]} 0 3
[junit4] 2> 36483 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1451868769159741440)]} 0 5
[junit4] 2> 36487 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1451868769166032896)]} 0 3
[junit4] 2> 36491 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1451868769169178624)]} 0 4
[junit4] 2> 36561 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1451868769244676096)]} 0 2
[junit4] 2> 36564 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1451868769248870400)]} 0 2
[junit4] 2> 36566 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1451868769250967552)]} 0 2
[junit4] 2> 36569 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1451868769254113280)]} 0 2
[junit4] 2> 36573 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1451868769256210432)]} 0 4
[junit4] 2> 36575 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1451868769258307584)]} 0 3
[junit4] 2> 36580 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1451868769262501888)]} 0 4
[junit4] 2> 36583 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1451868769265647616)]} 0 5
[junit4] 2> 36628 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1451868769315979264)]} 0 1
[junit4] 2> 36630 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1451868769318076416)]} 0 1
[junit4] 2> 36634 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1451868769322270720)]} 0 1
[junit4] 2> 36634 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1451868769322270721)]} 0 1
[junit4] 2> 36643 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1451868769328562176)]} 0 5
[junit4] 2> 36644 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1451868769328562177)]} 0 6
[junit4] 2> 36650 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1451868769336950784)]} 0 4
[junit4] 2> 36651 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1451868769337999360)]} 0 3
[junit4] 2> 36696 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1451868769385185280)]} 0 3
[junit4] 2> 36699 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1451868769387282432)]} 0 4
[junit4] 2> 36704 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1451868769392525312)]} 0 4
[junit4] 2> 36708 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1451868769396719616)]} 0 4
[junit4] 2> 36757 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1451868769450196992)]} 0 1
[junit4] 2> 36762 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1451868769456488448)]} 0 2
[junit4] 2> 36762 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1451868769456488448)]} 0 1
[junit4] 2> 36769 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1451868769462779904)]} 0 2
[junit4] 2> 36770 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1451868769461731328)]} 0 4
[junit4] 2> 36777 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1451868769469071360)]} 0 5
[junit4] 2> 36778 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1451868769470119936)]} 0 4
[junit4] 2> 36784 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1451868769476411392)]} 0 4
[junit4] 2> 36863 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1451868769560297472)]} 0 3
[junit4] 2> 36867 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1451868769564491776)]} 0 3
[junit4] 2> 36872 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1451868769568686080)]} 0 5
[junit4] 2> 36874 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1451868769571831808)]} 0 3
[junit4] 2> 36891 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1451868769588609024)]} 0 4
[junit4] 2> 36893 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1451868769590706176)]} 0 5
[junit4] 2> 36899 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1451868769596997632)]} 0 4
[junit4] 2> 36901 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1451868769599094784)]} 0 5
[junit4] 2> 36915 T15 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 36915 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 36918 T15 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
[junit4] 2> 36997 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (145186876970080
[...truncated too long message...]
ocketImpl.java:180)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
[junit4] 2> at java.net.Socket.connect(Socket.java:546)
[junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] 2> ... 19 more
[junit4] 2>
[junit4] 2> 92118 T97 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:23305 DONE. sync failed
[junit4] 2> 92119 T97 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4] 2> 92120 T97 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 92120 T97 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6c3b0b1e
[junit4] 2> 92149 T97 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=54,cumulative_deletesById=28,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
[junit4] 2> 92149 T97 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 92150 T97 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 92150 T97 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 92152 T97 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 92154 T97 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 92154 T97 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436;done=false>>]
[junit4] 2> 92155 T97 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436
[junit4] 2> 77926 T14 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> 92616 T97 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2;done=false>>]
[junit4] 2> 92616 T97 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2
[junit4] 2> 92617 T97 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 92617 T97 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:247)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:244)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:244)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Lucene46: {range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), text=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=Asserting), rnd_b=Lucene41(blocksize=128), intDefault=PostingsFormat(name=Asserting), id=Lucene41(blocksize=128), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_sl=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=Asserting), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_PR, timezone=America/Sitka
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=439169240,total=470548480
[junit4] 2> NOTE: All tests run in this JVM: [TestAddFieldRealTimeGet, RecoveryZkTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=BBCE0E7280CAB87B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PR -Dtests.timezone=America/Sitka -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4] > at __randomizedtesting.SeedInfo.seed([BBCE0E7280CAB87B]:0)
[junit4] > at java.lang.Thread.getStackTrace(Thread.java:1495)
[junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:556)
[junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1049)
[junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:947)
[junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:939)
[junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
[junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:253)
[junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:467)
[junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:758)
[junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:627)
[junit4] > at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
[junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:575)
[junit4] > at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:256)
[junit4] > at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:248)
[junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4] > Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6ac92e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6dbe2b55)
[junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4] > ... 1 more
[junit4] Completed on J0 in 78.75s, 1 test, 1 error <<< FAILURES!
[...truncated 1021 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:426: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:406: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:489: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1276: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:912: There were test failures: 340 suites, 1496 tests, 1 suite-level error, 35 ignored (6 assumptions)
Total time: 60 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2171 - Failure
Posted by Mark Miller <ma...@gmail.com>.
Hmm…looks new to me…
- Mark
On Nov 16, 2013, at 9:29 AM, Apache Jenkins Server <je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2171/
>
> 1 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
>
> Error Message:
> Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
> at __randomizedtesting.SeedInfo.seed([BBCE0E7280CAB87B]:0)
> at java.lang.Thread.getStackTrace(Thread.java:1495)
> at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
> at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:556)
> at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1049)
> at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:947)
> at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:939)
> at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
> at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
> at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:253)
> at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:467)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:758)
> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:627)
> at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:575)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:256)
> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:248)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6ac92e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6dbe2b55)
> at org.junit.Assert.fail(Assert.java:93)
> at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
> at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> ... 1 more
>
>
>
>
> Build Log:
> [...truncated 9401 lines...]
> [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
> [junit4] 2> 14595 T15 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
> [junit4] 2> 14609 T15 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
> [junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1384609934072
> [junit4] 2> 14624 T15 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 2> 14637 T16 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4] 2> 14938 T15 oasc.ZkTestServer.run start zk server on port:46548
> [junit4] 2> 15449 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 15660 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c0c74fe name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 15661 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 15669 T15 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 15825 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 15828 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@643f58bb name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 15829 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 15837 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4] 2> 15844 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4] 2> 15851 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4] 2> 15856 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4] 2> 15863 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
> [junit4] 2> 15864 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4] 2> 15882 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
> [junit4] 2> 15883 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4] 2> 15891 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 15892 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 15898 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4] 2> 15899 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4] 2> 15905 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4] 2> 15906 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4] 2> 15912 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4] 2> 15913 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4] 2> 15920 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
> [junit4] 2> 15921 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
> [junit4] 2> 15927 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4] 2> 15928 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4] 2> 15937 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 15938 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 15945 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4] 2> 15946 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4] 2> 15952 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4] 2> 15953 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4] 2> 17221 T15 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 17655 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16959
> [junit4] 2> 17703 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4] 2> 17704 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4] 2> 17704 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421
> [junit4] 2> 17705 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/'
> [junit4] 2> 17734 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/solr.xml
> [junit4] 2> 17823 T15 oasc.CoreContainer.<init> New CoreContainer 709323028
> [junit4] 2> 17824 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/]
> [junit4] 2> 17913 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
> [junit4] 2> 17913 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4] 2> 17914 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4] 2> 17914 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4] 2> 17915 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4] 2> 17915 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4] 2> 17916 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4] 2> 17916 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4] 2> 17917 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4] 2> 17926 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 17927 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 17927 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 17928 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
> [junit4] 2> 17968 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 17969 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 17974 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4af6fd54 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 17975 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 18000 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 18002 T38 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ce9927a name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 18002 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 18018 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4] 2> 18039 T15 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4] 2> 18065 T15 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4] 2> 18069 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16959_
> [junit4] 2> 18079 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16959_
> [junit4] 2> 18085 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4] 2> 18122 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4] 2> 18126 T15 oasc.Overseer.start Overseer (id=90741796671389699-127.0.0.1:16959_-n_0000000000) starting
> [junit4] 2> 18168 T15 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4] 2> 18190 T40 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4] 2> 18190 T15 oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4] 2> 18194 T15 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4] 2> 18197 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 18235 T39 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4] 2> 18251 T41 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4] 2> 18251 T41 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 18254 T41 oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 19747 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 19749 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:16959",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:16959_",
> [junit4] 2> "shard":null,
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 19750 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
> [junit4] 2> 19803 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 19833 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 20256 T41 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
> [junit4] 2> 20256 T41 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1
> [junit4] 2> 20257 T41 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
> [junit4] 2> 20258 T41 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 20258 T41 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
> [junit4] 2> 20261 T41 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/'
> [junit4] 2> 20262 T41 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/lib/classes/' to classloader
> [junit4] 2> 20263 T41 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/lib/README' to classloader
> [junit4] 2> 20365 T41 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
> [junit4] 2> 20437 T41 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 20440 T41 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4] 2> 20528 T41 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 20687 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.ByteField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 20713 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.ShortField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 20754 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableIntField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 20775 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableLongField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 20796 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableFloatField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 20817 T41 oasc.SolrResourceLoader.findClass WARN Solr loaded a deprecated plugin/analysis class [solr.SortableDoubleField]. Please consult documentation how to replace it accordingly.
> [junit4] 2> 21748 T41 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 21749 T41 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 21806 T41 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4] 2> 21839 T41 oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 21843 T41 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 22013 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 22028 T41 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 22034 T41 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 22040 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 22040 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 22041 T41 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 22043 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 22043 T41 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 22043 T41 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 22048 T41 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1384609935421/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/
> [junit4] 2> 22053 T41 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
> [junit4] 2> 22072 T41 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data
> [junit4] 2> 22072 T41 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index/
> [junit4] 2> 22072 T41 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index' doesn't exist. Creating new index...
> [junit4] 2> 22094 T41 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/control/data/index
> [junit4] 2> 22096 T41 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
> [junit4] 2> 22100 T41 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7f7c3056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476ff9e0),segFN=segments_1,generation=1}
> [junit4] 2> 22101 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 22106 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
> [junit4] 2> 22210 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
> [junit4] 2> 22220 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 22220 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
> [junit4] 2> 22221 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4] 2> 22221 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 22345 T41 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 22346 T41 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 22346 T41 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 22353 T41 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4] 2> 22354 T41 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4] 2> 22354 T41 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 22355 T41 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4] 2> 22356 T41 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4] 2> 22357 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 22383 T41 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 22390 T41 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 22429 T41 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 22430 T41 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 22431 T41 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
> [junit4] 2> 22483 T41 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7f7c3056 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@476ff9e0),segFN=segments_1,generation=1}
> [junit4] 2> 22484 T41 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 22485 T41 oass.SolrIndexSearcher.<init> Opening Searcher@695e0163 main
> [junit4] 2> 22537 T42 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@695e0163 main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 22545 T41 oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 22545 T41 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16959 collection:control_collection shard:shard1
> [junit4] 2> 22585 T41 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4] 2> 22619 T41 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
> [junit4] 2> 22624 T41 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4] 2> 22624 T41 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4] 2> 22634 T41 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16959/collection1/
> [junit4] 2> 22634 T41 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4] 2> 22635 T41 oasc.SyncStrategy.syncToMe http://127.0.0.1:16959/collection1/ has no replicas
> [junit4] 2> 22635 T41 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16959/collection1/ shard1
> [junit4] 2> 22636 T41 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
> [junit4] 2> 22844 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 22864 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 22901 T41 oasc.ZkController.register We are http://127.0.0.1:16959/collection1/ and leader is http://127.0.0.1:16959/collection1/
> [junit4] 2> 22901 T41 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16959
> [junit4] 2> 22902 T41 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4] 2> 22902 T41 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4] 2> 22903 T41 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 22905 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 22907 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
> [junit4] 2> 22908 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 23033 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 23036 T45 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@642ddc4c name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 23036 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 23039 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 23047 T15 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
> [junit4] 2> 23443 T15 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 23447 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35873
> [junit4] 2> 23448 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4] 2> 23448 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4] 2> 23449 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508
> [junit4] 2> 23450 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/'
> [junit4] 2> 23498 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/solr.xml
> [junit4] 2> 23602 T15 oasc.CoreContainer.<init> New CoreContainer 1950714347
> [junit4] 2> 23603 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/]
> [junit4] 2> 23605 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
> [junit4] 2> 23606 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4] 2> 23607 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4] 2> 23607 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4] 2> 23608 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4] 2> 23609 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4] 2> 23610 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4] 2> 23611 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4] 2> 23611 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4] 2> 23618 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 23619 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 23620 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 23621 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
> [junit4] 2> 23621 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 23623 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 23632 T56 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161def0 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 23632 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 23641 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 23644 T58 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188d92e name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 23644 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 23657 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 24380 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 24382 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:16959",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:16959_",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 24570 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 24570 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 24570 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 24664 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35873_
> [junit4] 2> 24667 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35873_
> [junit4] 2> 24671 T45 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4] 2> 24671 T38 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4] 2> 24671 T58 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4] 2> 24692 T59 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4] 2> 24693 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 24695 T59 oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 26078 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 26080 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:35873",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:35873_",
> [junit4] 2> "shard":null,
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 26080 T39 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
> [junit4] 2> 26081 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 26096 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 26096 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 26096 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 26697 T59 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
> [junit4] 2> 26697 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1
> [junit4] 2> 26698 T59 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4] 2> 26699 T59 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 26700 T59 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4] 2> 26702 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/'
> [junit4] 2> 26704 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/lib/classes/' to classloader
> [junit4] 2> 26705 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/lib/README' to classloader
> [junit4] 2> 26800 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
> [junit4] 2> 26884 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 26887 T59 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4] 2> 26994 T59 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 27735 T59 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 27736 T59 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 27737 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4] 2> 27754 T59 oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 27758 T59 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 27788 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 27795 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 27800 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 27802 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 27803 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 27803 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 27805 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 27805 T59 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 27805 T59 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 27806 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty1-1384609942508/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/
> [junit4] 2> 27806 T59 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
> [junit4] 2> 27808 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1
> [junit4] 2> 27809 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index/
> [junit4] 2> 27809 T59 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index' doesn't exist. Creating new index...
> [junit4] 2> 27809 T59 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty1/index
> [junit4] 2> 27810 T59 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
> [junit4] 2> 27811 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@74ba86ef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31d15a18),segFN=segments_1,generation=1}
> [junit4] 2> 27812 T59 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 27819 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
> [junit4] 2> 27819 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
> [junit4] 2> 27820 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 27820 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
> [junit4] 2> 27821 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4] 2> 27821 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 27822 T59 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 27822 T59 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 27822 T59 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 27824 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4] 2> 27826 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4] 2> 27826 T59 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 27827 T59 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4] 2> 27828 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4] 2> 27829 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 27831 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 27842 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 27848 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 27848 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 27850 T59 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
> [junit4] 2> 27902 T59 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@74ba86ef lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31d15a18),segFN=segments_1,generation=1}
> [junit4] 2> 27903 T59 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 27903 T59 oass.SolrIndexSearcher.<init> Opening Searcher@69365360 main
> [junit4] 2> 27936 T60 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69365360 main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 27942 T59 oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 27943 T59 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35873 collection:collection1 shard:shard1
> [junit4] 2> 27945 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 27970 T59 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
> [junit4] 2> 27974 T59 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4] 2> 27974 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4] 2> 27975 T59 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35873/collection1/
> [junit4] 2> 27975 T59 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4] 2> 27975 T59 oasc.SyncStrategy.syncToMe http://127.0.0.1:35873/collection1/ has no replicas
> [junit4] 2> 27976 T59 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35873/collection1/ shard1
> [junit4] 2> 27976 T59 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
> [junit4] 2> 29107 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 29138 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 29138 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 29138 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 29159 T59 oasc.ZkController.register We are http://127.0.0.1:35873/collection1/ and leader is http://127.0.0.1:35873/collection1/
> [junit4] 2> 29159 T59 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35873
> [junit4] 2> 29160 T59 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4] 2> 29160 T59 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4] 2> 29160 T59 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 29164 T59 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 29166 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
> [junit4] 2> 29167 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 29458 T15 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 29461 T15 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:23305
> [junit4] 2> 29462 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4] 2> 29463 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4] 2> 29464 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630
> [junit4] 2> 29464 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/'
> [junit4] 2> 29505 T15 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/solr.xml
> [junit4] 2> 29592 T15 oasc.CoreContainer.<init> New CoreContainer 574978041
> [junit4] 2> 29593 T15 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/]
> [junit4] 2> 29596 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
> [junit4] 2> 29597 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4] 2> 29597 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4] 2> 29598 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4] 2> 29599 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4] 2> 29599 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4] 2> 29600 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4] 2> 29601 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4] 2> 29601 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4] 2> 29608 T15 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 29609 T15 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 29610 T15 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 29611 T15 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46548/solr
> [junit4] 2> 29612 T15 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4] 2> 29613 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 29617 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f3c6654 name:ZooKeeperConnection Watcher:127.0.0.1:46548 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 29618 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 29628 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4] 2> 29631 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31313e0e name:ZooKeeperConnection Watcher:127.0.0.1:46548/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 29631 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4] 2> 29644 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4] 2> 30649 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 30650 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23305_
> [junit4] 2> 30651 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:35873",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:35873_",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 30678 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23305_
> [junit4] 2> 30694 T45 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4] 2> 30694 T74 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4] 2> 30694 T58 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4] 2> 30695 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 30696 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 30696 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 30706 T38 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4] 2> 30707 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 30713 T75 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4] 2> 30713 T75 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 30716 T75 oasc.ZkController.waitForCoreNodeName look for our core node name
> [junit4] 2> 32209 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 32210 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:23305",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:23305_",
> [junit4] 2> "shard":null,
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 32211 T39 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
> [junit4] 2> 32211 T39 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
> [junit4] 2> 32228 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 32228 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 32228 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 32228 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 32719 T75 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
> [junit4] 2> 32719 T75 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1
> [junit4] 2> 32719 T75 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4] 2> 32721 T75 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4] 2> 32721 T75 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4] 2> 32722 T75 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/'
> [junit4] 2> 32725 T75 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/lib/classes/' to classloader
> [junit4] 2> 32726 T75 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/lib/README' to classloader
> [junit4] 2> 32792 T75 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
> [junit4] 2> 32849 T75 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4] 2> 32851 T75 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4] 2> 32911 T75 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4] 2> 33462 T75 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 33462 T75 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4] 2> 33464 T75 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4] 2> 33481 T75 oass.IndexSchema.readSchema default search field in schema is text
> [junit4] 2> 33485 T75 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 33517 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 33523 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4] 2> 33529 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 33531 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 33531 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 33532 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4] 2> 33534 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4] 2> 33534 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4] 2> 33534 T75 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 33535 T75 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1384609948630/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/
> [junit4] 2> 33535 T75 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65988886
> [junit4] 2> 33537 T75 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2
> [junit4] 2> 33537 T75 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index/
> [junit4] 2> 33537 T75 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index' doesn't exist. Creating new index...
> [junit4] 2> 33538 T75 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index
> [junit4] 2> 33539 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=698412888, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
> [junit4] 2> 33540 T75 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1dacecf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ee964fe),segFN=segments_1,generation=1}
> [junit4] 2> 33540 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 33548 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
> [junit4] 2> 33549 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
> [junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
> [junit4] 2> 33550 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4] 2> 33551 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 33551 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 33552 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 33552 T75 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 33554 T75 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4] 2> 33556 T75 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4] 2> 33556 T75 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 33557 T75 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4] 2> 33559 T75 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4] 2> 33559 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 33561 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 33568 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 33573 T75 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 33574 T75 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 33576 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
> [junit4] 2> 33628 T75 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1dacecf3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ee964fe),segFN=segments_1,generation=1}
> [junit4] 2> 33629 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4] 2> 33629 T75 oass.SolrIndexSearcher.<init> Opening Searcher@69e1e999 main
> [junit4] 2> 33643 T76 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69e1e999 main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 33653 T75 oasc.CoreContainer.registerCore registering core: collection1
> [junit4] 2> 33653 T75 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:23305 collection:collection1 shard:shard1
> [junit4] 2> 33665 T75 oasc.ZkController.register We are http://127.0.0.1:23305/collection1/ and leader is http://127.0.0.1:35873/collection1/
> [junit4] 2> 33666 T75 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23305
> [junit4] 2> 33666 T75 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4] 2> 33667 T75 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4] 2> ASYNC NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6e34b350 url=http://127.0.0.1:23305/collection1 node=127.0.0.1:23305_ C1_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:23305, core=collection1, node_name=127.0.0.1:23305_}
> [junit4] 2> 33673 T77 C1 P23305 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4] 2> 33674 T77 C1 P23305 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4] 2> 33674 T75 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 33674 T77 C1 P23305 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4] 2> 33675 T77 C1 P23305 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 33676 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
> [junit4] 2> 33676 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4] 2> 33681 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 33736 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 33737 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"recovering",
> [junit4] 2> "base_url":"http://127.0.0.1:23305",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:23305_",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "shard_range":null,
> [junit4] 2> "shard_state":"active",
> [junit4] 2> "shard_parent":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node2"}
> [junit4] 2> 33757 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 33757 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 33757 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 33757 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 33896 T15 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 23305
> [junit4] 2> 33909 T15 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=574978041
> [junit4] 2> 35186 T48 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4] 2> 35188 T48 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 0 seconds.
> [junit4] 2> 35189 T48 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:23305_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=13
> [junit4] 2> 35265 T39 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4] 2> 35267 T39 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:23305",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:23305_",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "core_node_name":"core_node2"}
> [junit4] 2> 35270 T39 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
> [junit4] 2> 35288 T38 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 35288 T58 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 35288 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 35289 T45 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> ASYNC NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@2ac02d83 url=http://127.0.0.1:16959/collection1 node=127.0.0.1:16959_ C2_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:16959, core=collection1, node_name=127.0.0.1:16959_, leader=true}
> [junit4] 2> 35545 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1451868768091242496)]} 0 215
> [junit4] 2> 35553 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1451868768091242497)]} 0 223
> [junit4] 2> ASYNC NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@129a41d2 url=http://127.0.0.1:35873/collection1 node=127.0.0.1:35873_ C3_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:35873, core=collection1, node_name=127.0.0.1:35873_, leader=true}
> [junit4] 2> 35612 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10001 (1451868768245383168)]} 0 18
> [junit4] 2> 35615 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1451868768245383169)]} 0 18
> [junit4] 2> 35675 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1451868768313540608)]} 0 5
> [junit4] 2> 35677 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1451868768313540609)]} 0 6
> [junit4] 2> 35682 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10002 (1451868768321929216)]} 0 3
> [junit4] 2> 35684 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1451868768322977792)]} 0 4
> [junit4] 2> 35700 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1451868768339755008)]} 0 4
> [junit4] 2> 35704 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1451868768343949312)]} 0 5
> [junit4] 2> 35708 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10003 (1451868768348143616)]} 0 5
> [junit4] 2> 35711 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1451868768352337920)]} 0 3
> [junit4] 2> 35814 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1451868768461389824)]} 0 2
> [junit4] 2> 35817 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1451868768464535552)]} 0 1
> [junit4] 2> 35819 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10001 (-1451868768467681280)]} 0 1
> [junit4] 2> 35822 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[1 (-1451868768470827008)]} 0 1
> [junit4] 2> 35828 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1451868768473972736)]} 0 4
> [junit4] 2> 35830 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1451868768476069888)]} 0 4
> [junit4] 2> 35835 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10004 (1451868768482361344)]} 0 3
> [junit4] 2> 35837 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1451868768483409920)]} 0 4
> [junit4] 2> 35904 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1451868768553664512)]} 0 4
> [junit4] 2> 35907 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1451868768555761664)]} 0 6
> [junit4] 2> 35912 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10005 (1451868768562053120)]} 0 4
> [junit4] 2> 35915 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[5 (1451868768565198848)]} 0 5
> [junit4] 2> 35982 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1451868768635453440)]} 0 4
> [junit4] 2> 35985 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1451868768638599168)]} 0 5
> [junit4] 2> 35990 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10006 (1451868768643842048)]} 0 4
> [junit4] 2> 35993 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[6 (1451868768646987776)]} 0 5
> [junit4] 2> 36099 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1451868768758136832)]} 0 5
> [junit4] 2> 36101 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1451868768760233984)]} 0 5
> [junit4] 2> 36108 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10007 (1451868768767574016)]} 0 5
> [junit4] 2> 36110 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[7 (1451868768768622592)]} 0 5
> [junit4] 2> 36133 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1451868768793788416)]} 0 4
> [junit4] 2> 36136 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1451868768795885568)]} 0 6
> [junit4] 2> 36143 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10008 (1451868768804274176)]} 0 5
> [junit4] 2> 36145 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[8 (1451868768805322752)]} 0 6
> [junit4] 2> 36224 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1451868768890257408)]} 0 3
> [junit4] 2> 36226 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1451868768891305984)]} 0 4
> [junit4] 2> 36231 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10009 (1451868768897597440)]} 0 3
> [junit4] 2> 36233 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[9 (1451868768898646016)]} 0 4
> [junit4] 2> 36241 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1451868768908083200)]} 0 3
> [junit4] 2> 36245 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1451868768911228928)]} 0 4
> [junit4] 2> 36247 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10010 (1451868768914374656)]} 0 3
> [junit4] 2> 36252 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10 (1451868768919617536)]} 0 3
> [junit4] 2> 36279 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1451868768946880512)]} 0 4
> [junit4] 2> 36283 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1451868768952123392)]} 0 4
> [junit4] 2> 36286 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10011 (1451868768955269120)]} 0 3
> [junit4] 2> 36291 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[11 (1451868768959463424)]} 0 4
> [junit4] 2> 36305 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1451868768974143488)]} 0 5
> [junit4] 2> 36309 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1451868768979386368)]} 0 3
> [junit4] 2> 36314 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10012 (1451868768983580672)]} 0 5
> [junit4] 2> 36316 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[12 (1451868768985677824)]} 0 4
> [junit4] 2> 36342 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1451868769011892224)]} 0 5
> [junit4] 2> 36344 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1451868769013989376)]} 0 6
> [junit4] 2> 36349 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10013 (1451868769021329408)]} 0 4
> [junit4] 2> 36352 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[13 (1451868769023426560)]} 0 4
> [junit4] 2> 36396 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1451868769068515328)]} 0 5
> [junit4] 2> 36398 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1451868769071661056)]} 0 5
> [junit4] 2> 36404 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10014 (1451868769077952512)]} 0 5
> [junit4] 2> 36405 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[14 (1451868769080049664)]} 0 4
> [junit4] 2> 36468 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1451868769148207104)]} 0 1
> [junit4] 2> 36469 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1451868769149255680)]} 0 2
> [junit4] 2> 36473 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10002 (-1451868769153449984)]} 0 2
> [junit4] 2> 36474 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[2 (-1451868769154498560)]} 0 1
> [junit4] 2> 36481 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1451868769159741441)]} 0 3
> [junit4] 2> 36483 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1451868769159741440)]} 0 5
> [junit4] 2> 36487 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10015 (1451868769166032896)]} 0 3
> [junit4] 2> 36491 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[15 (1451868769169178624)]} 0 4
> [junit4] 2> 36561 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1451868769244676096)]} 0 2
> [junit4] 2> 36564 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1451868769248870400)]} 0 2
> [junit4] 2> 36566 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10006 (-1451868769250967552)]} 0 2
> [junit4] 2> 36569 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[6 (-1451868769254113280)]} 0 2
> [junit4] 2> 36573 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1451868769256210432)]} 0 4
> [junit4] 2> 36575 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1451868769258307584)]} 0 3
> [junit4] 2> 36580 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10016 (1451868769262501888)]} 0 4
> [junit4] 2> 36583 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[16 (1451868769265647616)]} 0 5
> [junit4] 2> 36628 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1451868769315979264)]} 0 1
> [junit4] 2> 36630 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1451868769318076416)]} 0 1
> [junit4] 2> 36634 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10007 (-1451868769322270720)]} 0 1
> [junit4] 2> 36634 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[7 (-1451868769322270721)]} 0 1
> [junit4] 2> 36643 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1451868769328562176)]} 0 5
> [junit4] 2> 36644 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1451868769328562177)]} 0 6
> [junit4] 2> 36650 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10017 (1451868769336950784)]} 0 4
> [junit4] 2> 36651 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[17 (1451868769337999360)]} 0 3
> [junit4] 2> 36696 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1451868769385185280)]} 0 3
> [junit4] 2> 36699 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1451868769387282432)]} 0 4
> [junit4] 2> 36704 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10018 (1451868769392525312)]} 0 4
> [junit4] 2> 36708 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[18 (1451868769396719616)]} 0 4
> [junit4] 2> 36757 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1451868769450196992)]} 0 1
> [junit4] 2> 36762 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[10008 (-1451868769456488448)]} 0 2
> [junit4] 2> 36762 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1451868769456488448)]} 0 1
> [junit4] 2> 36769 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {delete=[8 (-1451868769462779904)]} 0 2
> [junit4] 2> 36770 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1451868769461731328)]} 0 4
> [junit4] 2> 36777 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1451868769469071360)]} 0 5
> [junit4] 2> 36778 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10019 (1451868769470119936)]} 0 4
> [junit4] 2> 36784 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[19 (1451868769476411392)]} 0 4
> [junit4] 2> 36863 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1451868769560297472)]} 0 3
> [junit4] 2> 36867 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1451868769564491776)]} 0 3
> [junit4] 2> 36872 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10020 (1451868769568686080)]} 0 5
> [junit4] 2> 36874 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[20 (1451868769571831808)]} 0 3
> [junit4] 2> 36891 T29 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1451868769588609024)]} 0 4
> [junit4] 2> 36893 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1451868769590706176)]} 0 5
> [junit4] 2> 36899 T50 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10021 (1451868769596997632)]} 0 4
> [junit4] 2> 36901 T49 C3 P35873 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[21 (1451868769599094784)]} 0 5
> [junit4] 2> 36915 T15 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4] 2> 36915 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4] 2> 36918 T15 oasc.RecoveryStrategy.close WARN Stopping recovery for zkNodeName=core_node2core=collection1
> [junit4] 2> 36997 T28 C2 P16959 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[10022 (145186876970080
>
> [...truncated too long message...]
>
> ocketImpl.java:180)
> [junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
> [junit4] 2> at java.net.Socket.connect(Socket.java:546)
> [junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
> [junit4] 2> ... 19 more
> [junit4] 2>
> [junit4] 2> 92118 T97 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:23305 DONE. sync failed
> [junit4] 2> 92119 T97 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
> [junit4] 2> 92120 T97 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
> [junit4] 2> 92120 T97 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6c3b0b1e
> [junit4] 2> 92149 T97 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=54,cumulative_deletesById=28,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
> [junit4] 2> 92149 T97 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
> [junit4] 2> 92150 T97 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4] 2> 92150 T97 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
> [junit4] 2> 92152 T97 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
> [junit4] 2> 92154 T97 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
> [junit4] 2> 92154 T97 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436;done=false>>]
> [junit4] 2> 92155 T97 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2/index.20131116045246436
> [junit4] 2> 77926 T14 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
> [junit4] 2> 92616 T97 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2;done=false>>]
> [junit4] 2> 92616 T97 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1384609934071/jetty2
> [junit4] 2> 92617 T97 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4] 2> 92617 T97 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
> [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
> [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:247)
> [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:244)
> [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
> [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:244)
> [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
> [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
> [junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
> [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
> [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
> [junit4] 2>
> [junit4] 2> NOTE: test params are: codec=Lucene46: {range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), text=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=Asserting), rnd_b=Lucene41(blocksize=128), intDefault=PostingsFormat(name=Asserting), id=Lucene41(blocksize=128), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_sl=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), other_tl1=PostingsFormat(name=Asserting), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=Asserting)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es_PR, timezone=America/Sitka
> [junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=439169240,total=470548480
> [junit4] 2> NOTE: All tests run in this JVM: [TestAddFieldRealTimeGet, RecoveryZkTest]
> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=BBCE0E7280CAB87B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PR -Dtests.timezone=America/Sitka -Dtests.file.encoding=UTF-8
> [junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
> [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=98, name=coreLoadExecutor-43-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
> [junit4] > at __randomizedtesting.SeedInfo.seed([BBCE0E7280CAB87B]:0)
> [junit4] > at java.lang.Thread.getStackTrace(Thread.java:1495)
> [junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
> [junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:556)
> [junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:1049)
> [junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:947)
> [junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:939)
> [junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
> [junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:350)
> [junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:253)
> [junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:467)
> [junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:758)
> [junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:627)
> [junit4] > at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
> [junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:575)
> [junit4] > at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:256)
> [junit4] > at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:248)
> [junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> [junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [junit4] > at java.lang.Thread.run(Thread.java:679)
> [junit4] > Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@7d6ac92e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6dbe2b55)
> [junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
> [junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
> [junit4] > ... 1 more
> [junit4] Completed on J0 in 78.75s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 1021 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:426: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:406: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:489: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1276: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:912: There were test failures: 340 suites, 1496 tests, 1 suite-level error, 35 ignored (6 assumptions)
>
> Total time: 60 minutes 33 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org