You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/03/18 18:16:37 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_15) - Build # 4724
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4724/
Java: 32bit/jdk1.7.0_15 -client -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1375, name=coreLoadExecutor-631-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1375, name=coreLoadExecutor-631-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1567)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@ff7054 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c71220))
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1377, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1377, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1567)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327))
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9259 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 0 T1316 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 3 T1316 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363625252254
[junit4:junit4] 2> 4 T1316 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 4 T1317 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 5 T1317 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T1317 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 5 T1317 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 6 T1317 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 6 T1317 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363625252253/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 104 T1316 oasc.ZkTestServer.run start zk server on port:43090
[junit4:junit4] 2> 105 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1b29b34
[junit4:junit4] 2> 105 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 108 T1322 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 109 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45901
[junit4:junit4] 2> 109 T1322 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 110 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45901
[junit4:junit4] 2> 110 T1320 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 193 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00000 with negotiated timeout 10000 for client /127.0.0.1:45901
[junit4:junit4] 2> 193 T1322 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00000, negotiated timeout = 10000
[junit4:junit4] 2> 193 T1323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b29b34 name:ZooKeeperConnection Watcher:127.0.0.1:43090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 194 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 194 T1316 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 196 T1321 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7e6615a00000
[junit4:junit4] 2> 197 T1323 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 197 T1316 oaz.ZooKeeper.close Session: 0x13d7e6615a00000 closed
[junit4:junit4] 2> 197 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@59ea04
[junit4:junit4] 2> 198 T1318 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45901 which had sessionid 0x13d7e6615a00000
[junit4:junit4] 2> 199 T1324 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 198 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 199 T1324 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 199 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45902
[junit4:junit4] 2> 199 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45902
[junit4:junit4] 2> 200 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00001 with negotiated timeout 10000 for client /127.0.0.1:45902
[junit4:junit4] 2> 200 T1324 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00001, negotiated timeout = 10000
[junit4:junit4] 2> 200 T1325 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59ea04 name:ZooKeeperConnection Watcher:127.0.0.1:43090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 201 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 201 T1316 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 203 T1316 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 205 T1316 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 206 T1316 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 208 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 208 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 211 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 212 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 214 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 215 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 217 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 217 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 219 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 219 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 221 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 221 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 223 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 224 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 225 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 226 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 228 T1316 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 228 T1316 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 230 T1321 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7e6615a00001
[junit4:junit4] 2> 230 T1325 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 230 T1318 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45902 which had sessionid 0x13d7e6615a00001
[junit4:junit4] 2> 230 T1316 oaz.ZooKeeper.close Session: 0x13d7e6615a00001 closed
[junit4:junit4] 2> 295 T1316 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 298 T1316 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44407
[junit4:junit4] 2> 298 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 299 T1316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 299 T1316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481
[junit4:junit4] 2> 299 T1316 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/solr.xml
[junit4:junit4] 2> 300 T1316 oasc.CoreContainer.<init> New CoreContainer 30865138
[junit4:junit4] 2> 300 T1316 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/'
[junit4:junit4] 2> 301 T1316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/'
[junit4:junit4] 2> 321 T1316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 322 T1316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 322 T1316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 323 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 323 T1316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 323 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 324 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 324 T1316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 324 T1316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 325 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 332 T1316 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 348 T1316 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:43090/solr
[junit4:junit4] 2> 349 T1316 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 349 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6a361d
[junit4:junit4] 2> 350 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 350 T1335 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 351 T1335 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 351 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45903
[junit4:junit4] 2> 351 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45903
[junit4:junit4] 2> 353 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00002 with negotiated timeout 20000 for client /127.0.0.1:45903
[junit4:junit4] 2> 353 T1335 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00002, negotiated timeout = 20000
[junit4:junit4] 2> 354 T1336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a361d name:ZooKeeperConnection Watcher:127.0.0.1:43090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 354 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 355 T1321 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7e6615a00002
[junit4:junit4] 2> 355 T1336 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 355 T1318 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45903 which had sessionid 0x13d7e6615a00002
[junit4:junit4] 2> 355 T1316 oaz.ZooKeeper.close Session: 0x13d7e6615a00002 closed
[junit4:junit4] 2> 356 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 359 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@e4acbe
[junit4:junit4] 2> 360 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 360 T1337 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 360 T1337 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 361 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45904
[junit4:junit4] 2> 361 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45904
[junit4:junit4] 2> 362 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00003 with negotiated timeout 20000 for client /127.0.0.1:45904
[junit4:junit4] 2> 362 T1337 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00003, negotiated timeout = 20000
[junit4:junit4] 2> 362 T1338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e4acbe name:ZooKeeperConnection Watcher:127.0.0.1:43090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 362 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 364 T1316 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 365 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x7 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 366 T1316 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 368 T1316 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 369 T1316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44407_
[junit4:junit4] 2> 370 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:delete cxid:0x12 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:44407_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:44407_
[junit4:junit4] 2> 370 T1316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44407_
[junit4:junit4] 2> 372 T1316 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 375 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:delete cxid:0x20 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 376 T1316 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 377 T1316 oasc.Overseer.start Overseer (id=89366544531849219-127.0.0.1:44407_-n_0000000000) starting
[junit4:junit4] 2> 378 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x25 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 378 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x26 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 379 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x28 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 380 T1316 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 381 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x2e zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 382 T1340 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 383 T1316 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 384 T1316 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 385 T1316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 387 T1339 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 390 T1341 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/collection1
[junit4:junit4] 2> 390 T1341 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 390 T1341 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 390 T1341 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 391 T1341 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/collection1/'
[junit4:junit4] 2> 392 T1341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/collection1/lib/README' to classloader
[junit4:junit4] 2> 392 T1341 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 428 T1341 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 475 T1341 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 476 T1341 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 482 T1341 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 961 T1341 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 971 T1341 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 975 T1341 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 986 T1341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 989 T1341 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 992 T1341 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 993 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 994 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 994 T1341 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 995 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 995 T1341 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 995 T1341 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 995 T1341 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363625252481/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data/
[junit4:junit4] 2> 995 T1341 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c03865
[junit4:junit4] 2> 996 T1341 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 996 T1341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data forceNew: false
[junit4:junit4] 2> 996 T1341 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data/index/
[junit4:junit4] 2> 997 T1341 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 997 T1341 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data/index forceNew: false
[junit4:junit4] 2> 998 T1341 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9b36be lockFactory=org.apache.lucene.store.NativeFSLockFactory@cf6795),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 998 T1341 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 1000 T1341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1000 T1341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1001 T1341 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1001 T1341 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1001 T1341 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1002 T1341 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1002 T1341 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1002 T1341 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1002 T1341 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1005 T1341 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1007 T1341 oass.SolrIndexSearcher.<init> Opening Searcher@e46e67 main
[junit4:junit4] 2> 1008 T1341 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/control/data/tlog
[junit4:junit4] 2> 1008 T1341 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1008 T1341 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1011 T1342 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e46e67 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1011 T1341 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1011 T1341 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1898 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1898 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44407_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44407"}
[junit4:junit4] 2> 1899 T1339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1899 T1339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 1903 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2013 T1341 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2013 T1341 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44407 collection:control_collection shard:shard1
[junit4:junit4] 2> 2014 T1341 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2019 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:delete cxid:0x82 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2020 T1341 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2020 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x83 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2022 T1341 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2023 T1341 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2023 T1341 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44407/collection1/
[junit4:junit4] 2> 2023 T1341 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2024 T1341 oasc.SyncStrategy.syncToMe http://127.0.0.1:44407/collection1/ has no replicas
[junit4:junit4] 2> 2024 T1341 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44407/collection1/
[junit4:junit4] 2> 2024 T1341 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2030 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00003 type:create cxid:0x8e zxid:0x3a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3406 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3413 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3438 T1341 oasc.ZkController.register We are http://127.0.0.1:44407/collection1/ and leader is http://127.0.0.1:44407/collection1/
[junit4:junit4] 2> 3438 T1341 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44407
[junit4:junit4] 2> 3439 T1341 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3439 T1341 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3439 T1341 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3441 T1341 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3442 T1316 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3443 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3443 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3447 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3448 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18719d9
[junit4:junit4] 2> 3449 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3449 T1344 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3449 T1344 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 3449 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45936
[junit4:junit4] 2> 3450 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45936
[junit4:junit4] 2> 3451 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00004 with negotiated timeout 10000 for client /127.0.0.1:45936
[junit4:junit4] 2> 3451 T1344 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00004, negotiated timeout = 10000
[junit4:junit4] 2> 3452 T1345 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18719d9 name:ZooKeeperConnection Watcher:127.0.0.1:43090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3452 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3453 T1316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3455 T1316 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3523 T1316 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3529 T1316 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43698
[junit4:junit4] 2> 3530 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3531 T1316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3531 T1316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705
[junit4:junit4] 2> 3531 T1316 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/solr.xml
[junit4:junit4] 2> 3532 T1316 oasc.CoreContainer.<init> New CoreContainer 21715795
[junit4:junit4] 2> 3532 T1316 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/'
[junit4:junit4] 2> 3533 T1316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/'
[junit4:junit4] 2> 3558 T1316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3559 T1316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3559 T1316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3560 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3560 T1316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3560 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3561 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3561 T1316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3562 T1316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3562 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3571 T1316 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3587 T1316 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:43090/solr
[junit4:junit4] 2> 3588 T1316 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3589 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@42d381
[junit4:junit4] 2> 3590 T1355 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3591 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45937
[junit4:junit4] 2> 3591 T1355 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 3591 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3591 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45937
[junit4:junit4] 2> 3592 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00005 with negotiated timeout 20000 for client /127.0.0.1:45937
[junit4:junit4] 2> 3592 T1355 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00005, negotiated timeout = 20000
[junit4:junit4] 2> 3592 T1356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42d381 name:ZooKeeperConnection Watcher:127.0.0.1:43090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3593 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3593 T1321 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7e6615a00005
[junit4:junit4] 2> 3595 T1318 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45937 which had sessionid 0x13d7e6615a00005
[junit4:junit4] 2> 3594 T1316 oaz.ZooKeeper.close Session: 0x13d7e6615a00005 closed
[junit4:junit4] 2> 3595 T1356 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3595 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3598 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@fa8220
[junit4:junit4] 2> 3599 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3600 T1357 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3601 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45938
[junit4:junit4] 2> 3601 T1357 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 3602 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45938
[junit4:junit4] 2> 3603 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00006 with negotiated timeout 20000 for client /127.0.0.1:45938
[junit4:junit4] 2> 3603 T1357 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00006, negotiated timeout = 20000
[junit4:junit4] 2> 3604 T1358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fa8220 name:ZooKeeperConnection Watcher:127.0.0.1:43090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3604 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3605 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:create cxid:0x1 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3607 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:create cxid:0x3 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3611 T1316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4615 T1316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43698_
[junit4:junit4] 2> 4616 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:delete cxid:0xd zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:43698_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:43698_
[junit4:junit4] 2> 4616 T1316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43698_
[junit4:junit4] 2> 4619 T1338 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4619 T1358 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4619 T1345 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4619 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4627 T1359 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/collection1
[junit4:junit4] 2> 4628 T1359 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4628 T1359 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4629 T1359 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4630 T1359 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/collection1/'
[junit4:junit4] 2> 4631 T1359 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/collection1/lib/README' to classloader
[junit4:junit4] 2> 4631 T1359 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4669 T1359 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 4721 T1359 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4722 T1359 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4730 T1359 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 4919 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4920 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44407_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44407"}
[junit4:junit4] 2> 4925 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4925 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4925 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5335 T1359 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5344 T1359 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5347 T1359 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5363 T1359 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5367 T1359 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5371 T1359 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5372 T1359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5373 T1359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5373 T1359 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5374 T1359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5374 T1359 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5374 T1359 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5375 T1359 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363625255705/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1/
[junit4:junit4] 2> 5375 T1359 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c03865
[junit4:junit4] 2> 5376 T1359 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5376 T1359 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1 forceNew: false
[junit4:junit4] 2> 5377 T1359 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1/index/
[junit4:junit4] 2> 5377 T1359 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5377 T1359 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1/index forceNew: false
[junit4:junit4] 2> 5379 T1359 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1f24669 lockFactory=org.apache.lucene.store.NativeFSLockFactory@123a825),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5379 T1359 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5381 T1359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5382 T1359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5382 T1359 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5383 T1359 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5383 T1359 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5384 T1359 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5384 T1359 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5384 T1359 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5385 T1359 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5388 T1359 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5391 T1359 oass.SolrIndexSearcher.<init> Opening Searcher@1290e6d main
[junit4:junit4] 2> 5392 T1359 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty1/tlog
[junit4:junit4] 2> 5392 T1359 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5392 T1359 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5398 T1360 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1290e6d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5400 T1359 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5401 T1359 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6429 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6430 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43698_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43698"}
[junit4:junit4] 2> 6431 T1339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 6431 T1339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6436 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6436 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6436 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7402 T1359 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7403 T1359 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43698 collection:collection1 shard:shard1
[junit4:junit4] 2> 7404 T1359 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 7409 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:delete cxid:0x4a zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7409 T1359 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7410 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:create cxid:0x4b zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7415 T1359 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7416 T1359 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7416 T1359 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43698/collection1/
[junit4:junit4] 2> 7416 T1359 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7416 T1359 oasc.SyncStrategy.syncToMe http://127.0.0.1:43698/collection1/ has no replicas
[junit4:junit4] 2> 7417 T1359 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43698/collection1/
[junit4:junit4] 2> 7417 T1359 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 7421 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00006 type:create cxid:0x56 zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7941 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7947 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7947 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7948 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7974 T1359 oasc.ZkController.register We are http://127.0.0.1:43698/collection1/ and leader is http://127.0.0.1:43698/collection1/
[junit4:junit4] 2> 7975 T1359 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43698
[junit4:junit4] 2> 7975 T1359 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 7975 T1359 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 7976 T1359 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7979 T1359 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7980 T1316 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 7981 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 7982 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8069 T1316 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 8072 T1316 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47543
[junit4:junit4] 2> 8072 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8073 T1316 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8073 T1316 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237
[junit4:junit4] 2> 8074 T1316 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/solr.xml
[junit4:junit4] 2> 8074 T1316 oasc.CoreContainer.<init> New CoreContainer 13096126
[junit4:junit4] 2> 8074 T1316 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/'
[junit4:junit4] 2> 8075 T1316 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/'
[junit4:junit4] 2> 8101 T1316 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8101 T1316 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8102 T1316 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8102 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8102 T1316 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8103 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8103 T1316 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8104 T1316 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8104 T1316 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8105 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8120 T1316 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8136 T1316 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:43090/solr
[junit4:junit4] 2> 8136 T1316 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8136 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@9d210a
[junit4:junit4] 2> 8137 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8138 T1371 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8138 T1371 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 8139 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45942
[junit4:junit4] 2> 8139 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45942
[junit4:junit4] 2> 8140 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00007 with negotiated timeout 20000 for client /127.0.0.1:45942
[junit4:junit4] 2> 8140 T1371 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00007, negotiated timeout = 20000
[junit4:junit4] 2> 8141 T1372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9d210a name:ZooKeeperConnection Watcher:127.0.0.1:43090 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8141 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8144 T1321 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7e6615a00007
[junit4:junit4] 2> 8144 T1318 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45942 which had sessionid 0x13d7e6615a00007
[junit4:junit4] 2> 8144 T1316 oaz.ZooKeeper.close Session: 0x13d7e6615a00007 closed
[junit4:junit4] 2> 8145 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8146 T1372 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8150 T1316 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43090/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@b758b3
[junit4:junit4] 2> 8151 T1316 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8151 T1373 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:43090. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8152 T1373 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43090, initiating session
[junit4:junit4] 2> 8152 T1318 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45943
[junit4:junit4] 2> 8154 T1318 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45943
[junit4:junit4] 2> 8155 T1320 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7e6615a00008 with negotiated timeout 20000 for client /127.0.0.1:45943
[junit4:junit4] 2> 8155 T1373 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:43090, sessionid = 0x13d7e6615a00008, negotiated timeout = 20000
[junit4:junit4] 2> 8155 T1374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b758b3 name:ZooKeeperConnection Watcher:127.0.0.1:43090/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8155 T1316 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8157 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8159 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00008 type:create cxid:0x3 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8161 T1316 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9168 T1316 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47543_
[junit4:junit4] 2> 9169 T1321 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7e6615a00008 type:delete cxid:0xd zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:47543_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:47543_
[junit4:junit4] 2> 9170 T1316 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47543_
[junit4:junit4] 2> 9172 T1338 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9173 T1358 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9173 T1374 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9173 T1345 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9173 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9173 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9173 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9182 T1375 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/collection1
[junit4:junit4] 2> 9183 T1375 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9184 T1375 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9184 T1375 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9185 T1375 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/collection1/'
[junit4:junit4] 2> 9186 T1375 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/collection1/lib/README' to classloader
[junit4:junit4] 2> 9187 T1375 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9228 T1375 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 9280 T1375 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9281 T1375 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9286 T1375 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9452 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9453 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43698_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43698"}
[junit4:junit4] 2> 9457 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9457 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9458 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9458 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9818 T1375 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9825 T1375 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9829 T1375 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9845 T1375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9850 T1375 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9855 T1375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9856 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9856 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9856 T1375 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9858 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9858 T1375 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9858 T1375 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9858 T1375 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363625260237/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/
[junit4:junit4] 2> 9859 T1375 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c03865
[junit4:junit4] 2> 9859 T1375 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9859 T1375 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2 forceNew: false
[junit4:junit4] 2> 9860 T1375 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index/
[junit4:junit4] 2> 9861 T1375 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9861 T1375 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index forceNew: false
[junit4:junit4] 2> 9862 T1375 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17094d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@97b29a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9863 T1375 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9865 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9866 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9866 T1375 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9867 T1375 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9867 T1375 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9868 T1375 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9868 T1375 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9869 T1375 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9869 T1375 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9872 T1375 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9876 T1375 oass.SolrIndexSearcher.<init> Opening Searcher@1485f86 main
[junit4:junit4] 2> 9876 T1375 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/tlog
[junit4:junit4] 2> 9877 T1375 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9877 T1375 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9884 T1376 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1485f86 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9888 T1375 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9889 T1375 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 10961 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10962 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47543"}
[junit4:junit4] 2> 10962 T1339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 10962 T1339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 10967 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10967 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10968 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10968 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11891 T1375 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11891 T1375 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47543 collection:collection1 shard:shard1
[junit4:junit4] 2> 11895 T1375 oasc.ZkController.register We are http://127.0.0.1:47543/collection1/ and leader is http://127.0.0.1:43698/collection1/
[junit4:junit4] 2> 11895 T1375 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47543
[junit4:junit4] 2> 11895 T1375 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 11896 T1375 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C77 name=collection1 org.apache.solr.core.SolrCore@aff120 url=http://127.0.0.1:47543/collection1 node=127.0.0.1:47543_ C77_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47543_, base_url=http://127.0.0.1:47543}
[junit4:junit4] 2> 11896 T1377 C77 P47543 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 11901 T1375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11902 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 11902 T1377 C77 P47543 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 11903 T1377 C77 P47543 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11903 T1316 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 11904 T1316 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 11904 T1316 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11904 T1377 C77 P47543 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11909 T1316 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C78 name=collection1 org.apache.solr.core.SolrCore@1a07bf0 url=http://127.0.0.1:44407/collection1 node=127.0.0.1:44407_ C78_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:44407_, base_url=http://127.0.0.1:44407, leader=true}
[junit4:junit4] 2> 11927 T1332 C78 P44407 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@9b36be lockFactory=org.apache.lucene.store.NativeFSLockFactory@cf6795),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11928 T1332 C78 P44407 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11932 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10001 (1429864725006516224)]} 0 8
[junit4:junit4] 2> 11935 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[1 (1429864725008613376)]} 0 9
[junit4:junit4] 2> ASYNC NEW_CORE C79 name=collection1 org.apache.solr.core.SolrCore@db5d95 url=http://127.0.0.1:43698/collection1 node=127.0.0.1:43698_ C79_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:43698_, base_url=http://127.0.0.1:43698, leader=true}
[junit4:junit4] 2> 11948 T1354 C79 P43698 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1f24669 lockFactory=org.apache.lucene.store.NativeFSLockFactory@123a825),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11949 T1354 C79 P43698 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11951 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[1 (1429864725031682048)]} 0 3
[junit4:junit4] 2> 11954 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10001 (1429864725025390592)]} 0 12
[junit4:junit4] 2> 11966 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[2 (1429864725049507840)]} 0 1
[junit4:junit4] 2> 11968 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10002 (1429864725052653568)]} 0 0
[junit4:junit4] 2> 11969 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[2 (1429864725052653568)]} 0 1
[junit4:junit4] 2> 11972 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10002 (1429864725054750720)]} 0 2
[junit4:junit4] 2> 12034 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[2 (-1429864725121859584)]} 0 1
[junit4:junit4] 2> 12038 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[2 (-1429864725125005312)]} 0 1
[junit4:junit4] 2> 12038 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10002 (-1429864725125005312)]} 0 1
[junit4:junit4] 2> 12043 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10002 (-1429864725128151040)]} 0 3
[junit4:junit4] 2> 12043 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[3 (1429864725128151040)]} 0 3
[junit4:junit4] 2> 12048 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10003 (1429864725133393920)]} 0 3
[junit4:junit4] 2> 12048 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[3 (1429864725134442496)]} 0 2
[junit4:junit4] 2> 12052 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10003 (1429864725139685376)]} 0 1
[junit4:junit4] 2> 12095 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[4 (1429864725184774144)]} 0 1
[junit4:junit4] 2> 12098 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[4 (1429864725187919872)]} 0 1
[junit4:junit4] 2> 12098 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10004 (1429864725186871296)]} 0 2
[junit4:junit4] 2> 12102 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10004 (1429864725192114176)]} 0 1
[junit4:junit4] 2> 12123 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[5 (1429864725214134272)]} 0 2
[junit4:junit4] 2> 12127 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[5 (1429864725218328576)]} 0 2
[junit4:junit4] 2> 12129 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10005 (1429864725220425728)]} 0 1
[junit4:junit4] 2> 12132 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10005 (1429864725223571456)]} 0 1
[junit4:junit4] 2> 12194 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[6 (1429864725288583168)]} 0 1
[junit4:junit4] 2> 12197 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[6 (1429864725291728896)]} 0 1
[junit4:junit4] 2> 12198 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10006 (1429864725293826048)]} 0 0
[junit4:junit4] 2> 12201 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10006 (1429864725295923200)]} 0 1
[junit4:junit4] 2> 12223 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[5 (-1429864725320040448)]} 0 0
[junit4:junit4] 2> 12226 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[5 (-1429864725323186176)]} 0 1
[junit4:junit4] 2> 12227 T1332 C78 P44407 /update {wt=javabin&version=2} {delete=[10005 (-1429864725324234752)]} 0 0
[junit4:junit4] 2> 12229 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10005 (-1429864725326331904)]} 0 0
[junit4:junit4] 2> 12229 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[7 (1429864725325283328)]} 0 1
[junit4:junit4] 2> 12232 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10007 (1429864725329477632)]} 0 0
[junit4:junit4] 2> 12234 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[7 (1429864725330526208)]} 0 1
[junit4:junit4] 2> 12235 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10007 (1429864725332623360)]} 0 1
[junit4:junit4] 2> 12324 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[8 (1429864725423849472)]} 0 2
[junit4:junit4] 2> 12326 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10008 (1429864725426995200)]} 0 1
[junit4:junit4] 2> 12327 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[8 (1429864725428043776)]} 0 1
[junit4:junit4] 2> 12330 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10008 (1429864725431189504)]} 0 1
[junit4:junit4] 2> 12400 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[8 (-1429864725505638400)]} 0 1
[junit4:junit4] 2> 12403 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10008 (-1429864725508784128)]} 0 1
[junit4:junit4] 2> 12403 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[8 (-1429864725508784128)]} 0 1
[junit4:junit4] 2> 12405 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10008 (-1429864725510881280)]} 0 0
[junit4:junit4] 2> 12408 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[9 (1429864725512978432)]} 0 1
[junit4:junit4] 2> 12410 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10009 (1429864725515075584)]} 0 1
[junit4:junit4] 2> 12420 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[9 (1429864725525561344)]} 0 1
[junit4:junit4] 2> 12420 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10009 (1429864725525561345)]} 0 1
[junit4:junit4] 2> 12472 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12472 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47543"}
[junit4:junit4] 2> 12476 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12476 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12476 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12476 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12508 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[9 (-1429864725618884608)]} 0 1
[junit4:junit4] 2> 12508 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10009 (-1429864725618884609)]} 0 1
[junit4:junit4] 2> C77_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:47543_, base_url=http://127.0.0.1:47543}
[junit4:junit4] 2> 12531 T1367 C77 P47543 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@17094d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@97b29a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12532 T1367 C77 P47543 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12533 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[9 (-1429864725620981760)]} 0 3
[junit4:junit4] 2> 12533 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10009 (-1429864725620981761)]} 0 15
[junit4:junit4] 2> 12534 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[9 (-1429864725620981760)]} 0 24
[junit4:junit4] 2> 12534 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10009 (-1429864725620981761)]} 0 24
[junit4:junit4] 2> 12536 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10 (1429864725648244736)]} 0 0
[junit4:junit4] 2> 12537 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10010 (1429864725648244737)]} 0 1
[junit4:junit4] 2> 12543 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10010 (1429864725651390465)]} 0 2
[junit4:junit4] 2> 12544 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10010 (1429864725651390465)]} 0 5
[junit4:junit4] 2> 12544 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10 (1429864725651390464)]} 0 2
[junit4:junit4] 2> 12545 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10 (1429864725651390464)]} 0 7
[junit4:junit4] 2> 12571 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10010 (-1429864725684944896)]} 0 1
[junit4:junit4] 2> 12571 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10 (-1429864725684944897)]} 0 0
[junit4:junit4] 2> 12576 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10 (-1429864725688090625)]} 0 1
[junit4:junit4] 2> 12577 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10010 (-1429864725688090624)]} 0 2
[junit4:junit4] 2> 12577 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10 (-1429864725688090625)]} 0 4
[junit4:junit4] 2> 12577 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10010 (-1429864725688090624)]} 0 4
[junit4:junit4] 2> 12580 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[11 (1429864725693333504)]} 0 1
[junit4:junit4] 2> 12582 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10011 (1429864725695430656)]} 0 1
[junit4:junit4] 2> 12586 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[11 (1429864725696479232)]} 0 1
[junit4:junit4] 2> 12587 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[11 (1429864725696479232)]} 0 5
[junit4:junit4] 2> 12587 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10011 (1429864725698576384)]} 0 1
[junit4:junit4] 2> 12589 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10011 (1429864725698576384)]} 0 5
[junit4:junit4] 2> 12636 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[12 (1429864725752053760)]} 0 1
[junit4:junit4] 2> 12638 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10012 (1429864725754150912)]} 0 1
[junit4:junit4] 2> 12642 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1429864725756248064)]} 0 1
[junit4:junit4] 2> 12643 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1429864725757296640)]} 0 1
[junit4:junit4] 2> 12643 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[12 (1429864725756248064)]} 0 4
[junit4:junit4] 2> 12644 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10012 (1429864725757296640)]} 0 5
[junit4:junit4] 2> 12646 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[13 (1429864725762539520)]} 0 1
[junit4:junit4] 2> 12647 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10013 (1429864725763588096)]} 0 1
[junit4:junit4] 2> 12652 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1429864725766733824)]} 0 1
[junit4:junit4] 2> 12652 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1429864725766733825)]} 0 1
[junit4:junit4] 2> 12653 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[13 (1429864725766733824)]} 0 4
[junit4:junit4] 2> 12653 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10013 (1429864725766733825)]} 0 4
[junit4:junit4] 2> 12703 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10014 (1429864725823356928)]} 0 1
[junit4:junit4] 2> 12704 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[14 (1429864725823356929)]} 0 2
[junit4:junit4] 2> 12709 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1429864725826502656)]} 0 1
[junit4:junit4] 2> 12709 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[14 (1429864725826502656)]} 0 3
[junit4:junit4] 2> 12710 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1429864725826502657)]} 0 2
[junit4:junit4] 2> 12710 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10014 (1429864725826502657)]} 0 4
[junit4:junit4] 2> 12784 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[14 (-1429864725908291584)]} 0 0
[junit4:junit4] 2> 12784 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10014 (-1429864725908291585)]} 0 0
[junit4:junit4] 2> 12789 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[14 (-1429864725911437312)]} 0 1
[junit4:junit4] 2> 12789 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10014 (-1429864725911437313)]} 0 0
[junit4:junit4] 2> 12790 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[14 (-1429864725911437312)]} 0 4
[junit4:junit4] 2> 12790 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10014 (-1429864725911437313)]} 0 3
[junit4:junit4] 2> 12793 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[15 (1429864725916680192)]} 0 2
[junit4:junit4] 2> 12793 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10015 (1429864725916680193)]} 0 1
[junit4:junit4] 2> 12798 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1429864725919825920)]} 0 1
[junit4:junit4] 2> 12798 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1429864725919825921)]} 0 0
[junit4:junit4] 2> 12799 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[15 (1429864725919825920)]} 0 4
[junit4:junit4] 2> 12799 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10015 (1429864725919825921)]} 0 4
[junit4:junit4] 2> 12890 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[16 (1429864726018392064)]} 0 1
[junit4:junit4] 2> 12890 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10016 (1429864726018392065)]} 0 1
[junit4:junit4] 2> 12896 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1429864726021537792)]} 0 1
[junit4:junit4] 2> 12896 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1429864726021537793)]} 0 1
[junit4:junit4] 2> 12896 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[16 (1429864726021537792)]} 0 4
[junit4:junit4] 2> 12897 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10016 (1429864726021537793)]} 0 5
[junit4:junit4] 2> 12915 T1353 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&state=recovering&nodeName=127.0.0.1:47543_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 12957 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10017 (1429864726088646657)]} 0 1
[junit4:junit4] 2> 12957 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[17 (1429864726088646656)]} 0 1
[junit4:junit4] 2> 12962 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1429864726091792384)]} 0 1
[junit4:junit4] 2> 12963 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10017 (1429864726091792384)]} 0 4
[junit4:junit4] 2> 12964 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1429864726091792385)]} 0 1
[junit4:junit4] 2> 12964 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[17 (1429864726091792385)]} 0 5
[junit4:junit4] 2> 12993 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10018 (1429864726126395392)]} 0 1
[junit4:junit4] 2> 12995 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[18 (1429864726128492544)]} 0 1
[junit4:junit4] 2> 12998 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1429864726129541120)]} 0 0
[junit4:junit4] 2> 12999 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10018 (1429864726129541120)]} 0 4
[junit4:junit4] 2> 13001 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1429864726131638272)]} 0 1
[junit4:junit4] 2> 13001 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[18 (1429864726131638272)]} 0 4
[junit4:junit4] 2> 13022 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10019 (1429864726157852672)]} 0 1
[junit4:junit4] 2> 13024 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[19 (1429864726158901248)]} 0 1
[junit4:junit4] 2> 13028 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1429864726160998400)]} 0 1
[junit4:junit4] 2> 13028 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10019 (1429864726160998400)]} 0 3
[junit4:junit4] 2> 13029 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1429864726162046976)]} 0 1
[junit4:junit4] 2> 13030 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[19 (1429864726162046976)]} 0 4
[junit4:junit4] 2> 13099 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10020 (1429864726238593024)]} 0 1
[junit4:junit4] 2> 13100 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[20 (1429864726239641600)]} 0 0
[junit4:junit4] 2> 13104 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1429864726240690176)]} 0 0
[junit4:junit4] 2> 13105 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10020 (1429864726240690176)]} 0 4
[junit4:junit4] 2> 13105 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1429864726241738752)]} 0 0
[junit4:junit4] 2> 13106 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[20 (1429864726241738752)]} 0 4
[junit4:junit4] 2> 13162 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10021 (1429864726303604736)]} 0 1
[junit4:junit4] 2> 13162 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[21 (1429864726304653312)]} 0 0
[junit4:junit4] 2> 13168 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1429864726306750464)]} 0 1
[junit4:junit4] 2> 13168 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1429864726306750465)]} 0 1
[junit4:junit4] 2> 13169 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10021 (1429864726306750464)]} 0 5
[junit4:junit4] 2> 13169 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[21 (1429864726306750465)]} 0 5
[junit4:junit4] 2> 13204 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10019 (-1429864726347644928)]} 0 1
[junit4:junit4] 2> 13204 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[19 (-1429864726348693504)]} 0 0
[junit4:junit4] 2> 13208 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1429864726350790657)]} 0 0
[junit4:junit4] 2> 13208 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1429864726350790656)]} 0 0
[junit4:junit4] 2> 13209 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10019 (-1429864726350790656)]} 0 3
[junit4:junit4] 2> 13209 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[19 (-1429864726350790657)]} 0 3
[junit4:junit4] 2> 13212 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10022 (1429864726356033536)]} 0 1
[junit4:junit4] 2> 13213 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[22 (1429864726356033537)]} 0 1
[junit4:junit4] 2> 13218 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1429864726359179265)]} 0 1
[junit4:junit4] 2> 13218 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1429864726359179264)]} 0 1
[junit4:junit4] 2> 13218 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[22 (1429864726359179265)]} 0 4
[junit4:junit4] 2> 13219 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10022 (1429864726359179264)]} 0 5
[junit4:junit4] 2> 13272 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[21 (-1429864726419996672)]} 0 0
[junit4:junit4] 2> 13272 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10021 (-1429864726419996673)]} 0 0
[junit4:junit4] 2> 13276 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[21 (-1429864726423142401)]} 0 0
[junit4:junit4] 2> 13277 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[21 (-1429864726423142401)]} 0 2
[junit4:junit4] 2> 13277 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10021 (-1429864726423142400)]} 0 0
[junit4:junit4] 2> 13278 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10021 (-1429864726423142400)]} 0 3
[junit4:junit4] 2> 13280 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[23 (1429864726427336704)]} 0 1
[junit4:junit4] 2> 13281 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10023 (1429864726428385280)]} 0 1
[junit4:junit4] 2> 13285 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1429864726430482432)]} 0 0
[junit4:junit4] 2> 13286 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1429864726431531008)]} 0 1
[junit4:junit4] 2> 13286 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[23 (1429864726430482432)]} 0 4
[junit4:junit4] 2> 13287 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10023 (1429864726431531008)]} 0 4
[junit4:junit4] 2> 13359 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[24 (1429864726511222784)]} 0 1
[junit4:junit4] 2> 13360 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10024 (1429864726511222785)]} 0 2
[junit4:junit4] 2> 13365 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1429864726514368512)]} 0 1
[junit4:junit4] 2> 13367 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1429864726515417088)]} 0 1
[junit4:junit4] 2> 13367 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10024 (1429864726515417088)]} 0 4
[junit4:junit4] 2> 13368 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[24 (1429864726514368512)]} 0 6
[junit4:junit4] 2> 13446 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10025 (1429864726601400320)]} 0 1
[junit4:junit4] 2> 13447 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[25 (1429864726602448896)]} 0 1
[junit4:junit4] 2> 13451 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1429864726604546048)]} 0 1
[junit4:junit4] 2> 13452 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10025 (1429864726604546048)]} 0 4
[junit4:junit4] 2> 13453 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1429864726605594624)]} 0 1
[junit4:junit4] 2> 13454 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[25 (1429864726605594624)]} 0 5
[junit4:junit4] 2> 13531 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[26 (1429864726691577856)]} 0 1
[junit4:junit4] 2> 13533 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10026 (1429864726692626432)]} 0 1
[junit4:junit4] 2> 13537 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1429864726694723584)]} 0 1
[junit4:junit4] 2> 13537 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[26 (1429864726694723584)]} 0 4
[junit4:junit4] 2> 13538 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1429864726695772160)]} 0 1
[junit4:junit4] 2> 13538 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10026 (1429864726695772160)]} 0 3
[junit4:junit4] 2> 13622 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[26 (-1429864726786998272)]} 0 0
[junit4:junit4] 2> 13622 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10026 (-1429864726786998273)]} 0 0
[junit4:junit4] 2> 13626 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1429864726789095424)]} 0 0
[junit4:junit4] 2> 13627 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[26 (-1429864726789095424)]} 0 3
[junit4:junit4] 2> 13627 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1429864726789095425)]} 0 0
[junit4:junit4] 2> 13628 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10026 (-1429864726789095425)]} 0 4
[junit4:junit4] 2> 13630 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[27 (1429864726794338304)]} 0 1
[junit4:junit4] 2> 13630 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10027 (1429864726795386880)]} 0 0
[junit4:junit4] 2> 13635 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1429864726797484032)]} 0 1
[junit4:junit4] 2> 13636 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1429864726797484033)]} 0 1
[junit4:junit4] 2> 13636 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[27 (1429864726797484032)]} 0 4
[junit4:junit4] 2> 13636 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10027 (1429864726797484033)]} 0 4
[junit4:junit4] 2> 13639 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[27 (-1429864726804824064)]} 0 0
[junit4:junit4] 2> 13639 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10027 (-1429864726804824065)]} 0 0
[junit4:junit4] 2> 13643 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[27 (-1429864726807969793)]} 0 0
[junit4:junit4] 2> 13643 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10027 (-1429864726807969792)]} 0 0
[junit4:junit4] 2> 13644 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10027 (-1429864726807969792)]} 0 3
[junit4:junit4] 2> 13644 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[27 (-1429864726807969793)]} 0 3
[junit4:junit4] 2> 13647 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10028 (1429864726812164096)]} 0 1
[junit4:junit4] 2> 13647 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[28 (1429864726812164097)]} 0 1
[junit4:junit4] 2> 13652 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1429864726815309825)]} 0 1
[junit4:junit4] 2> 13653 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1429864726815309824)]} 0 1
[junit4:junit4] 2> 13653 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[28 (1429864726815309825)]} 0 4
[junit4:junit4] 2> 13653 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10028 (1429864726815309824)]} 0 4
[junit4:junit4] 2> 13710 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10029 (1429864726879272960)]} 0 1
[junit4:junit4] 2> 13711 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[29 (1429864726879272961)]} 0 2
[junit4:junit4] 2> 13716 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1429864726882418688)]} 0 1
[junit4:junit4] 2> 13716 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1429864726882418689)]} 0 1
[junit4:junit4] 2> 13717 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10029 (1429864726882418688)]} 0 5
[junit4:junit4] 2> 13717 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[29 (1429864726882418689)]} 0 4
[junit4:junit4] 2> 13769 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[30 (1429864726940090369)]} 0 1
[junit4:junit4] 2> 13770 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10030 (1429864726940090368)]} 0 2
[junit4:junit4] 2> 13774 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1429864726943236096)]} 0 0
[junit4:junit4] 2> 13775 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[30 (1429864726943236096)]} 0 4
[junit4:junit4] 2> 13776 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1429864726945333248)]} 0 1
[junit4:junit4] 2> 13777 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10030 (1429864726945333248)]} 0 5
[junit4:junit4] 2> 13816 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[31 (1429864726990422016)]} 0 0
[junit4:junit4] 2> 13817 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10031 (1429864726991470592)]} 0 0
[junit4:junit4] 2> 13822 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1429864726993567744)]} 0 1
[junit4:junit4] 2> 13822 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1429864726993567745)]} 0 1
[junit4:junit4] 2> 13823 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[31 (1429864726993567744)]} 0 4
[junit4:junit4] 2> 13823 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10031 (1429864726993567745)]} 0 4
[junit4:junit4] 2> 13850 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[32 (1429864727025025024)]} 0 1
[junit4:junit4] 2> 13851 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10032 (1429864727025025025)]} 0 2
[junit4:junit4] 2> 13856 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1429864727029219329)]} 0 1
[junit4:junit4] 2> 13856 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1429864727029219328)]} 0 1
[junit4:junit4] 2> 13856 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10032 (1429864727029219329)]} 0 4
[junit4:junit4] 2> 13857 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[32 (1429864727029219328)]} 0 5
[junit4:junit4] 2> 13871 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10030 (-1429864727048093696)]} 0 0
[junit4:junit4] 2> 13871 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[30 (-1429864727048093697)]} 0 0
[junit4:junit4] 2> 13875 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10030 (-1429864727051239424)]} 0 0
[junit4:junit4] 2> 13876 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[30 (-1429864727051239425)]} 0 1
[junit4:junit4] 2> 13876 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[30 (-1429864727051239425)]} 0 3
[junit4:junit4] 2> 13876 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10030 (-1429864727051239424)]} 0 3
[junit4:junit4] 2> 13879 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10033 (1429864727055433728)]} 0 1
[junit4:junit4] 2> 13879 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[33 (1429864727055433729)]} 0 1
[junit4:junit4] 2> 13884 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1429864727058579456)]} 0 1
[junit4:junit4] 2> 13885 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1429864727058579457)]} 0 1
[junit4:junit4] 2> 13885 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10033 (1429864727058579456)]} 0 4
[junit4:junit4] 2> 13885 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[33 (1429864727058579457)]} 0 4
[junit4:junit4] 2> 13913 T1316 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 47543
[junit4:junit4] 2> 13913 T1316 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=13096126
[junit4:junit4] 2> 13917 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[34 (1429864727095279616)]} 0 1
[junit4:junit4] 2> 13917 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10034 (1429864727095279617)]} 0 1
[junit4:junit4] 2> 13922 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1429864727098425344)]} 0 1
[junit4:junit4] 2> 13923 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1429864727098425345)]} 0 1
[junit4:junit4] 2> 13923 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[34 (1429864727098425344)]} 0 4
[junit4:junit4] 2> 13923 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10034 (1429864727098425345)]} 0 4
[junit4:junit4] 2> 13980 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13981 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47543"}
[junit4:junit4] 2> 13986 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13986 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13986 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13986 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14021 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10035 (1429864727204331520)]} 0 1
[junit4:junit4] 2> 14021 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[35 (1429864727204331521)]} 0 1
[junit4:junit4] 2> 14024 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[35 (1429864727208525824)]} 0 1
[junit4:junit4] 2> 14024 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10035 (1429864727207477248)]} 0 1
[junit4:junit4] 2> 14087 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10031 (-1429864727274586112)]} 0 0
[junit4:junit4] 2> 14087 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[31 (-1429864727274586113)]} 0 0
[junit4:junit4] 2> 14092 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10031 (-1429864727279828992)]} 0 0
[junit4:junit4] 2> 14092 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[31 (-1429864727279828993)]} 0 0
[junit4:junit4] 2> 14095 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[36 (1429864727282974721)]} 0 0
[junit4:junit4] 2> 14096 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10036 (1429864727282974720)]} 0 1
[junit4:junit4] 2> 14098 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[36 (1429864727286120448)]} 0 1
[junit4:junit4] 2> 14099 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10036 (1429864727286120449)]} 0 1
[junit4:junit4] 2> 14169 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10033 (-1429864727360569344)]} 0 1
[junit4:junit4] 2> 14169 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[33 (-1429864727360569345)]} 0 1
[junit4:junit4] 2> 14171 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[33 (-1429864727362666496)]} 0 0
[junit4:junit4] 2> 14171 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10033 (-1429864727362666497)]} 0 0
[junit4:junit4] 2> 14174 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[37 (1429864727364763648)]} 0 1
[junit4:junit4] 2> 14175 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10037 (1429864727364763649)]} 0 2
[junit4:junit4] 2> 14177 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10037 (1429864727368957952)]} 0 0
[junit4:junit4] 2> 14177 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[37 (1429864727368957953)]} 0 0
[junit4:junit4] 2> 14241 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[38 (1429864727435018240)]} 0 1
[junit4:junit4] 2> 14241 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10038 (1429864727436066816)]} 0 0
[junit4:junit4] 2> 14244 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[38 (1429864727438163968)]} 0 1
[junit4:junit4] 2> 14245 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10038 (1429864727440261120)]} 0 0
[junit4:junit4] 2> 14282 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[35 (-1429864727479058432)]} 0 1
[junit4:junit4] 2> 14282 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10035 (-1429864727479058433)]} 0 0
[junit4:junit4] 2> 14285 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[35 (-1429864727482204160)]} 0 0
[junit4:junit4] 2> 14285 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10035 (-1429864727482204161)]} 0 0
[junit4:junit4] 2> 14287 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[39 (1429864727484301312)]} 0 0
[junit4:junit4] 2> 14288 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10039 (1429864727484301313)]} 0 1
[junit4:junit4] 2> 14290 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[39 (1429864727486398464)]} 0 1
[junit4:junit4] 2> 14291 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10039 (1429864727487447040)]} 0 1
[junit4:junit4] 2> 14324 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[40 (1429864727522050048)]} 0 1
[junit4:junit4] 2> 14324 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10040 (1429864727523098624)]} 0 1
[junit4:junit4] 2> 14327 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10040 (1429864727525195777)]} 0 1
[junit4:junit4] 2> 14328 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[40 (1429864727525195776)]} 0 2
[junit4:junit4] 2> 14350 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10041 (1429864727550361600)]} 0 1
[junit4:junit4] 2> 14351 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[41 (1429864727550361601)]} 0 1
[junit4:junit4] 2> 14353 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[41 (1429864727553507329)]} 0 0
[junit4:junit4] 2> 14354 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10041 (1429864727553507328)]} 0 1
[junit4:junit4] 2> 14419 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[42 (1429864727621664768)]} 0 0
[junit4:junit4] 2> 14419 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10042 (1429864727621664769)]} 0 1
[junit4:junit4] 2> 14422 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[42 (1429864727624810496)]} 0 1
[junit4:junit4] 2> 14423 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10042 (1429864727625859072)]} 0 1
[junit4:junit4] 2> 14458 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[43 (1429864727662559232)]} 0 1
[junit4:junit4] 2> 14459 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10043 (1429864727663607808)]} 0 1
[junit4:junit4] 2> 14462 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10043 (1429864727666753536)]} 0 1
[junit4:junit4] 2> 14462 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[43 (1429864727666753537)]} 0 1
[junit4:junit4] 2> 14552 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10040 (-1429864727762173952)]} 0 0
[junit4:junit4] 2> 14552 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[40 (-1429864727762173953)]} 0 0
[junit4:junit4] 2> 14555 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[40 (-1429864727765319680)]} 0 0
[junit4:junit4] 2> 14555 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10040 (-1429864727765319681)]} 0 0
[junit4:junit4] 2> 14558 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10044 (1429864727767416833)]} 0 1
[junit4:junit4] 2> 14558 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[44 (1429864727767416832)]} 0 1
[junit4:junit4] 2> 14561 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10044 (1429864727770562560)]} 0 1
[junit4:junit4] 2> 14561 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[44 (1429864727770562561)]} 0 1
[junit4:junit4] 2> 14578 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10045 (1429864727788388352)]} 0 1
[junit4:junit4] 2> 14579 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[45 (1429864727788388353)]} 0 2
[junit4:junit4] 2> 14581 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[45 (1429864727791534080)]} 0 1
[junit4:junit4] 2> 14582 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10045 (1429864727791534081)]} 0 2
[junit4:junit4] 2> 14660 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10046 (1429864727875420161)]} 0 0
[junit4:junit4] 2> 14661 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[46 (1429864727875420160)]} 0 1
[junit4:junit4] 2> 14664 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10046 (1429864727878565888)]} 0 1
[junit4:junit4] 2> 14664 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[46 (1429864727878565889)]} 0 1
[junit4:junit4] 2> 14673 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10042 (-1429864727889051649)]} 0 0
[junit4:junit4] 2> 14673 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[42 (-1429864727889051648)]} 0 0
[junit4:junit4] 2> 14676 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[42 (-1429864727891148800)]} 0 1
[junit4:junit4] 2> 14676 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10042 (-1429864727892197376)]} 0 0
[junit4:junit4] 2> 14678 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[47 (1429864727894294528)]} 0 1
[junit4:junit4] 2> 14679 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10047 (1429864727894294529)]} 0 1
[junit4:junit4] 2> 14681 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[47 (1429864727896391680)]} 0 1
[junit4:junit4] 2> 14681 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10047 (1429864727897440256)]} 0 1
[junit4:junit4] 2> 14752 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[48 (1429864727970840576)]} 0 1
[junit4:junit4] 2> 14752 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10048 (1429864727970840577)]} 0 1
[junit4:junit4] 2> 14754 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[48 (1429864727973986304)]} 0 0
[junit4:junit4] 2> 14755 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10048 (1429864727973986305)]} 0 1
[junit4:junit4] 2> 14848 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10043 (-1429864728072552449)]} 0 0
[junit4:junit4] 2> 14848 T1332 C78 P44407 /update {wt=javabin&version=2} {delete=[43 (-1429864728072552448)]} 0 0
[junit4:junit4] 2> 14851 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10043 (-1429864728075698176)]} 0 1
[junit4:junit4] 2> 14851 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[43 (-1429864728075698177)]} 0 0
[junit4:junit4] 2> 14854 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10049 (1429864728077795328)]} 0 1
[junit4:junit4] 2> 14854 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[49 (1429864728077795329)]} 0 1
[junit4:junit4] 2> 14857 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10049 (1429864728080941056)]} 0 1
[junit4:junit4] 2> 14857 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[49 (1429864728080941057)]} 0 1
[junit4:junit4] 2> 14872 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10046 (-1429864728097718272)]} 0 0
[junit4:junit4] 2> 14873 T1332 C78 P44407 /update {wt=javabin&version=2} {delete=[46 (-1429864728098766848)]} 0 1
[junit4:junit4] 2> 14874 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10046 (-1429864728099815424)]} 0 0
[junit4:junit4] 2> 14875 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[46 (-1429864728100864000)]} 0 0
[junit4:junit4] 2> 14877 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10050 (1429864728102961152)]} 0 1
[junit4:junit4] 2> 14878 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[50 (1429864728102961153)]} 0 1
[junit4:junit4] 2> 14880 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10050 (1429864728105058304)]} 0 1
[junit4:junit4] 2> 14881 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[50 (1429864728107155456)]} 0 0
[junit4:junit4] 2> 14902 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10051 (1429864728128126976)]} 0 1
[junit4:junit4] 2> 14905 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10051 (1429864728132321280)]} 0 0
[junit4:junit4] 2> 14906 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[51 (1429864728133369856)]} 0 0
[junit4:junit4] 2> 14909 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[51 (1429864728136515584)]} 0 1
[junit4:junit4] 2> C77_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47543_, base_url=http://127.0.0.1:47543}
[junit4:junit4] 2> 14916 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43698/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 14916 T1377 C77 P47543 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:47543 START replicas=[http://127.0.0.1:43698/collection1/] nUpdates=100
[junit4:junit4] 2> 14918 T1377 C77 P47543 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 14918 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 14919 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 14918 T1351 C79 P43698 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 14919 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 14919 T1377 C77 P47543 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 14920 T1377 C77 P47543 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43698/collection1/. core=collection1
[junit4:junit4] 2> 14920 T1377 C77 P47543 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14924 T1353 C79 P43698 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14946 T1353 C79 P43698 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1f24669 lockFactory=org.apache.lucene.store.NativeFSLockFactory@123a825),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1f24669 lockFactory=org.apache.lucene.store.NativeFSLockFactory@123a825),segFN=segments_2,generation=2,filenames=[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt]
[junit4:junit4] 2> 14948 T1353 C79 P43698 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt]
[junit4:junit4] 2> 14999 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10052 (1429864728229838848)]} 0 1
[junit4:junit4] 2> 15002 T1353 C79 P43698 oass.SolrIndexSearcher.<init> Opening Searcher@125dc85 realtime
[junit4:junit4] 2> 15002 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[52 (1429864728232984576)]} 0 1
[junit4:junit4] 2> 15003 T1353 C79 P43698 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 15003 T1353 C79 P43698 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 79
[junit4:junit4] 2> 15003 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10052 (1429864728232984576)]} 0 2
[junit4:junit4] 2> 15004 T1377 C77 P47543 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 15004 T1377 C77 P47543 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 15007 T1352 C79 P43698 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 15007 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10053 (1429864728238227456)]} 0 2
[junit4:junit4] 2> 15007 T1352 C79 P43698 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 15008 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[52 (1429864728239276032)]} 0 1
[junit4:junit4] 2> 15008 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 15009 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 15009 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 15010 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[53 (1429864728242421760)]} 0 0
[junit4:junit4] 2> 15010 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10053 (1429864728242421760)]} 0 0
[junit4:junit4] 2> 15011 T1354 C79 P43698 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 15012 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 55
[junit4:junit4] 2> 15013 T1377 C77 P47543 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index.20130318174747263 forceNew: false
[junit4:junit4] 2> 15013 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[53 (1429864728245567488)]} 0 1
[junit4:junit4] 2> 15014 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)) fullCopy=true
[junit4:junit4] 2> 15016 T1352 C79 P43698 REQ /replication {file=_1_NestedPulsing_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15019 T1353 C79 P43698 REQ /replication {file=_1_TestBloomFilteredLucene41Postings_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15023 T1354 C79 P43698 REQ /replication {file=_0_TestBloomFilteredLucene41Postings_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15035 T1351 C79 P43698 REQ /replication {file=_0_TestBloomFilteredLucene41Postings_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15039 T1352 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15042 T1353 C79 P43698 REQ /replication {file=_1.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15046 T1354 C79 P43698 REQ /replication {file=_0_TestBloomFilteredLucene41Postings_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15049 T1351 C79 P43698 REQ /replication {file=_1_Asserting_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15052 T1352 C79 P43698 REQ /replication {file=_0_NestedPulsing_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15054 T1353 C79 P43698 REQ /replication {file=_1_Asserting_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15057 T1354 C79 P43698 REQ /replication {file=_1_NestedPulsing_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15060 T1351 C79 P43698 REQ /replication {file=_0_Asserting_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15064 T1352 C79 P43698 REQ /replication {file=_0_NestedPulsing_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15089 T1353 C79 P43698 REQ /replication {file=_1_Asserting_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15092 T1354 C79 P43698 REQ /replication {file=_1.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15095 T1351 C79 P43698 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15102 T1352 C79 P43698 REQ /replication {file=_0_TestBloomFilteredLucene41Postings_0.blm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15102 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10054 (1429864728337842177)]} 0 1
[junit4:junit4] 2> 15102 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[54 (1429864728337842176)]} 0 1
[junit4:junit4] 2> 15105 T1351 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15105 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10054 (1429864728340987904)]} 0 1
[junit4:junit4] 2> 15105 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[54 (1429864728340987905)]} 0 1
[junit4:junit4] 2> 15108 T1352 C79 P43698 REQ /replication {file=_1.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15111 T1351 C79 P43698 REQ /replication {file=_1_NestedPulsing_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15114 T1354 C79 P43698 REQ /replication {file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15117 T1353 C79 P43698 REQ /replication {file=_1_TestBloomFilteredLucene41Postings_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15135 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[10055 (1429864728373493760)]} 0 1
[junit4:junit4] 2> 15136 T1352 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15137 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[55 (1429864728374542336)]} 0 1
[junit4:junit4] 2> 15138 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10055 (1429864728376639488)]} 0 0
[junit4:junit4] 2> 15139 T1354 C79 P43698 REQ /replication {file=_1.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15141 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[55 (1429864728377688064)]} 0 2
[junit4:junit4] 2> 15142 T1352 C79 P43698 REQ /replication {file=_1.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15145 T1351 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15148 T1354 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15151 T1353 C79 P43698 REQ /replication {file=_0_Asserting_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15154 T1352 C79 P43698 REQ /replication {file=_1_TestBloomFilteredLucene41Postings_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15157 T1351 C79 P43698 REQ /replication {file=_0_Asserting_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15157 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[47 (-1429864728396562432)]} 0 0
[junit4:junit4] 2> 15159 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[47 (-1429864728398659584)]} 0 0
[junit4:junit4] 2> 15160 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10047 (-1429864728399708160)]} 0 0
[junit4:junit4] 2> 15161 T1353 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15162 T1352 C79 P43698 /update {wt=javabin&version=2} {delete=[10047 (-1429864728401805312)]} 0 0
[junit4:junit4] 2> 15162 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[56 (1429864728401805312)]} 0 0
[junit4:junit4] 2> 15178 T1354 C79 P43698 REQ /replication {file=_1_Asserting_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15178 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[56 (1429864728417533952)]} 0 1
[junit4:junit4] 2> 15179 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10056 (1429864728418582528)]} 0 1
[junit4:junit4] 2> 15181 T1353 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15182 T1377 C77 P47543 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_1_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4] 2> 15182 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10056 (1429864728421728256)]} 0 1
[junit4:junit4] 2> 15185 T1351 C79 P43698 REQ /replication {file=_1_TestBloomFilteredLucene41Postings_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15192 T1354 C79 P43698 REQ /replication {file=_0_NestedPulsing_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15195 T1353 C79 P43698 REQ /replication {file=_0_NestedPulsing_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15199 T1352 C79 P43698 REQ /replication {file=_1_NestedPulsing_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15203 T1351 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15203 T1377 C77 P47543 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_0_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4] 2> 15206 T1354 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15210 T1353 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15221 T1352 C79 P43698 REQ /replication {file=_0.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15225 T1351 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15229 T1354 C79 P43698 REQ /replication {file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15232 T1353 C79 P43698 REQ /replication {file=_0_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15236 T1352 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15239 T1351 C79 P43698 REQ /replication {file=_0.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15242 T1354 C79 P43698 REQ /replication {file=_1_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15245 T1353 C79 P43698 REQ /replication {file=_1_TestBloomFilteredLucene41Postings_0.blm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15249 T1352 C79 P43698 REQ /replication {file=_1_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15255 T1351 C79 P43698 REQ /replication {file=_1.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15258 T1354 C79 P43698 REQ /replication {file=_0_TestBloomFilteredLucene41Postings_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15266 T1353 C79 P43698 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15268 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[50 (-1429864728511905792)]} 0 1
[junit4:junit4] 2> 15269 T1352 C79 P43698 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15270 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[50 (-1429864728515051520)]} 0 0
[junit4:junit4] 2> 15272 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[10050 (-1429864728517148672)]} 0 1
[junit4:junit4] 2> 15272 T1354 C79 P43698 REQ /replication {file=_0_Asserting_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15273 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[57 (1429864728517148673)]} 0 1
[junit4:junit4] 2> 15274 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[10050 (-1429864728519245824)]} 0 0
[junit4:junit4] 2> 15275 T1351 C79 P43698 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15275 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[57 (1429864728520294400)]} 0 0
[junit4:junit4] 2> 15276 T1377 C77 P47543 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 15276 T1377 C77 P47543 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130318174747263
[junit4:junit4] 2> 15277 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10057 (1429864728521342976)]} 0 1
[junit4:junit4] 2> 15278 T1377 C77 P47543 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 15278 T1377 C77 P47543 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 15278 T1377 C77 P47543 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4] 2> 15280 T1377 C77 P47543 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index.20130318174747263
[junit4:junit4] 2> 15280 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10057 (1429864728525537280)]} 0 1
[junit4:junit4] 2> 15282 T1377 C77 P47543 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)),segFN=segments_2,generation=2,filenames=[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt]
[junit4:junit4] 2> 15283 T1377 C77 P47543 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt]
[junit4:junit4] 2> 15284 T1377 C77 P47543 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 15286 T1377 C77 P47543 oass.SolrIndexSearcher.<init> Opening Searcher@8226d2 main
[junit4:junit4] 2> 15287 T1376 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8226d2 main{StandardDirectoryReader(segments_2:5:nrt _0(4.3):C73/24 _1(4.3):C29/12)}
[junit4:junit4] 2> 15288 T1377 C77 P47543 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@17094d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@97b29a)
[junit4:junit4] 2> 15288 T1377 C77 P47543 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/index
[junit4:junit4] 2> 15289 T1377 C77 P47543 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@17094d7 lockFactory=org.apache.lucene.store.NativeFSLockFactory@97b29a)
[junit4:junit4] 2> 15292 T1377 C77 P47543 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 15292 T1385 C77 P47543 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363625252253/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=7070
[junit4:junit4] 2> 15294 T1385 C77 P47543 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 15297 T1385 C77 P47543 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)),segFN=segments_2,generation=2,filenames=[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, segments_2, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt]
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)),segFN=segments_3,generation=3,filenames=[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt]
[junit4:junit4] 2> 15298 T1385 C77 P47543 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt]
[junit4:junit4] 2> 15299 T1385 C77 P47543 oass.SolrIndexSearcher.<init> Opening Searcher@121cbac main
[junit4:junit4] 2> 15300 T1385 C77 P47543 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 15301 T1376 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@121cbac main{StandardDirectoryReader(segments_2:5:nrt _0(4.3):C73/24 _1(4.3):C29/12)}
[junit4:junit4] 2> 15302 T1385 C77 P47543 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=7070}
[junit4:junit4] 2> 15302 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 15302 T1377 C77 P47543 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 15303 T1377 C77 P47543 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15304 T1377 C77 P47543 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 15325 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[58 (1429864728571674624)]} 0 1
[junit4:junit4] 2> 15329 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[58 (1429864728575868928)]} 0 1
[junit4:junit4] 2> 15331 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10058 (1429864728577966080)]} 0 1
[junit4:junit4] 2> 15334 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10058 (1429864728581111808)]} 0 1
[junit4:junit4] 2> 15370 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[59 (1429864728618860544)]} 0 1
[junit4:junit4] 2> 15373 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[59 (1429864728622006272)]} 0 1
[junit4:junit4] 2> 15375 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[10059 (1429864728625152000)]} 0 1
[junit4:junit4] 2> 15378 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[10059 (1429864728628297728)]} 0 0
[junit4:junit4] 2> 15465 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[51 (-1429864728719523840)]} 0 0
[junit4:junit4] 2> 15468 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[51 (-1429864728722669568)]} 0 1
[junit4:junit4] 2> 15470 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[60 (1429864728724766720)]} 0 0
[junit4:junit4] 2> 15471 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10051 (-1429864728725815296)]} 0 0
[junit4:junit4] 2> 15473 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10051 (-1429864728727912449)]} 0 0
[junit4:junit4] 2> 15473 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[60 (1429864728727912448)]} 0 1
[junit4:junit4] 2> 15477 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10060 (1429864728730009600)]} 0 2
[junit4:junit4] 2> 15480 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10060 (1429864728734203904)]} 0 1
[junit4:junit4] 2> 15490 T1339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15491 T1339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:47543_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47543"}
[junit4:junit4] 2> 15495 T1374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15495 T1345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15496 T1338 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15496 T1358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15518 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[53 (-1429864728775098368)]} 0 0
[junit4:junit4] 2> C77_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:47543_, base_url=http://127.0.0.1:47543}
[junit4:junit4] 2> 15524 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[53 (-1429864728777195520)]} 0 2
[junit4:junit4] 2> 15525 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10053 (-1429864728782438400)]} 0 1
[junit4:junit4] 2> 15525 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[53 (-1429864728777195520)]} 0 5
[junit4:junit4] 2> 15528 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[61 (1429864728784535552)]} 0 1
[junit4:junit4] 2> 15529 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10053 (-1429864728784535552)]} 0 0
[junit4:junit4] 2> 15530 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10053 (-1429864728784535552)]} 0 3
[junit4:junit4] 2> 15533 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10061 (1429864728789778432)]} 0 1
[junit4:junit4] 2> 15533 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[61 (1429864728787681280)]} 0 1
[junit4:junit4] 2> 15534 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[61 (1429864728787681280)]} 0 4
[junit4:junit4] 2> 15538 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10061 (1429864728792924160)]} 0 1
[junit4:junit4] 2> 15539 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10061 (1429864728792924160)]} 0 4
[junit4:junit4] 2> 15540 T1334 C78 P44407 /update {wt=javabin&version=2} {delete=[54 (-1429864728798167040)]} 0 0
[junit4:junit4] 2> 15544 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[54 (-1429864728800264192)]} 0 0
[junit4:junit4] 2> 15545 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[54 (-1429864728800264192)]} 0 3
[junit4:junit4] 2> 15545 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[10054 (-1429864728803409920)]} 0 0
[junit4:junit4] 2> 15547 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[62 (1429864728805507072)]} 0 0
[junit4:junit4] 2> 15551 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10054 (-1429864728805507072)]} 0 0
[junit4:junit4] 2> 15552 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10054 (-1429864728805507072)]} 0 5
[junit4:junit4] 2> 15554 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10062 (1429864728812847104)]} 0 0
[junit4:junit4] 2> 15555 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[62 (1429864728810749952)]} 0 1
[junit4:junit4] 2> 15556 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[62 (1429864728810749952)]} 0 4
[junit4:junit4] 2> 15559 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10062 (1429864728814944256)]} 0 1
[junit4:junit4] 2> 15560 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10062 (1429864728814944256)]} 0 4
[junit4:junit4] 2> 15647 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[55 (-1429864728910364672)]} 0 0
[junit4:junit4] 2> 15651 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[55 (-1429864728912461824)]} 0 0
[junit4:junit4] 2> 15651 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10055 (-1429864728914558976)]} 0 0
[junit4:junit4] 2> 15652 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[55 (-1429864728912461824)]} 0 3
[junit4:junit4] 2> 15654 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[63 (1429864728917704704)]} 0 0
[junit4:junit4] 2> 15655 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10055 (-1429864728916656128)]} 0 0
[junit4:junit4] 2> 15655 T1351 C79 P43698 /update {wt=javabin&version=2} {delete=[10055 (-1429864728916656128)]} 0 2
[junit4:junit4] 2> 15658 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10063 (1429864728920850432)]} 0 1
[junit4:junit4] 2> 15659 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[63 (1429864728919801856)]} 0 0
[junit4:junit4] 2> 15660 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[63 (1429864728919801856)]} 0 4
[junit4:junit4] 2> 15663 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10063 (1429864728923996160)]} 0 1
[junit4:junit4] 2> 15663 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10063 (1429864728923996160)]} 0 3
[junit4:junit4] 2> 15761 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[64 (1429864729029902336)]} 0 0
[junit4:junit4] 2> 15765 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10064 (1429864729034096640)]} 0 0
[junit4:junit4] 2> 15767 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[64 (1429864729033048064)]} 0 1
[junit4:junit4] 2> 15767 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[64 (1429864729033048064)]} 0 3
[junit4:junit4] 2> 15771 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10064 (1429864729037242368)]} 0 0
[junit4:junit4] 2> 15772 T1351 C79 P43698 /update {wt=javabin&version=2} {add=[10064 (1429864729037242368)]} 0 4
[junit4:junit4] 2> 15836 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[65 (1429864729108545536)]} 0 1
[junit4:junit4] 2> 15841 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10065 (1429864729112739840)]} 0 1
[junit4:junit4] 2> 15842 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[65 (1429864729111691264)]} 0 1
[junit4:junit4] 2> 15842 T1354 C79 P43698 /update {wt=javabin&version=2} {add=[65 (1429864729111691264)]} 0 4
[junit4:junit4] 2> 15846 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10065 (1429864729115885568)]} 0 1
[junit4:junit4] 2> 15846 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[10065 (1429864729115885568)]} 0 3
[junit4:junit4] 2> 15863 T1331 C78 P44407 /update {wt=javabin&version=2} {delete=[58 (-1429864729136857088)]} 0 0
[junit4:junit4] 2> 15866 T1333 C78 P44407 /update {wt=javabin&version=2} {delete=[10058 (-1429864729140002816)]} 0 0
[junit4:junit4] 2> 15867 T1369 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[58 (-1429864729138954240)]} 0 0
[junit4:junit4] 2> 15867 T1353 C79 P43698 /update {wt=javabin&version=2} {delete=[58 (-1429864729138954240)]} 0 2
[junit4:junit4] 2> 15870 T1332 C78 P44407 /update {wt=javabin&version=2} {add=[66 (1429864729143148544)]} 0 1
[junit4:junit4] 2> 15871 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10058 (-1429864729143148544)]} 0 0
[junit4:junit4] 2> 15872 T1354 C79 P43698 /update {wt=javabin&version=2} {delete=[10058 (-1429864729143148544)]} 0 3
[junit4:junit4] 2> 15874 T1334 C78 P44407 /update {wt=javabin&version=2} {add=[10066 (1429864729148391424)]} 0 0
[junit4:junit4] 2> 15875 T1370 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[66 (1429864729147342848)]} 0 0
[junit4:junit4] 2> 15876 T1352 C79 P43698 /update {wt=javabin&version=2} {add=[66 (1429864729147342848)]} 0 3
[junit4:junit4] 2> 15879 T1368 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10066 (1429864729150488576)]} 0 0
[junit4:junit4] 2> 15880 T1353 C79 P43698 /update {wt=javabin&version=2} {add=[10066 (1429864729150488576)]} 0 4
[junit4:junit4] 2> 15915 T1316 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15916 T1316 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15919 T1316 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@aff120
[junit4:junit4] 2> 15928 T1316 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=12,adds=12,deletesById=8,deletesByQuery=0,errors=0,cumulative_adds=62,cumulative_deletesById=24,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15928 T1316 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 15929 T1316 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 15929 T1316 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 15938 T1331 C78 P44407 /update {wt=javabin&version=2} {add=[67 (1429864729214451712)]} 0 1
[junit4:junit4] 2> 15942 T1367 C77 P47543 /update {distrib.from=http://127.0.0.1:43698/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 0
[junit4:junit4] 2> 15943 T1333 C78 P44407 /update {wt=javabin&version=2} {add=[10067 (1429864729218646016)]} 0 2
[junit4:junit4] 2> 15943 T1367 C77 P47543 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:148)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:451)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:587)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:346)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:246)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 15945 T1316 C77 P47543 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)),segFN=segments_3,generation=3,filenames=[_1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _0_Asserting_0.doc, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockVariableIntBlock_0.tib, _1.fdx, _1.fdt, _0_MockVariableIntBlock_0.frq, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt]
[junit4:junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327)),segFN=segments_4,generation=4,filenames=[_2_NestedPulsing_0.tim, _1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _2_Asserting_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _2_NestedPulsing_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _1_MockVariableIntBlock_0.tii, _1.nvd, _2.fdt, _0_1.del, _1_NestedPulsing_0.pos, _1_TestBloomFilteredLucene41Postings_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _2_Asserting_0.tim, _1.fdx, _2.nvm, _2_Asserting_0.tip, _1.fdt, _2_TestBloomFilteredLucene41Postings_0.tip, _2_TestBloomFilteredLucene41Postings_0.pos, _2_NestedPulsing_0.pos, _2.nvd, _0_MockVariableIntBlock_0.frq, _2_TestBloomFilteredLucene41Postings_0.tim, _2_MockVariableIntBlock_0.skp, _2.si, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _2_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tim, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _1_TestBloomFilteredLucene41Postings_0.tip, _0_NestedPulsing_0.tip, _2_MockVariableIntBlock_0.doc, _0_NestedPulsing_0.tim, _1_NestedPulsing_0.doc, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _2_TestBloomFilteredLucene41Postings_0.blm, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _2_TestBloomFilteredLucene41Postings_0.doc, _0_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_1.del, _0.nvm, _2.fnm, _1_MockVariableIntBlock_0.frq, _1_TestBloomFilteredLucene41Postings_0.blm, _1.si, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fdx, _0_Asserting_0.pos, _0.fdt, _2_NestedPulsing_0.doc, segments_4]
[junit4:junit4] 2> 15946 T1367 C77 P47543 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@aff120
[junit4:junit4] 2> 15946 T1316 C77 P47543 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_2_NestedPulsing_0.tim, _1_NestedPulsing_0.tip, _1_TestBloomFilteredLucene41Postings_0.pos, _2_Asserting_0.pos, _0_TestBloomFilteredLucene41Postings_0.tim, _0_TestBloomFilteredLucene41Postings_0.tip, _0_MockVariableIntBlock_0.skp, _0_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _0_NestedPulsing_0.pos, _1_Asserting_0.tim, _1_Asserting_0.tip, _1_NestedPulsing_0.tim, _2_NestedPulsing_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _0_NestedPulsing_0.doc, _1.nvm, _1_Asserting_0.pos, _0.fnm, _0_TestBloomFilteredLucene41Posti
[...truncated too long message...]
] 2> 34311 T1358 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 34311 T1358 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 34751 T1319 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Asserting), text=PostingsFormat(name=NestedPulsing), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=MockVariableIntBlock(baseBlockSize=2), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), timestamp=PostingsFormat(name=Asserting), id=MockVariableIntBlock(baseBlockSize=2), a_t=PostingsFormat(name=Asserting), range_facet_sl=MockVariableIntBlock(baseBlockSize=2), range_facet_si=PostingsFormat(name=Asserting), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=Asserting), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=ar_MA, timezone=MET
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.7.0_15 (32-bit)/cpus=8,threads=1,free=18277720,total=88682496
[junit4:junit4] 2> NOTE: All tests run in this JVM: [LeaderElectionTest, PolyFieldTest, AnalysisAfterCoreReloadTest, TestQueryUtils, FastVectorHighlighterTest, TestBadConfig, TestReplicationHandler, TestCharFilters, AliasIntegrationTest, SynonymTokenizerTest, TestSolrDeletionPolicy1, ShardRoutingTest, QueryElevationComponentTest, PrimUtilsTest, BasicFunctionalityTest, DistributedTermsComponentTest, SuggesterFSTTest, ZkControllerTest, TestQuerySenderListener, SolrIndexSplitterTest, UUIDFieldTest, CoreAdminHandlerTest, RecoveryZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=573D2383CE00D58E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_MA -Dtests.timezone=MET -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1375, name=coreLoadExecutor-631-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@ff7054 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c71220))
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > ... 1 more
[junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1377, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a1f814 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1981327))
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] Completed on J0 in 35.00s, 1 test, 2 errors <<< FAILURES!
[...truncated 696 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 2 suite-level errors, 13 ignored (7 assumptions)
Total time: 50 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.7.0_15 -client -XX:+UseConcMarkSweepGC
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b79) - Build #
4728 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4728/
Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2303, name=coreLoadExecutor-1075-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2303, name=coreLoadExecutor-1075-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1569)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@507be456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@57686642)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2305, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2305, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1569)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9316 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 0 T2244 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_t/p
[junit4:junit4] 2> 3 T2244 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363662146658
[junit4:junit4] 2> 4 T2244 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 4 T2245 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 5 T2245 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T2245 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 5 T2245 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 6 T2245 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 6 T2245 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363662146658/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 104 T2244 oasc.ZkTestServer.run start zk server on port:58928
[junit4:junit4] 2> 105 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@53aa459e
[junit4:junit4] 2> 106 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 106 T2250 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 107 T2250 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 107 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59323
[junit4:junit4] 2> 107 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59323
[junit4:junit4] 2> 108 T2248 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 272 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650000 with negotiated timeout 10000 for client /127.0.0.1:59323
[junit4:junit4] 2> 272 T2250 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650000, negotiated timeout = 10000
[junit4:junit4] 2> 273 T2251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53aa459e name:ZooKeeperConnection Watcher:127.0.0.1:58928 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 273 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 274 T2244 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 279 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650000
[junit4:junit4] 2> 279 T2251 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 280 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59323 which had sessionid 0x13d80990c650000
[junit4:junit4] 2> 279 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650000 closed
[junit4:junit4] 2> 281 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@5d6db19b
[junit4:junit4] 2> 282 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 282 T2252 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 283 T2252 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 283 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59324
[junit4:junit4] 2> 284 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59324
[junit4:junit4] 2> 285 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650001 with negotiated timeout 10000 for client /127.0.0.1:59324
[junit4:junit4] 2> 285 T2252 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650001, negotiated timeout = 10000
[junit4:junit4] 2> 286 T2253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d6db19b name:ZooKeeperConnection Watcher:127.0.0.1:58928/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 286 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 286 T2244 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 292 T2244 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 295 T2244 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 298 T2244 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 301 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 302 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 319 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 320 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 323 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 323 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 325 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 325 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 326 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 327 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 329 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 329 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 330 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 331 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 332 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 333 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 334 T2244 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 334 T2244 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 336 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650001
[junit4:junit4] 2> 336 T2253 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 336 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59324 which had sessionid 0x13d80990c650001
[junit4:junit4] 2> 336 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650001 closed
[junit4:junit4] 2> 397 T2244 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 402 T2244 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48529
[junit4:junit4] 2> 403 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 403 T2244 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 403 T2244 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991
[junit4:junit4] 2> 403 T2244 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/solr.xml
[junit4:junit4] 2> 404 T2244 oasc.CoreContainer.<init> New CoreContainer 1709932105
[junit4:junit4] 2> 404 T2244 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/'
[junit4:junit4] 2> 404 T2244 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/'
[junit4:junit4] 2> 422 T2244 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 423 T2244 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 423 T2244 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 423 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 424 T2244 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 424 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 424 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 424 T2244 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 425 T2244 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 425 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 430 T2244 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 439 T2244 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58928/solr
[junit4:junit4] 2> 440 T2244 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 440 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f18bb1e
[junit4:junit4] 2> 441 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 441 T2263 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 441 T2263 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 441 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59326
[junit4:junit4] 2> 442 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59326
[junit4:junit4] 2> 443 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650002 with negotiated timeout 20000 for client /127.0.0.1:59326
[junit4:junit4] 2> 443 T2263 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650002, negotiated timeout = 20000
[junit4:junit4] 2> 444 T2264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f18bb1e name:ZooKeeperConnection Watcher:127.0.0.1:58928 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 444 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 445 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650002
[junit4:junit4] 2> 446 T2264 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 446 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59326 which had sessionid 0x13d80990c650002
[junit4:junit4] 2> 446 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650002 closed
[junit4:junit4] 2> 446 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 448 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@e3a0e25
[junit4:junit4] 2> 449 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449 T2265 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 450 T2265 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 450 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59327
[junit4:junit4] 2> 450 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59327
[junit4:junit4] 2> 451 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650003 with negotiated timeout 20000 for client /127.0.0.1:59327
[junit4:junit4] 2> 451 T2265 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650003, negotiated timeout = 20000
[junit4:junit4] 2> 452 T2266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3a0e25 name:ZooKeeperConnection Watcher:127.0.0.1:58928/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 454 T2244 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 456 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x7 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 457 T2244 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 460 T2244 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 462 T2244 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48529__t%2Fp
[junit4:junit4] 2> 463 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:delete cxid:0x12 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:48529__t%2Fp Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:48529__t%2Fp
[junit4:junit4] 2> 464 T2244 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48529__t%2Fp
[junit4:junit4] 2> 467 T2244 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 473 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:delete cxid:0x20 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 474 T2244 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 476 T2244 oasc.Overseer.start Overseer (id=89368962443575299-127.0.0.1:48529__t%2Fp-n_0000000000) starting
[junit4:junit4] 2> 477 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x25 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 478 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x26 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 480 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x28 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 481 T2244 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 484 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x2e zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 486 T2268 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 486 T2244 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 489 T2244 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 492 T2244 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 495 T2267 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 499 T2269 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/collection1
[junit4:junit4] 2> 500 T2269 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 500 T2269 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 501 T2269 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 502 T2269 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/collection1/'
[junit4:junit4] 2> 503 T2269 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/collection1/lib/README' to classloader
[junit4:junit4] 2> 503 T2269 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 542 T2269 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 569 T2269 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 571 T2269 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 574 T2269 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 917 T2269 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 922 T2269 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 924 T2269 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 933 T2269 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 935 T2269 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 938 T2269 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 939 T2269 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 939 T2269 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 939 T2269 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 940 T2269 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 940 T2269 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 940 T2269 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 940 T2269 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363662146991/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data/
[junit4:junit4] 2> 940 T2269 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3ef14b6a
[junit4:junit4] 2> 941 T2269 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 941 T2269 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data forceNew: false
[junit4:junit4] 2> 941 T2269 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data/index/
[junit4:junit4] 2> 942 T2269 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 942 T2269 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data/index forceNew: false
[junit4:junit4] 2> 943 T2269 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fbeed79 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bb07663),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 944 T2269 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 945 T2269 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 945 T2269 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 946 T2269 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 946 T2269 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 947 T2269 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 947 T2269 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 947 T2269 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 947 T2269 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 948 T2269 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 950 T2269 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 952 T2269 oass.SolrIndexSearcher.<init> Opening Searcher@16d3d5da main
[junit4:junit4] 2> 952 T2269 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/control/data/tlog
[junit4:junit4] 2> 953 T2269 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 953 T2269 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 956 T2270 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16d3d5da main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 957 T2269 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 957 T2269 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1998 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1999 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48529/_t/p",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48529__t%2Fp",
[junit4:junit4] 2> "core_node_name":"1"}
[junit4:junit4] 2> 1999 T2267 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1999 T2267 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 2002 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2959 T2269 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2959 T2269 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48529/_t/p collection:control_collection shard:shard1
[junit4:junit4] 2> 2960 T2269 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2964 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:delete cxid:0x82 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2964 T2269 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2965 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x83 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2966 T2269 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2966 T2269 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2966 T2269 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48529/_t/p/collection1/
[junit4:junit4] 2> 2966 T2269 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2967 T2269 oasc.SyncStrategy.syncToMe http://127.0.0.1:48529/_t/p/collection1/ has no replicas
[junit4:junit4] 2> 2967 T2269 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48529/_t/p/collection1/
[junit4:junit4] 2> 2967 T2269 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2969 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650003 type:create cxid:0x8e zxid:0x3a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3506 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3511 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3521 T2269 oasc.ZkController.register We are http://127.0.0.1:48529/_t/p/collection1/ and leader is http://127.0.0.1:48529/_t/p/collection1/
[junit4:junit4] 2> 3522 T2269 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48529/_t/p
[junit4:junit4] 2> 3522 T2269 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3522 T2269 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3522 T2269 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3523 T2269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3524 T2244 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3525 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3525 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3530 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3530 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1501981f
[junit4:junit4] 2> 3531 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3532 T2272 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3532 T2272 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 3532 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59333
[junit4:junit4] 2> 3532 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59333
[junit4:junit4] 2> 3533 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650004 with negotiated timeout 10000 for client /127.0.0.1:59333
[junit4:junit4] 2> 3533 T2272 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650004, negotiated timeout = 10000
[junit4:junit4] 2> 3533 T2273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1501981f name:ZooKeeperConnection Watcher:127.0.0.1:58928/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3533 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3534 T2244 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3536 T2244 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3604 T2244 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3606 T2244 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43653
[junit4:junit4] 2> 3607 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3607 T2244 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3607 T2244 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190
[junit4:junit4] 2> 3608 T2244 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/solr.xml
[junit4:junit4] 2> 3608 T2244 oasc.CoreContainer.<init> New CoreContainer 2032777681
[junit4:junit4] 2> 3608 T2244 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/'
[junit4:junit4] 2> 3609 T2244 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/'
[junit4:junit4] 2> 3626 T2244 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3627 T2244 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3627 T2244 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3627 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3628 T2244 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3628 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3628 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3628 T2244 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3629 T2244 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3629 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3634 T2244 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3643 T2244 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58928/solr
[junit4:junit4] 2> 3644 T2244 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3644 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@34dd3237
[junit4:junit4] 2> 3645 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3645 T2283 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3645 T2283 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 3645 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59334
[junit4:junit4] 2> 3646 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59334
[junit4:junit4] 2> 3646 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650005 with negotiated timeout 20000 for client /127.0.0.1:59334
[junit4:junit4] 2> 3646 T2283 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650005, negotiated timeout = 20000
[junit4:junit4] 2> 3647 T2284 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34dd3237 name:ZooKeeperConnection Watcher:127.0.0.1:58928 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3647 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3648 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650005
[junit4:junit4] 2> 3648 T2284 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3648 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59334 which had sessionid 0x13d80990c650005
[junit4:junit4] 2> 3648 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650005 closed
[junit4:junit4] 2> 3648 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3650 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@48fe6ff4
[junit4:junit4] 2> 3651 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3651 T2285 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3652 T2285 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 3652 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59335
[junit4:junit4] 2> 3652 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59335
[junit4:junit4] 2> 3653 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650006 with negotiated timeout 20000 for client /127.0.0.1:59335
[junit4:junit4] 2> 3653 T2285 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650006, negotiated timeout = 20000
[junit4:junit4] 2> 3653 T2286 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48fe6ff4 name:ZooKeeperConnection Watcher:127.0.0.1:58928/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3653 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3654 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:create cxid:0x1 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3655 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:create cxid:0x3 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3656 T2244 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4658 T2244 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43653__t%2Fp
[junit4:junit4] 2> 4659 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:delete cxid:0xd zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:43653__t%2Fp Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:43653__t%2Fp
[junit4:junit4] 2> 4659 T2244 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43653__t%2Fp
[junit4:junit4] 2> 4662 T2273 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4662 T2286 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4662 T2266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4662 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4666 T2287 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/collection1
[junit4:junit4] 2> 4666 T2287 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4667 T2287 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4667 T2287 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4668 T2287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/collection1/'
[junit4:junit4] 2> 4668 T2287 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/collection1/lib/README' to classloader
[junit4:junit4] 2> 4669 T2287 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4691 T2287 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 4718 T2287 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4719 T2287 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4722 T2287 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 5015 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 5016 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:48529/_t/p",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:48529__t%2Fp",
[junit4:junit4] 2> "core_node_name":"1"}
[junit4:junit4] 2> 5018 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5018 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5018 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5046 T2287 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5051 T2287 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5053 T2287 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5062 T2287 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5065 T2287 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5067 T2287 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5068 T2287 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5069 T2287 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5069 T2287 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5070 T2287 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5070 T2287 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5070 T2287 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5070 T2287 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363662150190/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1/
[junit4:junit4] 2> 5071 T2287 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3ef14b6a
[junit4:junit4] 2> 5071 T2287 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5071 T2287 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1 forceNew: false
[junit4:junit4] 2> 5071 T2287 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1/index/
[junit4:junit4] 2> 5072 T2287 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5072 T2287 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1/index forceNew: false
[junit4:junit4] 2> 5073 T2287 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49cf2361 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4a25941),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5073 T2287 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5075 T2287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5075 T2287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5076 T2287 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5076 T2287 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5076 T2287 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5076 T2287 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5077 T2287 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5077 T2287 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5077 T2287 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5079 T2287 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5081 T2287 oass.SolrIndexSearcher.<init> Opening Searcher@8877885 main
[junit4:junit4] 2> 5082 T2287 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty1/tlog
[junit4:junit4] 2> 5082 T2287 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5082 T2287 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5086 T2288 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8877885 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5087 T2287 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5087 T2287 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6524 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6525 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43653/_t/p",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43653__t%2Fp",
[junit4:junit4] 2> "core_node_name":"2"}
[junit4:junit4] 2> 6525 T2267 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 6525 T2267 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6532 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6532 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6532 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7089 T2287 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7089 T2287 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43653/_t/p collection:collection1 shard:shard1
[junit4:junit4] 2> 7090 T2287 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 7094 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:delete cxid:0x4a zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7094 T2287 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7095 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:create cxid:0x4b zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7096 T2287 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7096 T2287 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7096 T2287 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43653/_t/p/collection1/
[junit4:junit4] 2> 7096 T2287 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7096 T2287 oasc.SyncStrategy.syncToMe http://127.0.0.1:43653/_t/p/collection1/ has no replicas
[junit4:junit4] 2> 7097 T2287 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43653/_t/p/collection1/
[junit4:junit4] 2> 7097 T2287 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 7099 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650006 type:create cxid:0x56 zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8038 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8044 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8044 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8044 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8052 T2287 oasc.ZkController.register We are http://127.0.0.1:43653/_t/p/collection1/ and leader is http://127.0.0.1:43653/_t/p/collection1/
[junit4:junit4] 2> 8052 T2287 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43653/_t/p
[junit4:junit4] 2> 8052 T2287 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 8053 T2287 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 8053 T2287 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 8055 T2287 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8056 T2244 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 8056 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 8057 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8125 T2244 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 8126 T2244 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39818
[junit4:junit4] 2> 8128 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8128 T2244 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8129 T2244 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716
[junit4:junit4] 2> 8129 T2244 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/solr.xml
[junit4:junit4] 2> 8129 T2244 oasc.CoreContainer.<init> New CoreContainer 375249702
[junit4:junit4] 2> 8130 T2244 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/'
[junit4:junit4] 2> 8131 T2244 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/'
[junit4:junit4] 2> 8155 T2244 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8155 T2244 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8155 T2244 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8156 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8156 T2244 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8156 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8157 T2244 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8157 T2244 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8157 T2244 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8158 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8163 T2244 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8173 T2244 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:58928/solr
[junit4:junit4] 2> 8174 T2244 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8174 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1e45583c
[junit4:junit4] 2> 8175 T2299 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8176 T2299 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 8176 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59339
[junit4:junit4] 2> 8176 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8176 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59339
[junit4:junit4] 2> 8177 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650007 with negotiated timeout 20000 for client /127.0.0.1:59339
[junit4:junit4] 2> 8177 T2299 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650007, negotiated timeout = 20000
[junit4:junit4] 2> 8177 T2300 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e45583c name:ZooKeeperConnection Watcher:127.0.0.1:58928 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8178 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8179 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650007
[junit4:junit4] 2> 8180 T2300 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8180 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59339 which had sessionid 0x13d80990c650007
[junit4:junit4] 2> 8180 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650007 closed
[junit4:junit4] 2> 8180 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8183 T2244 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58928/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@be6e55f
[junit4:junit4] 2> 8183 T2244 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8183 T2301 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:58928. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8184 T2301 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58928, initiating session
[junit4:junit4] 2> 8184 T2246 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:59340
[junit4:junit4] 2> 8184 T2246 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:59340
[junit4:junit4] 2> 8185 T2248 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d80990c650008 with negotiated timeout 20000 for client /127.0.0.1:59340
[junit4:junit4] 2> 8185 T2301 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:58928, sessionid = 0x13d80990c650008, negotiated timeout = 20000
[junit4:junit4] 2> 8185 T2302 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@be6e55f name:ZooKeeperConnection Watcher:127.0.0.1:58928/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8186 T2244 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8186 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8187 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650008 type:create cxid:0x3 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8188 T2244 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9190 T2244 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39818__t%2Fp
[junit4:junit4] 2> 9192 T2249 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d80990c650008 type:delete cxid:0xd zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:39818__t%2Fp Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:39818__t%2Fp
[junit4:junit4] 2> 9192 T2244 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39818__t%2Fp
[junit4:junit4] 2> 9195 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9195 T2286 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9195 T2266 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9196 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9196 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9196 T2302 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9197 T2273 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9205 T2303 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/collection1
[junit4:junit4] 2> 9205 T2303 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9206 T2303 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9206 T2303 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9208 T2303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/collection1/'
[junit4:junit4] 2> 9209 T2303 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/collection1/lib/README' to classloader
[junit4:junit4] 2> 9209 T2303 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9245 T2303 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 9294 T2303 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9295 T2303 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9299 T2303 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9549 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9550 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:43653/_t/p",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:43653__t%2Fp",
[junit4:junit4] 2> "core_node_name":"2"}
[junit4:junit4] 2> 9554 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9554 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9554 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9554 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9692 T2303 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9699 T2303 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9702 T2303 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9714 T2303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9717 T2303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9720 T2303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9721 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9722 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9722 T2303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9723 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9723 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9723 T2303 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9724 T2303 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363662154716/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/
[junit4:junit4] 2> 9724 T2303 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3ef14b6a
[junit4:junit4] 2> 9724 T2303 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9725 T2303 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2 forceNew: false
[junit4:junit4] 2> 9725 T2303 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index/
[junit4:junit4] 2> 9725 T2303 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9725 T2303 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index forceNew: false
[junit4:junit4] 2> 9727 T2303 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@24bac9aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e2c582b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9727 T2303 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9729 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9729 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9730 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9730 T2303 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9731 T2303 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9731 T2303 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9732 T2303 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9732 T2303 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9733 T2303 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9736 T2303 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9738 T2303 oass.SolrIndexSearcher.<init> Opening Searcher@4b7594ee main
[junit4:junit4] 2> 9739 T2303 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/tlog
[junit4:junit4] 2> 9739 T2303 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9740 T2303 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9743 T2304 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b7594ee main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9745 T2303 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9745 T2303 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11057 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11058 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39818/_t/p",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39818__t%2Fp",
[junit4:junit4] 2> "core_node_name":"3"}
[junit4:junit4] 2> 11058 T2267 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 11058 T2267 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 11062 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11062 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11062 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11062 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11746 T2303 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11746 T2303 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39818/_t/p collection:collection1 shard:shard1
[junit4:junit4] 2> 11749 T2303 oasc.ZkController.register We are http://127.0.0.1:39818/_t/p/collection1/ and leader is http://127.0.0.1:43653/_t/p/collection1/
[junit4:junit4] 2> 11749 T2303 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39818/_t/p
[junit4:junit4] 2> 11749 T2303 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 11749 T2303 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C102 name=collection1 org.apache.solr.core.SolrCore@7ea731d8 url=http://127.0.0.1:39818/_t/p/collection1 node=127.0.0.1:39818__t%2Fp C102_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, state=down, base_url=http://127.0.0.1:39818/_t/p, node_name=127.0.0.1:39818__t%2Fp}
[junit4:junit4] 2> 11750 T2305 C102 P39818 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 11750 T2303 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11750 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 11751 T2305 C102 P39818 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 11751 T2305 C102 P39818 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11751 T2244 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 11752 T2305 C102 P39818 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11752 T2244 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 11752 T2244 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11755 T2244 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C103 name=collection1 org.apache.solr.core.SolrCore@57a0170c url=http://127.0.0.1:48529/_t/p/collection1 node=127.0.0.1:48529__t%2Fp C103_STATE=coll:control_collection core:collection1 props:{collection=control_collection, core=collection1, state=active, base_url=http://127.0.0.1:48529/_t/p, shard=shard1, node_name=127.0.0.1:48529__t%2Fp, leader=true}
[junit4:junit4] 2> 11766 T2262 C103 P48529 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@fbeed79 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5bb07663),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11766 T2262 C103 P48529 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11768 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10001 (1429903411424264192)]} 0 5
[junit4:junit4] 2> 11769 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[1 (1429903411429507072)]} 0 1
[junit4:junit4] 2> ASYNC NEW_CORE C104 name=collection1 org.apache.solr.core.SolrCore@6658239a url=http://127.0.0.1:43653/_t/p/collection1 node=127.0.0.1:43653__t%2Fp C104_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, state=active, base_url=http://127.0.0.1:43653/_t/p, shard=shard1, node_name=127.0.0.1:43653__t%2Fp, leader=true}
[junit4:junit4] 2> 11777 T2280 C104 P43653 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49cf2361 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4a25941),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11778 T2280 C104 P43653 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11780 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[1 (1429903411437895680)]} 0 4
[junit4:junit4] 2> 11780 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10001 (1429903411439992832)]} 0 3
[junit4:junit4] 2> 11850 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10001 (-1429903411515490304)]} 0 0
[junit4:junit4] 2> 11850 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[1 (-1429903411515490305)]} 0 0
[junit4:junit4] 2> 11853 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10001 (-1429903411518636032)]} 0 0
[junit4:junit4] 2> 11853 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[1 (-1429903411518636033)]} 0 0
[junit4:junit4] 2> 11856 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10002 (1429903411520733184)]} 0 1
[junit4:junit4] 2> 11856 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[2 (1429903411520733185)]} 0 1
[junit4:junit4] 2> 11859 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[2 (1429903411523878913)]} 0 0
[junit4:junit4] 2> 11858 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10002 (1429903411523878912)]} 0 0
[junit4:junit4] 2> 11944 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10003 (1429903411613007872)]} 0 1
[junit4:junit4] 2> 11944 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[3 (1429903411613007873)]} 0 1
[junit4:junit4] 2> 11946 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10003 (1429903411616153600)]} 0 0
[junit4:junit4] 2> 11947 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[3 (1429903411616153601)]} 0 1
[junit4:junit4] 2> 11979 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10004 (1429903411649708033)]} 0 1
[junit4:junit4] 2> 11979 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[4 (1429903411649708032)]} 0 1
[junit4:junit4] 2> 11981 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[4 (1429903411652853760)]} 0 0
[junit4:junit4] 2> 11981 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10004 (1429903411652853761)]} 0 0
[junit4:junit4] 2> 11999 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10005 (1429903411671728128)]} 0 0
[junit4:junit4] 2> 11999 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[5 (1429903411671728129)]} 0 0
[junit4:junit4] 2> 12002 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10005 (1429903411674873856)]} 0 1
[junit4:junit4] 2> 12003 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[5 (1429903411674873857)]} 0 1
[junit4:junit4] 2> 12088 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10005 (-1429903411765051393)]} 0 0
[junit4:junit4] 2> 12088 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[5 (-1429903411765051392)]} 0 0
[junit4:junit4] 2> 12090 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[5 (-1429903411767148544)]} 0 0
[junit4:junit4] 2> 12090 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10005 (-1429903411767148545)]} 0 0
[junit4:junit4] 2> 12093 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10006 (1429903411769245697)]} 0 1
[junit4:junit4] 2> 12093 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[6 (1429903411769245696)]} 0 1
[junit4:junit4] 2> 12095 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10006 (1429903411772391424)]} 0 0
[junit4:junit4] 2> 12095 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[6 (1429903411772391425)]} 0 0
[junit4:junit4] 2> 12124 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[7 (1429903411802800128)]} 0 0
[junit4:junit4] 2> 12124 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10007 (1429903411802800129)]} 0 0
[junit4:junit4] 2> 12127 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10007 (1429903411804897280)]} 0 1
[junit4:junit4] 2> 12127 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[7 (1429903411805945856)]} 0 1
[junit4:junit4] 2> 12203 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10008 (1429903411884589057)]} 0 1
[junit4:junit4] 2> 12203 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[8 (1429903411884589056)]} 0 1
[junit4:junit4] 2> 12207 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10008 (1429903411888783360)]} 0 1
[junit4:junit4] 2> 12207 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[8 (1429903411888783361)]} 0 1
[junit4:junit4] 2> 12238 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10009 (1429903411921289216)]} 0 1
[junit4:junit4] 2> 12238 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[9 (1429903411921289217)]} 0 1
[junit4:junit4] 2> 12241 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10009 (1429903411924434944)]} 0 1
[junit4:junit4] 2> 12241 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[9 (1429903411925483520)]} 0 0
[junit4:junit4] 2> 12284 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10006 (-1429903411970572288)]} 0 0
[junit4:junit4] 2> 12284 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[6 (-1429903411970572289)]} 0 0
[junit4:junit4] 2> 12287 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10006 (-1429903411973718016)]} 0 1
[junit4:junit4] 2> 12287 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[6 (-1429903411973718017)]} 0 0
[junit4:junit4] 2> 12290 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10010 (1429903411975815168)]} 0 1
[junit4:junit4] 2> 12290 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10 (1429903411976863744)]} 0 0
[junit4:junit4] 2> 12294 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10010 (1429903411980009472)]} 0 1
[junit4:junit4] 2> 12294 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10 (1429903411980009473)]} 0 1
[junit4:junit4] 2> 12387 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10007 (-1429903412078575616)]} 0 1
[junit4:junit4] 2> 12387 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[7 (-1429903412078575617)]} 0 0
[junit4:junit4] 2> 12390 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10007 (-1429903412081721344)]} 0 0
[junit4:junit4] 2> 12390 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[7 (-1429903412081721345)]} 0 0
[junit4:junit4] 2> 12393 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10011 (1429903412084867072)]} 0 1
[junit4:junit4] 2> 12394 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[11 (1429903412084867073)]} 0 1
[junit4:junit4] 2> 12397 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10011 (1429903412088012800)]} 0 1
[junit4:junit4] 2> 12397 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[11 (1429903412088012801)]} 0 1
[junit4:junit4] 2> 12480 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[8 (-1429903412176093185)]} 0 0
[junit4:junit4] 2> 12480 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10008 (-1429903412176093184)]} 0 1
[junit4:junit4] 2> 12482 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[8 (-1429903412178190336)]} 0 0
[junit4:junit4] 2> 12482 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10008 (-1429903412178190337)]} 0 0
[junit4:junit4] 2> 12484 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[12 (1429903412180287488)]} 0 0
[junit4:junit4] 2> 12484 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10012 (1429903412180287489)]} 0 0
[junit4:junit4] 2> 12486 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[12 (1429903412182384640)]} 0 0
[junit4:junit4] 2> 12487 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10012 (1429903412182384641)]} 0 1
[junit4:junit4] 2> 12492 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[13 (1429903412188676096)]} 0 1
[junit4:junit4] 2> 12492 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10013 (1429903412188676097)]} 0 0
[junit4:junit4] 2> 12494 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[13 (1429903412190773248)]} 0 0
[junit4:junit4] 2> 12494 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10013 (1429903412190773249)]} 0 0
[junit4:junit4] 2> 12525 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[10009 (-1429903412223279105)]} 0 1
[junit4:junit4] 2> 12525 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[9 (-1429903412223279104)]} 0 1
[junit4:junit4] 2> 12527 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[9 (-1429903412225376257)]} 0 0
[junit4:junit4] 2> 12527 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10009 (-1429903412225376256)]} 0 0
[junit4:junit4] 2> 12530 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10014 (1429903412227473408)]} 0 1
[junit4:junit4] 2> 12530 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[14 (1429903412227473409)]} 0 1
[junit4:junit4] 2> 12532 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10014 (1429903412230619136)]} 0 1
[junit4:junit4] 2> 12532 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[14 (1429903412230619137)]} 0 0
[junit4:junit4] 2> 12565 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12566 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39818/_t/p",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39818__t%2Fp",
[junit4:junit4] 2> "core_node_name":"3"}
[junit4:junit4] 2> 12569 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12569 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12569 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12569 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12596 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10015 (1429903412296679424)]} 0 1
[junit4:junit4] 2> 12596 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[15 (1429903412296679425)]} 0 1
[junit4:junit4] 2> C102_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, state=recovering, base_url=http://127.0.0.1:39818/_t/p, shard=shard1, node_name=127.0.0.1:39818__t%2Fp}
[junit4:junit4] 2> 12605 T2297 C102 P39818 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@24bac9aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e2c582b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12605 T2297 C102 P39818 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12615 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[15 (1429903412299825152)]} 0 12
[junit4:junit4] 2> 12615 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10015 (1429903412299825153)]} 0 12
[junit4:junit4] 2> 12616 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[15 (1429903412299825152)]} 0 18
[junit4:junit4] 2> 12616 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10015 (1429903412299825153)]} 0 18
[junit4:junit4] 2> 12686 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10 (-1429903412391051264)]} 0 1
[junit4:junit4] 2> 12686 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10010 (-1429903412391051265)]} 0 1
[junit4:junit4] 2> 12690 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10 (-1429903412394196992)]} 0 0
[junit4:junit4] 2> 12690 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10010 (-1429903412394196993)]} 0 0
[junit4:junit4] 2> 12691 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10 (-1429903412394196992)]} 0 3
[junit4:junit4] 2> 12691 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10010 (-1429903412394196993)]} 0 3
[junit4:junit4] 2> 12693 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[16 (1429903412399439872)]} 0 0
[junit4:junit4] 2> 12694 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10016 (1429903412399439873)]} 0 1
[junit4:junit4] 2> 12698 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[16 (1429903412401537024)]} 0 0
[junit4:junit4] 2> 12698 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10016 (1429903412401537025)]} 0 0
[junit4:junit4] 2> 12698 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[16 (1429903412401537024)]} 0 3
[junit4:junit4] 2> 12699 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10016 (1429903412401537025)]} 0 4
[junit4:junit4] 2> 12756 T2279 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&nodeName=127.0.0.1:39818__t%252Fp&action=PREPRECOVERY&core=collection1&onlyIfLeader=true&version=2&state=recovering&checkLive=true&wt=javabin} status=0 QTime=1001
[junit4:junit4] 2> 12774 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[17 (1429903412484374528)]} 0 0
[junit4:junit4] 2> 12774 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10017 (1429903412484374529)]} 0 0
[junit4:junit4] 2> 12780 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[17 (1429903412487520257)]} 0 1
[junit4:junit4] 2> 12780 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10017 (1429903412487520256)]} 0 1
[junit4:junit4] 2> 12781 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[17 (1429903412487520257)]} 0 4
[junit4:junit4] 2> 12781 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10017 (1429903412487520256)]} 0 4
[junit4:junit4] 2> 12796 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[18 (1429903412506394624)]} 0 1
[junit4:junit4] 2> 12796 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10018 (1429903412507443200)]} 0 0
[junit4:junit4] 2> 12801 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[18 (1429903412509540352)]} 0 1
[junit4:junit4] 2> 12802 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10018 (1429903412510588928)]} 0 1
[junit4:junit4] 2> 12802 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[18 (1429903412509540352)]} 0 4
[junit4:junit4] 2> 12802 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10018 (1429903412510588928)]} 0 4
[junit4:junit4] 2> 12881 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[11 (-1429903412596572160)]} 0 0
[junit4:junit4] 2> 12881 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10011 (-1429903412596572161)]} 0 0
[junit4:junit4] 2> 12886 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[11 (-1429903412599717888)]} 0 0
[junit4:junit4] 2> 12886 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10011 (-1429903412599717889)]} 0 0
[junit4:junit4] 2> 12887 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[11 (-1429903412599717888)]} 0 3
[junit4:junit4] 2> 12887 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10011 (-1429903412599717889)]} 0 3
[junit4:junit4] 2> 12890 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[19 (1429903412606009344)]} 0 0
[junit4:junit4] 2> 12891 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10019 (1429903412606009345)]} 0 1
[junit4:junit4] 2> 12896 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[19 (1429903412609155072)]} 0 0
[junit4:junit4] 2> 12897 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10019 (1429903412610203648)]} 0 1
[junit4:junit4] 2> 12897 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[19 (1429903412609155072)]} 0 4
[junit4:junit4] 2> 12898 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10019 (1429903412610203648)]} 0 5
[junit4:junit4] 2> 12986 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[12 (-1429903412706672640)]} 0 0
[junit4:junit4] 2> 12987 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10012 (-1429903412706672641)]} 0 1
[junit4:junit4] 2> 12991 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[12 (-1429903412709818368)]} 0 0
[junit4:junit4] 2> 12991 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10012 (-1429903412709818369)]} 0 0
[junit4:junit4] 2> 12991 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[12 (-1429903412709818368)]} 0 2
[junit4:junit4] 2> 12991 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10012 (-1429903412709818369)]} 0 2
[junit4:junit4] 2> 12994 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[20 (1429903412715061248)]} 0 0
[junit4:junit4] 2> 12995 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10020 (1429903412715061249)]} 0 1
[junit4:junit4] 2> 13000 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[20 (1429903412718206976)]} 0 1
[junit4:junit4] 2> 13000 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10020 (1429903412718206977)]} 0 0
[junit4:junit4] 2> 13000 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[20 (1429903412718206976)]} 0 3
[junit4:junit4] 2> 13001 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10020 (1429903412718206977)]} 0 4
[junit4:junit4] 2> 13080 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[21 (1429903412805238784)]} 0 0
[junit4:junit4] 2> 13081 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10021 (1429903412805238785)]} 0 1
[junit4:junit4] 2> 13085 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[21 (1429903412807335936)]} 0 1
[junit4:junit4] 2> 13085 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10021 (1429903412808384512)]} 0 1
[junit4:junit4] 2> 13085 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[21 (1429903412807335936)]} 0 3
[junit4:junit4] 2> 13085 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10021 (1429903412808384512)]} 0 3
[junit4:junit4] 2> 13156 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[22 (1429903412883881985)]} 0 1
[junit4:junit4] 2> 13156 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10022 (1429903412883881984)]} 0 1
[junit4:junit4] 2> 13162 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[22 (1429903412888076288)]} 0 1
[junit4:junit4] 2> 13162 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10022 (1429903412888076289)]} 0 1
[junit4:junit4] 2> 13163 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[22 (1429903412888076288)]} 0 5
[junit4:junit4] 2> 13163 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10022 (1429903412888076289)]} 0 4
[junit4:junit4] 2> 13200 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[23 (1429903412931067904)]} 0 0
[junit4:junit4] 2> 13201 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10023 (1429903412931067905)]} 0 1
[junit4:junit4] 2> 13206 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[23 (1429903412934213632)]} 0 0
[junit4:junit4] 2> 13207 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10023 (1429903412935262208)]} 0 1
[junit4:junit4] 2> 13207 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[23 (1429903412934213632)]} 0 4
[junit4:junit4] 2> 13208 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10023 (1429903412935262208)]} 0 5
[junit4:junit4] 2> 13278 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[13 (-1429903413012856832)]} 0 0
[junit4:junit4] 2> 13279 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[10013 (-1429903413013905408)]} 0 1
[junit4:junit4] 2> 13284 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[13 (-1429903413016002560)]} 0 1
[junit4:junit4] 2> 13284 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10013 (-1429903413016002561)]} 0 0
[junit4:junit4] 2> 13285 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[13 (-1429903413016002560)]} 0 4
[junit4:junit4] 2> 13285 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10013 (-1429903413016002561)]} 0 4
[junit4:junit4] 2> 13288 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[24 (1429903413023342592)]} 0 1
[junit4:junit4] 2> 13289 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10024 (1429903413023342593)]} 0 2
[junit4:junit4] 2> 13294 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[24 (1429903413026488320)]} 0 1
[junit4:junit4] 2> 13295 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10024 (1429903413026488321)]} 0 1
[junit4:junit4] 2> 13295 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[24 (1429903413026488320)]} 0 4
[junit4:junit4] 2> 13295 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10024 (1429903413026488321)]} 0 4
[junit4:junit4] 2> 13340 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[25 (1429903413077868544)]} 0 0
[junit4:junit4] 2> 13341 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10025 (1429903413077868545)]} 0 1
[junit4:junit4] 2> 13347 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[25 (1429903413081014272)]} 0 1
[junit4:junit4] 2> 13347 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10025 (1429903413082062848)]} 0 1
[junit4:junit4] 2> 13347 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[25 (1429903413081014272)]} 0 4
[junit4:junit4] 2> 13348 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10025 (1429903413082062848)]} 0 5
[junit4:junit4] 2> 13419 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[15 (-1429903413160706048)]} 0 0
[junit4:junit4] 2> 13420 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10015 (-1429903413161754624)]} 0 0
[junit4:junit4] 2> 13425 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[15 (-1429903413163851776)]} 0 1
[junit4:junit4] 2> 13425 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10015 (-1429903413164900352)]} 0 0
[junit4:junit4] 2> 13425 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[15 (-1429903413163851776)]} 0 3
[junit4:junit4] 2> 13426 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10015 (-1429903413164900352)]} 0 3
[junit4:junit4] 2> 13429 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[26 (1429903413170143232)]} 0 1
[junit4:junit4] 2> 13429 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10026 (1429903413171191808)]} 0 0
[junit4:junit4] 2> 13435 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[26 (1429903413174337536)]} 0 1
[junit4:junit4] 2> 13436 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10026 (1429903413174337537)]} 0 1
[junit4:junit4] 2> 13436 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[26 (1429903413174337536)]} 0 5
[junit4:junit4] 2> 13437 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10026 (1429903413174337537)]} 0 5
[junit4:junit4] 2> 13499 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[27 (1429903413243543552)]} 0 1
[junit4:junit4] 2> 13500 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10027 (1429903413244592128)]} 0 2
[junit4:junit4] 2> 13506 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[27 (1429903413247737856)]} 0 1
[junit4:junit4] 2> 13506 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10027 (1429903413248786432)]} 0 1
[junit4:junit4] 2> 13506 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[27 (1429903413247737856)]} 0 4
[junit4:junit4] 2> 13507 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10027 (1429903413248786432)]} 0 4
[junit4:junit4] 2> 13550 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[28 (1429903413297020928)]} 0 1
[junit4:junit4] 2> 13550 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10028 (1429903413298069504)]} 0 1
[junit4:junit4] 2> 13556 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[28 (1429903413301215232)]} 0 1
[junit4:junit4] 2> 13557 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10028 (1429903413301215233)]} 0 1
[junit4:junit4] 2> 13557 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[28 (1429903413301215232)]} 0 4
[junit4:junit4] 2> 13557 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10028 (1429903413301215233)]} 0 4
[junit4:junit4] 2> 13659 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[18 (-1429903413412364288)]} 0 0
[junit4:junit4] 2> 13659 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10018 (-1429903413412364289)]} 0 0
[junit4:junit4] 2> 13664 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[18 (-1429903413415510016)]} 0 0
[junit4:junit4] 2> 13664 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10018 (-1429903413415510017)]} 0 0
[junit4:junit4] 2> 13665 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[18 (-1429903413415510016)]} 0 3
[junit4:junit4] 2> 13665 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10018 (-1429903413415510017)]} 0 3
[junit4:junit4] 2> 13669 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[29 (1429903413421801472)]} 0 1
[junit4:junit4] 2> 13669 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10029 (1429903413421801473)]} 0 1
[junit4:junit4] 2> 13675 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[29 (1429903413424947200)]} 0 1
[junit4:junit4] 2> 13675 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10029 (1429903413425995776)]} 0 0
[junit4:junit4] 2> 13676 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[29 (1429903413424947200)]} 0 4
[junit4:junit4] 2> 13676 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10029 (1429903413425995776)]} 0 4
[junit4:junit4] 2> 13756 T2244 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 39818
[junit4:junit4] 2> 13757 T2244 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=375249702
[junit4:junit4] 2> 13768 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[19 (-1429903413525610496)]} 0 1
[junit4:junit4] 2> 13768 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10019 (-1429903413526659072)]} 0 0
[junit4:junit4] 2> 13773 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[19 (-1429903413528756224)]} 0 1
[junit4:junit4] 2> 13773 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10019 (-1429903413529804800)]} 0 0
[junit4:junit4] 2> 13774 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[19 (-1429903413528756224)]} 0 4
[junit4:junit4] 2> 13774 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10019 (-1429903413529804800)]} 0 3
[junit4:junit4] 2> 13777 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[30 (1429903413535047680)]} 0 1
[junit4:junit4] 2> 13778 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10030 (1429903413536096256)]} 0 2
[junit4:junit4] 2> 13783 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[30 (1429903413539241984)]} 0 1
[junit4:junit4] 2> 13784 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10030 (1429903413539241985)]} 0 1
[junit4:junit4] 2> 13784 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[30 (1429903413539241984)]} 0 4
[junit4:junit4] 2> 13785 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10030 (1429903413539241985)]} 0 5
[junit4:junit4] 2> 13815 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[21 (-1429903413575942144)]} 0 0
[junit4:junit4] 2> 13816 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10021 (-1429903413575942145)]} 0 1
[junit4:junit4] 2> 13820 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[21 (-1429903413579087872)]} 0 0
[junit4:junit4] 2> 13821 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[21 (-1429903413579087872)]} 0 3
[junit4:junit4] 2> 13821 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10021 (-1429903413580136448)]} 0 0
[junit4:junit4] 2> 13822 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10021 (-1429903413580136448)]} 0 4
[junit4:junit4] 2> 13824 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[31 (1429903413584330752)]} 0 1
[junit4:junit4] 2> 13825 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10031 (1429903413585379328)]} 0 1
[junit4:junit4] 2> 13830 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[31 (1429903413588525056)]} 0 1
[junit4:junit4] 2> 13831 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[31 (1429903413588525056)]} 0 4
[junit4:junit4] 2> 13831 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10031 (1429903413589573632)]} 0 1
[junit4:junit4] 2> 13832 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10031 (1429903413589573632)]} 0 4
[junit4:junit4] 2> 13925 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[32 (1429903413691285504)]} 0 0
[junit4:junit4] 2> 13926 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10032 (1429903413692334080)]} 0 0
[junit4:junit4] 2> 13931 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[32 (1429903413694431232)]} 0 1
[junit4:junit4] 2> 13932 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[32 (1429903413694431232)]} 0 4
[junit4:junit4] 2> 13933 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10032 (1429903413695479808)]} 0 1
[junit4:junit4] 2> 13934 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10032 (1429903413695479808)]} 0 5
[junit4:junit4] 2> 13975 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[33 (1429903413742665728)]} 0 1
[junit4:junit4] 2> 13976 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10033 (1429903413743714304)]} 0 1
[junit4:junit4] 2> 13981 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[33 (1429903413746860032)]} 0 1
[junit4:junit4] 2> 13982 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10033 (1429903413747908608)]} 0 1
[junit4:junit4] 2> 13982 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[33 (1429903413746860032)]} 0 5
[junit4:junit4] 2> 13983 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10033 (1429903413747908608)]} 0 4
[junit4:junit4] 2> 14000 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[22 (-1429903413769928704)]} 0 0
[junit4:junit4] 2> 14001 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10022 (-1429903413770977280)]} 0 1
[junit4:junit4] 2> 14006 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[22 (-1429903413774123008)]} 0 0
[junit4:junit4] 2> 14006 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10022 (-1429903413774123009)]} 0 0
[junit4:junit4] 2> 14007 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[22 (-1429903413774123008)]} 0 4
[junit4:junit4] 2> 14007 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10022 (-1429903413774123009)]} 0 3
[junit4:junit4] 2> 14011 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[34 (1429903413780414464)]} 0 1
[junit4:junit4] 2> 14011 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10034 (1429903413780414465)]} 0 1
[junit4:junit4] 2> 14017 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[34 (1429903413784608768)]} 0 1
[junit4:junit4] 2> 14017 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10034 (1429903413784608769)]} 0 0
[junit4:junit4] 2> 14017 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[34 (1429903413784608768)]} 0 3
[junit4:junit4] 2> 14018 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10034 (1429903413784608769)]} 0 4
[junit4:junit4] 2> 14073 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14074 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39818/_t/p",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39818__t%2Fp",
[junit4:junit4] 2> "core_node_name":"3"}
[junit4:junit4] 2> 14078 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14078 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14078 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14078 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14096 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[23 (-1429903413870592000)]} 0 0
[junit4:junit4] 2> 14096 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10023 (-1429903413870592001)]} 0 0
[junit4:junit4] 2> 14098 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10023 (-1429903413872689153)]} 0 0
[junit4:junit4] 2> 14098 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[23 (-1429903413872689152)]} 0 0
[junit4:junit4] 2> 14101 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10035 (1429903413874786304)]} 0 1
[junit4:junit4] 2> 14101 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[35 (1429903413874786305)]} 0 1
[junit4:junit4] 2> 14103 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10035 (1429903413877932032)]} 0 1
[junit4:junit4] 2> 14103 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[35 (1429903413877932033)]} 0 0
[junit4:junit4] 2> 14149 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10025 (-1429903413926166529)]} 0 0
[junit4:junit4] 2> 14149 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[25 (-1429903413926166528)]} 0 0
[junit4:junit4] 2> 14152 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10025 (-1429903413928263681)]} 0 1
[junit4:junit4] 2> 14152 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[25 (-1429903413928263680)]} 0 1
[junit4:junit4] 2> 14155 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[36 (1429903413931409408)]} 0 1
[junit4:junit4] 2> 14155 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10036 (1429903413931409409)]} 0 1
[junit4:junit4] 2> 14158 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[36 (1429903413935603712)]} 0 1
[junit4:junit4] 2> 14158 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10036 (1429903413935603713)]} 0 0
[junit4:junit4] 2> 14248 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[28 (-1429903414029975552)]} 0 1
[junit4:junit4] 2> 14248 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10028 (-1429903414029975553)]} 0 1
[junit4:junit4] 2> 14250 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[28 (-1429903414032072704)]} 0 0
[junit4:junit4] 2> 14250 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10028 (-1429903414032072705)]} 0 0
[junit4:junit4] 2> 14253 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[37 (1429903414035218432)]} 0 0
[junit4:junit4] 2> 14253 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10037 (1429903414035218433)]} 0 0
[junit4:junit4] 2> 14256 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[37 (1429903414037315584)]} 0 1
[junit4:junit4] 2> 14256 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10037 (1429903414038364160)]} 0 1
[junit4:junit4] 2> 14330 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[38 (1429903414115958784)]} 0 0
[junit4:junit4] 2> 14331 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10038 (1429903414115958785)]} 0 1
[junit4:junit4] 2> 14333 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10038 (1429903414119104512)]} 0 0
[junit4:junit4] 2> 14333 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[38 (1429903414119104513)]} 0 0
[junit4:junit4] 2> 14425 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[10030 (-1429903414215573505)]} 0 1
[junit4:junit4] 2> 14425 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[30 (-1429903414215573504)]} 0 1
[junit4:junit4] 2> 14428 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10030 (-1429903414218719232)]} 0 1
[junit4:junit4] 2> 14428 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[30 (-1429903414218719233)]} 0 0
[junit4:junit4] 2> 14430 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10039 (1429903414220816384)]} 0 0
[junit4:junit4] 2> 14431 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[39 (1429903414220816385)]} 0 1
[junit4:junit4] 2> 14433 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10039 (1429903414223962112)]} 0 0
[junit4:junit4] 2> 14433 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[39 (1429903414223962113)]} 0 0
[junit4:junit4] 2> 14457 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10040 (1429903414248079360)]} 0 1
[junit4:junit4] 2> 14457 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[40 (1429903414248079361)]} 0 1
[junit4:junit4] 2> 14459 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10040 (1429903414251225088)]} 0 0
[junit4:junit4] 2> 14459 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[40 (1429903414251225089)]} 0 0
[junit4:junit4] 2> 14508 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[41 (1429903414302605312)]} 0 0
[junit4:junit4] 2> 14509 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10041 (1429903414302605313)]} 0 1
[junit4:junit4] 2> 14512 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[41 (1429903414305751040)]} 0 1
[junit4:junit4] 2> 14512 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10041 (1429903414305751041)]} 0 1
[junit4:junit4] 2> 14537 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[10033 (-1429903414333014016)]} 0 0
[junit4:junit4] 2> 14537 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[33 (-1429903414333014017)]} 0 0
[junit4:junit4] 2> 14539 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10033 (-1429903414335111168)]} 0 0
[junit4:junit4] 2> 14539 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[33 (-1429903414335111169)]} 0 0
[junit4:junit4] 2> 14542 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10042 (1429903414337208320)]} 0 1
[junit4:junit4] 2> 14542 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[42 (1429903414337208321)]} 0 1
[junit4:junit4] 2> 14544 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10042 (1429903414340354048)]} 0 0
[junit4:junit4] 2> 14544 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[42 (1429903414340354049)]} 0 0
[junit4:junit4] 2> 14645 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10043 (1429903414446260224)]} 0 1
[junit4:junit4] 2> 14645 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[43 (1429903414446260225)]} 0 1
[junit4:junit4] 2> 14648 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10043 (1429903414449405952)]} 0 1
[junit4:junit4] 2> 14648 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[43 (1429903414449405953)]} 0 1
[junit4:junit4] 2> 14685 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10034 (-1429903414488203264)]} 0 1
[junit4:junit4] 2> 14685 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[34 (-1429903414488203265)]} 0 1
[junit4:junit4] 2> 14687 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10034 (-1429903414490300417)]} 0 0
[junit4:junit4] 2> 14687 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[34 (-1429903414490300416)]} 0 0
[junit4:junit4] 2> 14691 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10044 (1429903414493446144)]} 0 1
[junit4:junit4] 2> 14691 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[44 (1429903414493446145)]} 0 1
[junit4:junit4] 2> 14694 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10044 (1429903414496591872)]} 0 1
[junit4:junit4] 2> 14694 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[44 (1429903414497640448)]} 0 0
[junit4:junit4] 2> C102_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, state=down, base_url=http://127.0.0.1:39818/_t/p, shard=shard1, node_name=127.0.0.1:39818__t%2Fp}
[junit4:junit4] 2> 14757 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43653/_t/p/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 14757 T2305 C102 P39818 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39818/_t/p START replicas=[http://127.0.0.1:43653/_t/p/collection1/] nUpdates=100
[junit4:junit4] 2> 14757 T2305 C102 P39818 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 14758 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 14758 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 14758 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 14758 T2305 C102 P39818 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 14759 T2305 C102 P39818 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43653/_t/p/collection1/. core=collection1
[junit4:junit4] 2> 14759 T2305 C102 P39818 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14759 T2279 C104 P43653 REQ /get {distrib=false&getVersions=100&version=2&qt=/get&wt=javabin} status=0 QTime=0
[junit4:junit4] 2> 14762 T2280 C104 P43653 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14771 T2280 C104 P43653 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49cf2361 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4a25941),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49cf2361 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4a25941),segFN=segments_2,generation=2,filenames=[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1.fnm, _1_TestBloomFilteredLucene41Postings_0.pos, _1_MockRandom_0.doc, segments_2, _0.si, _1_Direct_0.tip, _1_Lucene41WithOrds_0.tii, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, _1_TestBloomFilteredLucene41Postings_0.tim, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0_MockRandom_0.doc, _0_Direct_0.tim, _0.nvd, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.doc, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 14772 T2280 C104 P43653 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1.fnm, _1_TestBloomFilteredLucene41Postings_0.pos, _1_MockRandom_0.doc, segments_2, _0.si, _1_Direct_0.tip, _1_Lucene41WithOrds_0.tii, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, _1_TestBloomFilteredLucene41Postings_0.tim, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0_MockRandom_0.doc, _0_Direct_0.tim, _0.nvd, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0.fnm, _0_TestBloomFilteredLucene41Postings_0.doc, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 14775 T2280 C104 P43653 oass.SolrIndexSearcher.<init> Opening Searcher@15e9374c realtime
[junit4:junit4] 2> 14775 T2280 C104 P43653 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 14775 T2280 C104 P43653 /update {openSearcher=false&commit_end_point=true&version=2&commit=true&waitSearcher=true&softCommit=false&wt=javabin} {commit=} 0 13
[junit4:junit4] 2> 14776 T2305 C102 P39818 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14776 T2305 C102 P39818 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 14780 T2279 C104 P43653 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14781 T2279 C104 P43653 REQ /replication {command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=4
[junit4:junit4] 2> 14782 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 14782 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 14782 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 14784 T2281 C104 P43653 REQ /replication {command=filelist&version=2&qt=/replication&wt=javabin&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14785 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 54
[junit4:junit4] 2> 14785 T2305 C102 P39818 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index.20130319050241439 forceNew: false
[junit4:junit4] 2> 14785 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384) fullCopy=true
[junit4:junit4] 2> 14786 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10035 (-1429903414594109440)]} 0 0
[junit4:junit4] 2> 14786 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[35 (-1429903414594109441)]} 0 0
[junit4:junit4] 2> 14788 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.tib&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14798 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[10035 (-1429903414596206592)]} 0 10
[junit4:junit4] 2> 14800 T2281 C104 P43653 REQ /replication {checksum=true&file=_0_Direct_0.tip&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14801 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10045 (1429903414609838080)]} 0 1
[junit4:junit4] 2> 14801 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[35 (-1429903414596206593)]} 0 13
[junit4:junit4] 2> 14804 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10045 (1429903414611935232)]} 0 1
[junit4:junit4] 2> 14804 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[45 (1429903414612983808)]} 0 0
[junit4:junit4] 2> 14804 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_TestBloomFilteredLucene41Postings_0.blm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14807 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[45 (1429903414615080960)]} 0 1
[junit4:junit4] 2> 14807 T2279 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.skp&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14811 T2280 C104 P43653 REQ /replication {checksum=true&file=_0.fdt&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14814 T2282 C104 P43653 REQ /replication {checksum=true&file=_1_Lucene41WithOrds_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14817 T2281 C104 P43653 REQ /replication {checksum=true&file=_1_TestBloomFilteredLucene41Postings_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14821 T2279 C104 P43653 REQ /replication {checksum=true&file=_1.fnm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14824 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_MockRandom_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14827 T2282 C104 P43653 REQ /replication {checksum=true&file=segments_2&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14831 T2281 C104 P43653 REQ /replication {checksum=true&file=_0.si&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14834 T2279 C104 P43653 REQ /replication {checksum=true&file=_1_Lucene41WithOrds_0.tii&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14875 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_Direct_0.tip&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14888 T2282 C104 P43653 REQ /replication {checksum=true&file=_1_MockRandom_0.tip&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14890 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10046 (1429903414702112768)]} 0 1
[junit4:junit4] 2> 14893 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[46 (1429903414705258496)]} 0 1
[junit4:junit4] 2> 14893 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10046 (1429903414705258496)]} 0 1
[junit4:junit4] 2> 14893 T2279 C104 P43653 REQ /replication {checksum=true&file=_1.si&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14896 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[46 (1429903414709452800)]} 0 1
[junit4:junit4] 2> 14897 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_Direct_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14900 T2281 C104 P43653 REQ /replication {checksum=true&file=_1_Direct_0.tim&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14904 T2279 C104 P43653 REQ /replication {checksum=true&file=_0_TestBloomFilteredLucene41Postings_0.tim&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14904 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10047 (1429903414717841408)]} 0 1
[junit4:junit4] 2> 14907 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[47 (1429903414720987136)]} 0 0
[junit4:junit4] 2> 14907 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10047 (1429903414720987136)]} 0 0
[junit4:junit4] 2> 14908 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_1.del&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14910 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[47 (1429903414724132864)]} 0 1
[junit4:junit4] 2> 14912 T2279 C104 P43653 REQ /replication {checksum=true&file=_0_TestBloomFilteredLucene41Postings_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14925 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14938 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_Lucene41WithOrds_0.tib&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14941 T2281 C104 P43653 REQ /replication {checksum=true&file=_0.nvm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14945 T2279 C104 P43653 REQ /replication {checksum=true&file=_1.nvd&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14948 T2282 C104 P43653 REQ /replication {checksum=true&file=_1_TestBloomFilteredLucene41Postings_0.tim&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14952 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_TestBloomFilteredLucene41Postings_0.tip&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14956 T2281 C104 P43653 REQ /replication {checksum=true&file=_1.fdt&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14959 T2279 C104 P43653 REQ /replication {checksum=true&file=_0.nvd&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14963 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_Direct_0.tim&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14980 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15007 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10036 (-1429903414825844736)]} 0 1
[junit4:junit4] 2> 15007 T2281 C104 P43653 REQ /replication {checksum=true&file=_1_MockRandom_0.sd&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15007 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[36 (-1429903414825844737)]} 0 0
[junit4:junit4] 2> 15009 T2282 C104 P43653 /update {version=2&wt=javabin} {delete=[36 (-1429903414827941888)]} 0 0
[junit4:junit4] 2> 15009 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10036 (-1429903414827941889)]} 0 0
[junit4:junit4] 2> 15011 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_MockRandom_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15012 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10048 (1429903414831087616)]} 0 0
[junit4:junit4] 2> 15013 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[48 (1429903414832136192)]} 0 1
[junit4:junit4] 2> 15014 T2281 C104 P43653 REQ /replication {checksum=true&file=_1_Direct_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15015 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10048 (1429903414834233344)]} 0 1
[junit4:junit4] 2> 15016 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[48 (1429903414834233345)]} 0 1
[junit4:junit4] 2> 15018 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_Lucene41WithOrds_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15022 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_TestBloomFilteredLucene41Postings_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15025 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[38 (-1429903414843670528)]} 0 1
[junit4:junit4] 2> 15025 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[10038 (-1429903414844719104)]} 0 1
[junit4:junit4] 2> 15025 T2281 C104 P43653 REQ /replication {checksum=true&file=_0.fnm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15027 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[38 (-1429903414846816257)]} 0 0
[junit4:junit4] 2> 15027 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10038 (-1429903414846816256)]} 0 0
[junit4:junit4] 2> 15029 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.frq&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15030 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[49 (1429903414848913408)]} 0 1
[junit4:junit4] 2> 15030 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10049 (1429903414849961984)]} 0 0
[junit4:junit4] 2> 15032 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[49 (1429903414852059136)]} 0 0
[junit4:junit4] 2> 15033 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_Lucene41WithOrds_0.tii&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15033 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10049 (1429903414852059137)]} 0 1
[junit4:junit4] 2> 15036 T2282 C104 P43653 REQ /replication {checksum=true&file=_1.nvm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15040 T2281 C104 P43653 REQ /replication {checksum=true&file=_0_Lucene41WithOrds_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15057 T2279 C104 P43653 REQ /replication {checksum=true&file=_1_TestBloomFilteredLucene41Postings_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15069 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_TestBloomFilteredLucene41Postings_0.tip&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15073 T2282 C104 P43653 REQ /replication {checksum=true&file=_1_MockRandom_0.tim&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15077 T2281 C104 P43653 REQ /replication {checksum=true&file=_0_Lucene41WithOrds_0.pos&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15081 T2279 C104 P43653 REQ /replication {checksum=true&file=_0_Direct_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15085 T2280 C104 P43653 REQ /replication {checksum=true&file=_1_Direct_0.doc&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15088 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[10045 (-1429903414909730817)]} 0 1
[junit4:junit4] 2> 15088 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[45 (-1429903414909730816)]} 0 1
[junit4:junit4] 2> 15089 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_1.del&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15090 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10045 (-1429903414912876544)]} 0 0
[junit4:junit4] 2> 15090 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[45 (-1429903414912876545)]} 0 0
[junit4:junit4] 2> 15093 T2281 C104 P43653 REQ /replication {checksum=true&file=_1_TestBloomFilteredLucene41Postings_0.blm&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15093 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10050 (1429903414916022272)]} 0 1
[junit4:junit4] 2> 15094 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[50 (1429903414917070848)]} 0 0
[junit4:junit4] 2> 15097 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10050 (1429903414919168000)]} 0 1
[junit4:junit4] 2> 15097 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[50 (1429903414919168001)]} 0 1
[junit4:junit4] 2> 15097 T2282 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.sd&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15101 T2281 C104 P43653 REQ /replication {checksum=true&file=_0.fdx&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15164 T2280 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.pyl&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15165 T2305 C102 P39818 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_0_MockRandom_0.pyl, size=0}
[junit4:junit4] 2> 15167 T2279 C104 P43653 REQ /replication {checksum=true&file=_0_MockRandom_0.tii&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15171 T2282 C104 P43653 REQ /replication {checksum=true&file=_1_Lucene41WithOrds_0.tib&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15174 T2281 C104 P43653 REQ /replication {checksum=true&file=_1.fdx&command=filecontent&qt=/replication&wt=filestream&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15176 T2305 C102 P39818 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 15177 T2305 C102 P39818 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130319050241439
[junit4:junit4] 2> 15178 T2305 C102 P39818 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 15178 T2305 C102 P39818 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 15178 T2305 C102 P39818 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4] 2> 15180 T2305 C102 P39818 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index.20130319050241439
[junit4:junit4] 2> 15181 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10051 (1429903415007248384)]} 0 1
[junit4:junit4] 2> 15181 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[51 (1429903415007248385)]} 0 1
[junit4:junit4] 2> 15181 T2305 C102 P39818 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384),segFN=segments_2,generation=2,filenames=[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, segments_2, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15182 T2305 C102 P39818 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, segments_2, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15182 T2305 C102 P39818 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 15184 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[51 (1429903415011442689)]} 0 1
[junit4:junit4] 2> 15185 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10051 (1429903415011442688)]} 0 2
[junit4:junit4] 2> 15201 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10050 (-1429903415028219905)]} 0 1
[junit4:junit4] 2> 15201 T2260 C103 P48529 /update {version=2&wt=javabin} {delete=[50 (-1429903415028219904)]} 0 1
[junit4:junit4] 2> 15203 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10050 (-1429903415031365633)]} 0 0
[junit4:junit4] 2> 15203 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[50 (-1429903415031365632)]} 0 0
[junit4:junit4] 2> 15205 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[52 (1429903415033462784)]} 0 0
[junit4:junit4] 2> 15206 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10052 (1429903415033462785)]} 0 1
[junit4:junit4] 2> 15208 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[52 (1429903415035559936)]} 0 1
[junit4:junit4] 2> 15208 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10052 (1429903415036608512)]} 0 1
[junit4:junit4] 2> 15235 T2305 C102 P39818 oass.SolrIndexSearcher.<init> Opening Searcher@2ef47fa3 main
[junit4:junit4] 2> 15236 T2304 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ef47fa3 main{StandardDirectoryReader(segments_2:5:nrt _0(4.3):C66/31 _1(4.3):C22/11)}
[junit4:junit4] 2> 15236 T2305 C102 P39818 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@24bac9aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e2c582b)
[junit4:junit4] 2> 15236 T2305 C102 P39818 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/index
[junit4:junit4] 2> 15237 T2305 C102 P39818 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@24bac9aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e2c582b)
[junit4:junit4] 2> 15237 T2305 C102 P39818 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 15237 T2313 C102 P39818 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363662146658/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=4539
[junit4:junit4] 2> 15237 T2313 C102 P39818 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 15238 T2313 C102 P39818 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384),segFN=segments_2,generation=2,filenames=[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, segments_2, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384),segFN=segments_3,generation=3,filenames=[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, segments_3, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15239 T2313 C102 P39818 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, segments_3, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15239 T2313 C102 P39818 oass.SolrIndexSearcher.<init> Opening Searcher@5cfe79f4 main
[junit4:junit4] 2> 15240 T2313 C102 P39818 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 15240 T2304 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5cfe79f4 main{StandardDirectoryReader(segments_2:5:nrt _0(4.3):C66/31 _1(4.3):C22/11)}
[junit4:junit4] 2> 15241 T2313 C102 P39818 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=4539}
[junit4:junit4] 2> 15241 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 15241 T2305 C102 P39818 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 15241 T2305 C102 P39818 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15242 T2305 C102 P39818 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 15243 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[53 (1429903415073308672)]} 0 0
[junit4:junit4] 2> 15243 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10053 (1429903415073308673)]} 0 0
[junit4:junit4] 2> 15246 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[53 (1429903415075405824)]} 0 1
[junit4:junit4] 2> 15246 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10053 (1429903415075405825)]} 0 1
[junit4:junit4] 2> 15299 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[10054 (1429903415132028929)]} 0 1
[junit4:junit4] 2> 15299 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[54 (1429903415132028928)]} 0 1
[junit4:junit4] 2> 15302 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10054 (1429903415134126080)]} 0 1
[junit4:junit4] 2> 15302 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[54 (1429903415134126081)]} 0 1
[junit4:junit4] 2> 15356 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10051 (-1429903415191797760)]} 0 0
[junit4:junit4] 2> 15357 T2259 C103 P48529 /update {version=2&wt=javabin} {delete=[51 (-1429903415191797761)]} 0 1
[junit4:junit4] 2> 15359 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10051 (-1429903415194943488)]} 0 0
[junit4:junit4] 2> 15359 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[51 (-1429903415194943489)]} 0 0
[junit4:junit4] 2> 15362 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[55 (1429903415197040640)]} 0 1
[junit4:junit4] 2> 15362 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10055 (1429903415197040641)]} 0 1
[junit4:junit4] 2> 15365 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[55 (1429903415200186368)]} 0 1
[junit4:junit4] 2> 15365 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10055 (1429903415200186369)]} 0 1
[junit4:junit4] 2> 15435 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[55 (-1429903415274635264)]} 0 0
[junit4:junit4] 2> 15435 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10055 (-1429903415274635265)]} 0 0
[junit4:junit4] 2> 15438 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[55 (-1429903415277780992)]} 0 1
[junit4:junit4] 2> 15438 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[10055 (-1429903415277780993)]} 0 0
[junit4:junit4] 2> 15440 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[56 (1429903415279878144)]} 0 1
[junit4:junit4] 2> 15440 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10056 (1429903415279878145)]} 0 0
[junit4:junit4] 2> 15442 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[56 (1429903415281975296)]} 0 0
[junit4:junit4] 2> 15442 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10056 (1429903415281975297)]} 0 0
[junit4:junit4] 2> 15451 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[56 (-1429903415291412480)]} 0 0
[junit4:junit4] 2> 15452 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[10056 (-1429903415291412481)]} 0 1
[junit4:junit4] 2> 15454 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[10056 (-1429903415294558208)]} 0 0
[junit4:junit4] 2> 15454 T2280 C104 P43653 /update {version=2&wt=javabin} {delete=[56 (-1429903415294558209)]} 0 0
[junit4:junit4] 2> 15456 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10057 (1429903415296655360)]} 0 0
[junit4:junit4] 2> 15456 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[57 (1429903415296655361)]} 0 0
[junit4:junit4] 2> 15458 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10057 (1429903415298752512)]} 0 0
[junit4:junit4] 2> 15458 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[57 (1429903415298752513)]} 0 0
[junit4:junit4] 2> 15510 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[58 (1429903415352229888)]} 0 1
[junit4:junit4] 2> 15510 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10058 (1429903415352229889)]} 0 1
[junit4:junit4] 2> 15513 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[10058 (1429903415356424193)]} 0 0
[junit4:junit4] 2> 15513 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[58 (1429903415356424192)]} 0 1
[junit4:junit4] 2> 15577 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[59 (1429903415422484480)]} 0 1
[junit4:junit4] 2> 15577 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10059 (1429903415422484481)]} 0 1
[junit4:junit4] 2> 15579 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[59 (1429903415425630209)]} 0 0
[junit4:junit4] 2> 15580 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10059 (1429903415425630208)]} 0 1
[junit4:junit4] 2> 15580 T2267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15581 T2267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:39818/_t/p",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "node_name":"127.0.0.1:39818__t%2Fp",
[junit4:junit4] 2> "core_node_name":"3"}
[junit4:junit4] 2> 15584 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15584 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15584 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15584 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15647 T2262 C103 P48529 /update {version=2&wt=javabin} {add=[60 (1429903415496933376)]} 0 0
[junit4:junit4] 2> 15647 T2261 C103 P48529 /update {version=2&wt=javabin} {add=[10060 (1429903415496933377)]} 0 0
[junit4:junit4] 2> C102_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, state=active, base_url=http://127.0.0.1:39818/_t/p, shard=shard1, node_name=127.0.0.1:39818__t%2Fp}
[junit4:junit4] 2> 15653 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10060 (1429903415500079105)]} 0 2
[junit4:junit4] 2> 15653 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[60 (1429903415500079104)]} 0 2
[junit4:junit4] 2> 15654 T2281 C104 P43653 /update {version=2&wt=javabin} {add=[10060 (1429903415500079105)]} 0 5
[junit4:junit4] 2> 15654 T2279 C104 P43653 /update {version=2&wt=javabin} {add=[60 (1429903415500079104)]} 0 5
[junit4:junit4] 2> 15676 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[61 (1429903415526293504)]} 0 1
[junit4:junit4] 2> 15676 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[10061 (1429903415526293505)]} 0 1
[junit4:junit4] 2> 15681 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[61 (1429903415529439232)]} 0 1
[junit4:junit4] 2> 15682 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10061 (1429903415529439233)]} 0 1
[junit4:junit4] 2> 15682 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[61 (1429903415529439232)]} 0 4
[junit4:junit4] 2> 15682 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[10061 (1429903415529439233)]} 0 4
[junit4:junit4] 2> 15725 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10059 (-1429903415578722304)]} 0 0
[junit4:junit4] 2> 15725 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[59 (-1429903415578722305)]} 0 0
[junit4:junit4] 2> 15730 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[59 (-1429903415581868033)]} 0 1
[junit4:junit4] 2> 15730 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10059 (-1429903415581868032)]} 0 0
[junit4:junit4] 2> 15730 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10059 (-1429903415581868032)]} 0 2
[junit4:junit4] 2> 15730 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[59 (-1429903415581868033)]} 0 2
[junit4:junit4] 2> 15733 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10062 (1429903415586062336)]} 0 1
[junit4:junit4] 2> 15733 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[62 (1429903415586062337)]} 0 1
[junit4:junit4] 2> 15737 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10062 (1429903415589208064)]} 0 0
[junit4:junit4] 2> 15738 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[62 (1429903415589208065)]} 0 1
[junit4:junit4] 2> 15738 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10062 (1429903415589208064)]} 0 3
[junit4:junit4] 2> 15738 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[62 (1429903415589208065)]} 0 3
[junit4:junit4] 2> 15749 T2261 C103 P48529 /update {version=2&wt=javabin} {delete=[10060 (-1429903415603888128)]} 0 0
[junit4:junit4] 2> 15750 T2262 C103 P48529 /update {version=2&wt=javabin} {delete=[60 (-1429903415604936704)]} 0 0
[junit4:junit4] 2> 15754 T2297 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[10060 (-1429903415607033856)]} 0 1
[junit4:junit4] 2> 15754 T2279 C104 P43653 /update {version=2&wt=javabin} {delete=[10060 (-1429903415607033856)]} 0 2
[junit4:junit4] 2> 15754 T2296 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {delete=[60 (-1429903415607033857)]} 0 0
[junit4:junit4] 2> 15755 T2281 C104 P43653 /update {version=2&wt=javabin} {delete=[60 (-1429903415607033857)]} 0 3
[junit4:junit4] 2> 15757 T2259 C103 P48529 /update {version=2&wt=javabin} {add=[10063 (1429903415611228160)]} 0 1
[junit4:junit4] 2> 15758 T2260 C103 P48529 /update {version=2&wt=javabin} {add=[63 (1429903415612276736)]} 0 1
[junit4:junit4] 2> 15759 T2244 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15760 T2244 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15764 T2244 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7ea731d8
[junit4:junit4] 2> 15765 T2298 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[10063 (1429903415617519616)]} 0 1
[junit4:junit4] 2> 15766 T2280 C104 P43653 /update {version=2&wt=javabin} {add=[10063 (1429903415617519616)]} 0 4
[junit4:junit4] 2> 15766 T2295 C102 P39818 /update {version=2&distrib.from=http://127.0.0.1:43653/_t/p/collection1/&update.distrib=FROMLEADER&wt=javabin} {add=[63 (1429903415617519617)]} 0 1
[junit4:junit4] 2> 15767 T2295 C102 P39818 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7ea731d8
[junit4:junit4] 2> 15770 T2295 C102 P39818 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: Failed to unregister info bean: fieldCache
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:178)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:116)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.close(SolrCore.java:970)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> Caused by: javax.management.InstanceNotFoundException: solr/collection1:type=fieldCache,id=org.apache.solr.search.SolrFieldCacheMBean
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4] 2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4] 2> at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:536)
[junit4:junit4] 2> at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:175)
[junit4:junit4] 2> ... 29 more
[junit4:junit4] 2>
[junit4:junit4] 2> 15771 T2295 C102 P39818 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=8,adds=8,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=48,cumulative_deletesById=22,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15773 T2295 C102 P39818 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 15773 T2244 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=8,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=48,cumulative_deletesById=22,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15774 T2295 C102 P39818 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 15774 T2295 C102 P39818 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 15780 T2295 C102 P39818 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384),segFN=segments_3,generation=3,filenames=[_0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _1_MockRandom_0.doc, _0.si, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _0.nvm, segments_3, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _1_TestBloomFilteredLucene41Postings_0.tip, _1.fdt, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _1_Direct_0.pos, _1_MockRandom_0.pos, _1_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _0_MockRandom_0.frq, _0_Lucene41WithOrds_0.tii, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _0_MockRandom_0.tii, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384),segFN=segments_4,generation=4,filenames=[_2.fnm, _0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _2_TestBloomFilteredLucene41Postings_0.tim, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _2_TestBloomFilteredLucene41Postings_0.blm, _2_MockRandom_0.pos, _2_Direct_0.tip, _2_Lucene41WithOrds_0.doc, _2_TestBloomFilteredLucene41Postings_0.pos, _1_MockRandom_0.doc, _0.si, _2_MockRandom_0.sd, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _2_1.del, _2.fdt, _0.nvm, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _2_TestBloomFilteredLucene41Postings_0.doc, _2_Direct_0.tim, _2_Lucene41WithOrds_0.tib, _2_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.tip, _2_Direct_0.doc, _1.fdt, _2_MockRandom_0.tip, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _2_Lucene41WithOrds_0.tii, _1_Direct_0.pos, _1_MockRandom_0.pos, _2.nvm, _1_Lucene41WithOrds_0.doc, segments_4, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _2.fdx, _0_MockRandom_0.frq, _2.nvd, _0_Lucene41WithOrds_0.tii, _2_TestBloomFilteredLucene41Postings_0.tip, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _2.si, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _2_Direct_0.pos, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _2_MockRandom_0.tim, _0_MockRandom_0.tii, _2_MockRandom_0.doc, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15781 T2295 C102 P39818 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_2.fnm, _0_MockRandom_0.tib, _0_Direct_0.tip, _0_TestBloomFilteredLucene41Postings_0.blm, _2_TestBloomFilteredLucene41Postings_0.tim, _0_MockRandom_0.skp, _0.fdt, _1_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.pos, _1.fnm, _2_TestBloomFilteredLucene41Postings_0.blm, _2_MockRandom_0.pos, _2_Direct_0.tip, _2_Lucene41WithOrds_0.doc, _2_TestBloomFilteredLucene41Postings_0.pos, _1_MockRandom_0.doc, _0.si, _2_MockRandom_0.sd, _1_Lucene41WithOrds_0.tii, _1_Direct_0.tip, _1_MockRandom_0.tip, _1.si, _0_Direct_0.pos, _1_Direct_0.tim, _0_TestBloomFilteredLucene41Postings_0.tim, _1_1.del, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Lucene41WithOrds_0.tib, _0_MockRandom_0.pos, _2_1.del, _2.fdt, _0.nvm, _1.nvd, _1_TestBloomFilteredLucene41Postings_0.tim, _2_TestBloomFilteredLucene41Postings_0.doc, _2_Direct_0.tim, _2_Lucene41WithOrds_0.tib, _2_Lucene41WithOrds_0.pos, _1_TestBloomFilteredLucene41Postings_0.tip, _2_Direct_0.doc, _1.fdt, _2_MockRandom_0.tip, _0.nvd, _0_Direct_0.tim, _0_MockRandom_0.doc, _1_MockRandom_0.sd, _2_Lucene41WithOrds_0.tii, _1_Direct_0.pos, _1_MockRandom_0.pos, _2.nvm, _1_Lucene41WithOrds_0.doc, segments_4, _0_TestBloomFilteredLucene41Postings_0.doc, _0.fnm, _2.fdx, _0_MockRandom_0.frq, _2.nvd, _0_Lucene41WithOrds_0.tii, _2_TestBloomFilteredLucene41Postings_0.tip, _1.nvm, _0_Lucene41WithOrds_0.doc, _0_TestBloomFilteredLucene41Postings_0.tip, _1_TestBloomFilteredLucene41Postings_0.doc, _1_MockRandom_0.tim, _0_Lucene41WithOrds_0.pos, _2.si, _1_Direct_0.doc, _0_Direct_0.doc, _0_1.del, _1_TestBloomFilteredLucene41Postings_0.blm, _2_Direct_0.pos, _0_MockRandom_0.sd, _0.fdx, _0_MockRandom_0.pyl, _2_MockRandom_0.tim, _0_MockRandom_0.tii, _2_MockRandom_0.doc, _1_Lucene41WithOrds_0.tib, _1.fdx]
[junit4:junit4] 2> 15783 T2295 C102 P39818 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 15783 T2244 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 15784 T2249 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d80990c650008
[junit4:junit4] 2> 15784 T2282 C104 P43653 /update {version=2&wt=javabin} {add=[63 (1429903415617519617)]} 0 22
[junit4:junit4] 2> 15785 T2302 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15785 T2246 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59340 which had sessionid 0x13d80990c650008
[junit4:junit4] 2> 15785 T2302 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 15785 T2244 oaz.ZooKeeper.close Session: 0x13d80990c650008 closed
[junit4:junit4] 2> 15786 T2266 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15785 T2286 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15786 T2266 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15785 T2302 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 15786 T2286 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15786 T2273 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15786 T2302 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 15786 T2273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[jun
[...truncated too long message...]
rmat(name=Lucene41WithOrds), rnd_b=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_l=PostingsFormat(name=Lucene41WithOrds), multiDefault=PostingsFormat(name=Lucene41WithOrds)}, docValues:{}, sim=DefaultSimilarity, locale=ru_RU, timezone=Egypt
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=2,free=110215248,total=222298112
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSort, TestDocSet, BinaryUpdateRequestHandlerTest, TestExtendedDismaxParser, QueryElevationComponentTest, SolrIndexSplitterTest, TestFaceting, TestSurroundQueryParser, TestSolrCoreProperties, SuggesterFSTTest, ExternalFileFieldSortTest, OverseerTest, ReturnFieldsTest, UnloadDistributedZkTest, TestReversedWildcardFilterFactory, TestFunctionQuery, TestSolrIndexConfig, CoreAdminHandlerTest, LeaderElectionTest, TestFastWriter, TestValueSourceCache, TermVectorComponentTest, TestGroupingSearch, ZkControllerTest, JsonLoaderTest, SoftAutoCommitTest, TestCollationKeyRangeQueries, AnalysisAfterCoreReloadTest, TestPropInjectDefaults, TestTrie, TestLFUCache, SliceStateTest, MoreLikeThisHandlerTest, PeerSyncTest, TestBadConfig, TestPhraseSuggestions, TestLRUCache, CSVRequestHandlerTest, CacheHeaderTest, PreAnalyzedFieldTest, DistributedTermsComponentTest, ScriptEngineTest, TestPseudoReturnFields, OverseerCollectionProcessorTest, LegacyHTMLStripCharFilterTest, TestFieldTypeResource, TestPerFieldSimilarity, DateFieldTest, SchemaVersionSpecificBehaviorTest, AutoCommitTest, SOLR749Test, PolyFieldTest, SpatialFilterTest, SyncSliceTest, SimpleFacetsTest, SystemInfoHandlerTest, RAMDirectoryFactoryTest, TestSweetSpotSimilarityFactory, ShowFileRequestHandlerTest, SliceStateUpdateTest, SolrInfoMBeanTest, FastVectorHighlighterTest, TestJoin, LukeRequestHandlerTest, DistributedSpellCheckComponentTest, PrimitiveFieldTypeTest, TestStressRecovery, TestRemoteStreaming, ChaosMonkeySafeLeaderTest, LeaderElectionIntegrationTest, RecoveryZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=A090C4EC1890DCEA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ru_RU -Dtests.timezone=Egypt -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2303, name=coreLoadExecutor-1075-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1569)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@507be456 lockFactory=org.apache.lucene.store.NativeFSLockFactory@57686642)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > ... 1 more
[junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=2305, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1569)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f56a750 lockFactory=org.apache.lucene.store.NativeFSLockFactory@570bf384)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] Completed on J0 in 37.73s, 1 test, 2 errors <<< FAILURES!
[...truncated 404 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 2 suite-level errors, 13 ignored (7 assumptions)
Total time: 38 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseG1GC
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b79) - Build #
4727 - Still Failing!
Posted by Robert Muir <rc...@gmail.com>.
Test bug: I committed a fix.
On Mon, Mar 18, 2013 at 9:41 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4727/
> Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC
>
> 1 tests failed.
> REGRESSION: org.apache.lucene.index.TestStoredFieldsFormat.testWriteReadMerge
>
> Error Message:
> sub-readers have different BytesRef.Comparators; cannot merge
>
> Stack Trace:
> java.lang.IllegalStateException: sub-readers have different BytesRef.Comparators; cannot merge
> at __randomizedtesting.SeedInfo.seed([BBFBD4F1C6ECC4D5:6D4BEE1A5D1D9FF7]:0)
> at org.apache.lucene.index.MultiTerms.<init>(MultiTerms.java:67)
> at org.apache.lucene.index.MultiFields.terms(MultiFields.java:234)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:69)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
> at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1847)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1148)
> at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1129)
> at org.apache.lucene.index.BaseStoredFieldsFormatTestCase.testWriteReadMerge(BaseStoredFieldsFormatTestCase.java:541)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:487)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:722)
>
>
>
>
> Build Log:
> [...truncated 1279 lines...]
> [junit4:junit4] Suite: org.apache.lucene.index.TestStoredFieldsFormat
> [junit4:junit4] IGNOR/A 0.00s J0 | TestStoredFieldsFormat.testBigDocuments
> [junit4:junit4] > Assumption #1: 'nightly' test group is disabled (@Nightly)
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestStoredFieldsFormat -Dtests.method=testWriteReadMerge -Dtests.seed=BBFBD4F1C6ECC4D5 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_CH -Dtests.timezone=Antarctica/McMurdo -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] ERROR 0.77s J0 | TestStoredFieldsFormat.testWriteReadMerge <<<
> [junit4:junit4] > Throwable #1: java.lang.IllegalStateException: sub-readers have different BytesRef.Comparators; cannot merge
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([BBFBD4F1C6ECC4D5:6D4BEE1A5D1D9FF7]:0)
> [junit4:junit4] > at org.apache.lucene.index.MultiTerms.<init>(MultiTerms.java:67)
> [junit4:junit4] > at org.apache.lucene.index.MultiFields.terms(MultiFields.java:234)
> [junit4:junit4] > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:69)
> [junit4:junit4] > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
> [junit4:junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
> [junit4:junit4] > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1847)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1148)
> [junit4:junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1129)
> [junit4:junit4] > at org.apache.lucene.index.BaseStoredFieldsFormatTestCase.testWriteReadMerge(BaseStoredFieldsFormatTestCase.java:541)
> [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:487)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=it_CH, timezone=Antarctica/McMurdo
> [junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=121774952,total=287899648
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestNGramPhraseQuery, TestMultiFields, TestDocumentsWriterStallControl, TestNRTThreads, TestIndexWriterConfig, TestBlockPostingsFormat3, TestSloppyPhraseQuery2, Nested1, Nested2, Nested3, TestDoubleBarrelLRUCache, TestCachingCollector, TestLazyProxSkipping, TestShardSearching, TestMockDirectoryWrapper, TestMultiValuedNumericRangeQuery, TestIntsRef, Test2BDocs, TestSearch, TestTotalHitCountCollector, TestSimilarityBase, TestSimilarityProvider, TestBagOfPositions, TestExplanations, TestTieredMergePolicy, TestNearSpansOrdered, TestTimeLimitingCollector, TestNRTManager, Nested1, TestLucene40PostingsFormat, TestSloppyPhraseQuery, TestPerFieldPostingsFormat2, TestRateLimiter, TestMultiDocValues, TestSegmentReader, TestWeakIdentityMap, TestPayloadsOnVectors, TestDateSort, TestIndexWriterReader, TestRegexpRandom, TestCollectionUtil, TestLevenshteinAutomata, TestRegexpRandom2, TestLucene40PostingsReader, TestFlushByRamOrCountsPolicy, TestSimilarity2, TestSort, TestDocument, TestNoDeletionPolicy, TestPrefixQuery, TestStressNRT, TestDateTools, TestWindowsMMap, TestCompressingStoredFieldsFormat, TestOmitPositions, TestSpanMultiTermQueryWrapper, TestWorstCaseTestBehavior, TestSimpleExplanations, TestPerSegmentDeletes, TestIndexWriterLockRelease, TestDoc, TestFieldMaskingSpanQuery, TestIndexWriter, TestSimpleAttributeImpl, TestConstantScoreQuery, TestSearcherManager, TestIndexWriterUnicode, TestIndexWriterCommit, TestToken, TestTopScoreDocCollector, TestDocIdSet, TestTermsEnum2, TestRollingUpdates, TestCodecHoldsOpenFiles, TestBagOfPostings, TestCharFilter, TestTermInfosReaderIndex, TestCopyBytes, MultiCollectorTest, TestForUtil, TestSpanSearchEquivalence, TestCharTermAttributeImpl, TestPositionIncrement, TestAutomatonQuery, TestAttributeSource, TestStressAdvance, TestSubScorerFreqs, TestMockAnalyzer, TestFieldCacheRewriteMethod, TestLucene3xPostingsFormat, TestDocValuesFormat, TestNumericRangeQuery32, TestLookaheadTokenFilter, TestAddIndexes, TestDeletionPolicy, TestBooleanOr, TestIndexWriterDelete, TestLucene40DocValuesFormat, TestIndexWriterMergePolicy, TestDuelingCodecs, TestSearchAfter, TestTopDocsMerge, TestCompoundFile, TestStressIndexing, TestIndexWriterForceMerge, TestArrayUtil, TestConsistentFieldNumbers, TestSegmentMerger, TestIndexWriterNRTIsCurrent, TestTransactions, TestCrash, TestSegmentTermDocs, TestBitVector, TestFieldsReader, TestCustomSearcherSort, TestPostingsFormat, TestDeterminizeLexicon, TestFieldCache, TestMixedCodecs, TestSpanExplanations, TestDocTermOrds, Nested, TestStressIndexing2, TestDeterminism, TestRamUsageEstimatorOnWildAnimals, TestCodecs, TestBytesRefHash, TestSpecialOperations, TestForceMergeForever, TestPayloadNearQuery, TestCrashCausesCorruptIndex, TestLiveFieldValues, TestForTooMuchCloning, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestTermVectors, TestTermVectorsReader, TestTopDocsCollector, TestSegmentTermEnum, TestBooleanQuery, TestSortRandom, TestAllFilesHaveCodecHeader, TestTermScorer, TestFileSwitchDirectory, TestIndexableField, TestMultiLevelSkipList, TestCheckIndex, TestFieldValueFilter, TestBooleanQueryVisitSubscorers, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestIndexFileDeleter, TestCompiledAutomaton, TestDocValuesWithThreads, Before3, Before3, TestParallelTermEnum, TestElevationComparator, TestFilteredSearch, TestBinaryTerms, TestConjunctions, Nested, Nested, TestSpanExplanationsOfNonMatches, TestSmallFloat, TestSetOnce, TestCachingTokenFilter, TestSpansAdvanced, TestDateFilter, TestSameTokenSamePosition, TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, TestRecyclingByteBlockAllocator, Nested, TestNoMergePolicy, TestNamedSPILoader, TestByteBlockPool, Nested1, TestFieldInfos, TestVersionComparator, TestVersion, TestDocBoost, Test2BPositions, TestIOUtils, TestNewestSegment, TestImpersonation, TestLucene3xTermVectorsFormat, TestSurrogates, TestStoredFieldsFormat]
> [junit4:junit4] Completed on J0 in 25.11s, 10 tests, 1 error, 1 skipped <<< FAILURES!
>
> [...truncated 10 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:50: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 356 suites, 2244 tests, 1 error, 45 ignored (33 assumptions)
>
> Total time: 10 minutes 25 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Description set: Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b79) - Build #
4727 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4727/
Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.lucene.index.TestStoredFieldsFormat.testWriteReadMerge
Error Message:
sub-readers have different BytesRef.Comparators; cannot merge
Stack Trace:
java.lang.IllegalStateException: sub-readers have different BytesRef.Comparators; cannot merge
at __randomizedtesting.SeedInfo.seed([BBFBD4F1C6ECC4D5:6D4BEE1A5D1D9FF7]:0)
at org.apache.lucene.index.MultiTerms.<init>(MultiTerms.java:67)
at org.apache.lucene.index.MultiFields.terms(MultiFields.java:234)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:69)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1847)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1148)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1129)
at org.apache.lucene.index.BaseStoredFieldsFormatTestCase.testWriteReadMerge(BaseStoredFieldsFormatTestCase.java:541)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:487)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 1279 lines...]
[junit4:junit4] Suite: org.apache.lucene.index.TestStoredFieldsFormat
[junit4:junit4] IGNOR/A 0.00s J0 | TestStoredFieldsFormat.testBigDocuments
[junit4:junit4] > Assumption #1: 'nightly' test group is disabled (@Nightly)
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestStoredFieldsFormat -Dtests.method=testWriteReadMerge -Dtests.seed=BBFBD4F1C6ECC4D5 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_CH -Dtests.timezone=Antarctica/McMurdo -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 0.77s J0 | TestStoredFieldsFormat.testWriteReadMerge <<<
[junit4:junit4] > Throwable #1: java.lang.IllegalStateException: sub-readers have different BytesRef.Comparators; cannot merge
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([BBFBD4F1C6ECC4D5:6D4BEE1A5D1D9FF7]:0)
[junit4:junit4] > at org.apache.lucene.index.MultiTerms.<init>(MultiTerms.java:67)
[junit4:junit4] > at org.apache.lucene.index.MultiFields.terms(MultiFields.java:234)
[junit4:junit4] > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:69)
[junit4:junit4] > at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
[junit4:junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3693)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3296)
[junit4:junit4] > at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1847)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1484)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1148)
[junit4:junit4] > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1129)
[junit4:junit4] > at org.apache.lucene.index.BaseStoredFieldsFormatTestCase.testWriteReadMerge(BaseStoredFieldsFormatTestCase.java:541)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] > at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=it_CH, timezone=Antarctica/McMurdo
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=121774952,total=287899648
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestNGramPhraseQuery, TestMultiFields, TestDocumentsWriterStallControl, TestNRTThreads, TestIndexWriterConfig, TestBlockPostingsFormat3, TestSloppyPhraseQuery2, Nested1, Nested2, Nested3, TestDoubleBarrelLRUCache, TestCachingCollector, TestLazyProxSkipping, TestShardSearching, TestMockDirectoryWrapper, TestMultiValuedNumericRangeQuery, TestIntsRef, Test2BDocs, TestSearch, TestTotalHitCountCollector, TestSimilarityBase, TestSimilarityProvider, TestBagOfPositions, TestExplanations, TestTieredMergePolicy, TestNearSpansOrdered, TestTimeLimitingCollector, TestNRTManager, Nested1, TestLucene40PostingsFormat, TestSloppyPhraseQuery, TestPerFieldPostingsFormat2, TestRateLimiter, TestMultiDocValues, TestSegmentReader, TestWeakIdentityMap, TestPayloadsOnVectors, TestDateSort, TestIndexWriterReader, TestRegexpRandom, TestCollectionUtil, TestLevenshteinAutomata, TestRegexpRandom2, TestLucene40PostingsReader, TestFlushByRamOrCountsPolicy, TestSimilarity2, TestSort, TestDocument, TestNoDeletionPolicy, TestPrefixQuery, TestStressNRT, TestDateTools, TestWindowsMMap, TestCompressingStoredFieldsFormat, TestOmitPositions, TestSpanMultiTermQueryWrapper, TestWorstCaseTestBehavior, TestSimpleExplanations, TestPerSegmentDeletes, TestIndexWriterLockRelease, TestDoc, TestFieldMaskingSpanQuery, TestIndexWriter, TestSimpleAttributeImpl, TestConstantScoreQuery, TestSearcherManager, TestIndexWriterUnicode, TestIndexWriterCommit, TestToken, TestTopScoreDocCollector, TestDocIdSet, TestTermsEnum2, TestRollingUpdates, TestCodecHoldsOpenFiles, TestBagOfPostings, TestCharFilter, TestTermInfosReaderIndex, TestCopyBytes, MultiCollectorTest, TestForUtil, TestSpanSearchEquivalence, TestCharTermAttributeImpl, TestPositionIncrement, TestAutomatonQuery, TestAttributeSource, TestStressAdvance, TestSubScorerFreqs, TestMockAnalyzer, TestFieldCacheRewriteMethod, TestLucene3xPostingsFormat, TestDocValuesFormat, TestNumericRangeQuery32, TestLookaheadTokenFilter, TestAddIndexes, TestDeletionPolicy, TestBooleanOr, TestIndexWriterDelete, TestLucene40DocValuesFormat, TestIndexWriterMergePolicy, TestDuelingCodecs, TestSearchAfter, TestTopDocsMerge, TestCompoundFile, TestStressIndexing, TestIndexWriterForceMerge, TestArrayUtil, TestConsistentFieldNumbers, TestSegmentMerger, TestIndexWriterNRTIsCurrent, TestTransactions, TestCrash, TestSegmentTermDocs, TestBitVector, TestFieldsReader, TestCustomSearcherSort, TestPostingsFormat, TestDeterminizeLexicon, TestFieldCache, TestMixedCodecs, TestSpanExplanations, TestDocTermOrds, Nested, TestStressIndexing2, TestDeterminism, TestRamUsageEstimatorOnWildAnimals, TestCodecs, TestBytesRefHash, TestSpecialOperations, TestForceMergeForever, TestPayloadNearQuery, TestCrashCausesCorruptIndex, TestLiveFieldValues, TestForTooMuchCloning, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestTermVectors, TestTermVectorsReader, TestTopDocsCollector, TestSegmentTermEnum, TestBooleanQuery, TestSortRandom, TestAllFilesHaveCodecHeader, TestTermScorer, TestFileSwitchDirectory, TestIndexableField, TestMultiLevelSkipList, TestCheckIndex, TestFieldValueFilter, TestBooleanQueryVisitSubscorers, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestIndexFileDeleter, TestCompiledAutomaton, TestDocValuesWithThreads, Before3, Before3, TestParallelTermEnum, TestElevationComparator, TestFilteredSearch, TestBinaryTerms, TestConjunctions, Nested, Nested, TestSpanExplanationsOfNonMatches, TestSmallFloat, TestSetOnce, TestCachingTokenFilter, TestSpansAdvanced, TestDateFilter, TestSameTokenSamePosition, TestAutomatonQueryUnicode, TestFieldCacheTermsFilter, TestRecyclingByteBlockAllocator, Nested, TestNoMergePolicy, TestNamedSPILoader, TestByteBlockPool, Nested1, TestFieldInfos, TestVersionComparator, TestVersion, TestDocBoost, Test2BPositions, TestIOUtils, TestNewestSegment, TestImpersonation, TestLucene3xTermVectorsFormat, TestSurrogates, TestStoredFieldsFormat]
[junit4:junit4] Completed on J0 in 25.11s, 10 tests, 1 error, 1 skipped <<< FAILURES!
[...truncated 10 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:50: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 356 suites, 2244 tests, 1 error, 45 ignored (33 assumptions)
Total time: 10 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b79) - Build #
4726 - Still Failing!
Posted by Robert Muir <rm...@apache.org>.
This is doing a lot of lookups by term (from previous readers) in its
setNextReader to go back to ord, i'm not really sure its the most
efficient way.
Also it doesn't help the test gets simpletext codec.
But from my perspective, DocTermsOrds should not have such a slow
seekExact(), but implement both lookupTerm() and termsEnum(). the
latter is trivial and makes the test finish in 4 minutes. I'll look at
the seekCeil logic and see if I can address lookupTerm too.
Index: lucene/core/src/java/org/apache/lucene/index/DocTermOrds.java
===================================================================
--- lucene/core/src/java/org/apache/lucene/index/DocTermOrds.java (revision
1458071)
+++ lucene/core/src/java/org/apache/lucene/index/DocTermOrds.java (working copy)
@@ -880,5 +880,12 @@
public long getValueCount() {
return numTerms();
}
+
+ // TODO: implement lookupTerm based on logic in seekCeil()
+
+ @Override
+ public TermsEnum termsEnum() {
+ return te;
+ }
}
}
On Mon, Mar 18, 2013 at 8:45 PM, Robert Muir <rm...@apache.org> wrote:
> I'll dig. it seems to emit heartbeats with jdk7 too.
>
> On Mon, Mar 18, 2013 at 8:39 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4726/
>> Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
>>
>> 2 tests failed.
>> FAILED: junit.framework.TestSuite.org.apache.lucene.search.grouping.GroupFacetCollectorTest
>>
>> Error Message:
>> Suite timeout exceeded (>= 7200000 msec).
>>
>> Stack Trace:
>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>> at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>>
>>
>> REGRESSION: org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom
>>
>> Error Message:
>> Test abandoned because suite timeout was reached.
>>
>> Stack Trace:
>> java.lang.Exception: Test abandoned because suite timeout was reached.
>> at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 7101 lines...]
>> [junit4:junit4] Suite: org.apache.lucene.search.grouping.GroupFacetCollectorTest
>> [junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>> [junit4:junit4] 2> Avvertenza: Suite execution timed out: org.apache.lucene.search.grouping.GroupFacetCollectorTest
>> [junit4:junit4] 2> ==== jstack at approximately timeout time ====
>> [junit4:junit4] 2> "TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440]" ID=10 RUNNABLE
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:494)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
>> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> "SUITE-GroupFacetCollectorTest-seed#[66116EFB07D97440]" ID=9 RUNNABLE
>> [junit4:junit4] 2> at sun.management.ThreadImpl.dumpThreads0(Native Method)
>> [junit4:junit4] 2> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:655)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$800(ThreadLeakControl.java:60)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:396)
>> [junit4:junit4] 2> - locked java.lang.Object@45d66c
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:556)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:79)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:492)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> "Signal Dispatcher" ID=5 RUNNABLE
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3aae4a
>> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
>> [junit4:junit4] 2> - waiting on java.lang.ref.ReferenceQueue$Lock@3aae4a
>> [junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>> [junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>> [junit4:junit4] 2> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@152f172
>> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
>> [junit4:junit4] 2> - waiting on java.lang.ref.Reference$Lock@152f172
>> [junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
>> [junit4:junit4] 2> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:142)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
>> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
>> [junit4:junit4] 2> - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
>> [junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1261)
>> [junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1335)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:502)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:399)
>> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:180)
>> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:275)
>> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> ^^==============================================
>> [junit4:junit4] 2>
>> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.method=testRandom -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
>> [junit4:junit4] ERROR 7199s J1 | GroupFacetCollectorTest.testRandom <<<
>> [junit4:junit4] > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
>> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>> [junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>> [junit4:junit4] 2> Avvertenza: Will linger awaiting termination of 1 leaked thread(s).
>> [junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>> [junit4:junit4] 2> Grave: 1 thread leaked from SUITE scope at org.apache.lucene.search.grouping.GroupFacetCollectorTest:
>> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.newPair(PairOutputs.java:74)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:134)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:135)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
>> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
>> [junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>> [junit4:junit4] 2> Informazioni: Starting to interrupt leaked threads:
>> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
>> [junit4:junit4] 2> mar 18, 2013 9:37:03 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>> [junit4:junit4] 2> Grave: There are still zombie threads that couldn't be terminated:
>> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.valid(PairOutputs.java:93)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:133)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
>> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
>> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
>> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
>> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
>> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
>> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
>> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
>> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
>> [junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {group=PostingsFormat(name=SimpleText), hotel=Lucene41(blocksize=128), facet=PostingsFormat(name=SimpleText), content=PostingsFormat(name=NestedPulsing), x=Lucene41(blocksize=128), airport=PostingsFormat(name=NestedPulsing)}, docValues:{facet_dv=DocValuesFormat(name=SimpleText), group_dv=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=it_IT, timezone=America/Jujuy
>> [junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=2,free=12003920,total=25985024
>> [junit4:junit4] 2> NOTE: All tests run in this JVM: [GroupFacetCollectorTest]
>> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
>> [junit4:junit4] ERROR 0.00s J1 | GroupFacetCollectorTest (suite) <<<
>> [junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>> [junit4:junit4] Completed on J1 in 7223.36s, 2 tests, 2 errors <<< FAILURES!
>>
>> [...truncated 14 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:548: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1757: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 6 suites, 11 tests, 1 suite-level error, 1 error, 1 ignored
>>
>> Total time: 151 minutes 31 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Recording test results
>> Description set: Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b79) - Build #
4726 - Still Failing!
Posted by Robert Muir <rm...@apache.org>.
I'll dig. it seems to emit heartbeats with jdk7 too.
On Mon, Mar 18, 2013 at 8:39 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4726/
> Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
>
> 2 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.lucene.search.grouping.GroupFacetCollectorTest
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>
>
> REGRESSION: org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
> at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
>
>
>
>
> Build Log:
> [...truncated 7101 lines...]
> [junit4:junit4] Suite: org.apache.lucene.search.grouping.GroupFacetCollectorTest
> [junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
> [junit4:junit4] 2> Avvertenza: Suite execution timed out: org.apache.lucene.search.grouping.GroupFacetCollectorTest
> [junit4:junit4] 2> ==== jstack at approximately timeout time ====
> [junit4:junit4] 2> "TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440]" ID=10 RUNNABLE
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:494)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2>
> [junit4:junit4] 2> "SUITE-GroupFacetCollectorTest-seed#[66116EFB07D97440]" ID=9 RUNNABLE
> [junit4:junit4] 2> at sun.management.ThreadImpl.dumpThreads0(Native Method)
> [junit4:junit4] 2> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:655)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$800(ThreadLeakControl.java:60)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:396)
> [junit4:junit4] 2> - locked java.lang.Object@45d66c
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:556)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:79)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:492)
> [junit4:junit4] 2>
> [junit4:junit4] 2> "Signal Dispatcher" ID=5 RUNNABLE
> [junit4:junit4] 2>
> [junit4:junit4] 2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3aae4a
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> - waiting on java.lang.ref.ReferenceQueue$Lock@3aae4a
> [junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
> [junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
> [junit4:junit4] 2> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
> [junit4:junit4] 2>
> [junit4:junit4] 2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@152f172
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> - waiting on java.lang.ref.Reference$Lock@152f172
> [junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
> [junit4:junit4] 2> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:142)
> [junit4:junit4] 2>
> [junit4:junit4] 2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
> [junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1261)
> [junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1335)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:502)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:399)
> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:180)
> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:275)
> [junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
> [junit4:junit4] 2>
> [junit4:junit4] 2> ^^==============================================
> [junit4:junit4] 2>
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.method=testRandom -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] ERROR 7199s J1 | GroupFacetCollectorTest.testRandom <<<
> [junit4:junit4] > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
> [junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> [junit4:junit4] 2> Avvertenza: Will linger awaiting termination of 1 leaked thread(s).
> [junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> [junit4:junit4] 2> Grave: 1 thread leaked from SUITE scope at org.apache.lucene.search.grouping.GroupFacetCollectorTest:
> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.newPair(PairOutputs.java:74)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:134)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:135)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
> [junit4:junit4] 2> Informazioni: Starting to interrupt leaked threads:
> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
> [junit4:junit4] 2> mar 18, 2013 9:37:03 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
> [junit4:junit4] 2> Grave: There are still zombie threads that couldn't be terminated:
> [junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.valid(PairOutputs.java:93)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:133)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
> [junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
> [junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
> [junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
> [junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
> [junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
> [junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> [junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {group=PostingsFormat(name=SimpleText), hotel=Lucene41(blocksize=128), facet=PostingsFormat(name=SimpleText), content=PostingsFormat(name=NestedPulsing), x=Lucene41(blocksize=128), airport=PostingsFormat(name=NestedPulsing)}, docValues:{facet_dv=DocValuesFormat(name=SimpleText), group_dv=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=it_IT, timezone=America/Jujuy
> [junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=2,free=12003920,total=25985024
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [GroupFacetCollectorTest]
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] ERROR 0.00s J1 | GroupFacetCollectorTest (suite) <<<
> [junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
> [junit4:junit4] Completed on J1 in 7223.36s, 2 tests, 2 errors <<< FAILURES!
>
> [...truncated 14 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:548: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1757: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 6 suites, 11 tests, 1 suite-level error, 1 error, 1 ignored
>
> Total time: 151 minutes 31 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Description set: Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b79) - Build #
4726 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4726/
Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.lucene.search.grouping.GroupFacetCollectorTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
REGRESSION: org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
Build Log:
[...truncated 7101 lines...]
[junit4:junit4] Suite: org.apache.lucene.search.grouping.GroupFacetCollectorTest
[junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
[junit4:junit4] 2> Avvertenza: Suite execution timed out: org.apache.lucene.search.grouping.GroupFacetCollectorTest
[junit4:junit4] 2> ==== jstack at approximately timeout time ====
[junit4:junit4] 2> "TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440]" ID=10 RUNNABLE
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:494)
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
[junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
[junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> "SUITE-GroupFacetCollectorTest-seed#[66116EFB07D97440]" ID=9 RUNNABLE
[junit4:junit4] 2> at sun.management.ThreadImpl.dumpThreads0(Native Method)
[junit4:junit4] 2> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:655)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$800(ThreadLeakControl.java:60)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:396)
[junit4:junit4] 2> - locked java.lang.Object@45d66c
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:556)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:79)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:492)
[junit4:junit4] 2>
[junit4:junit4] 2> "Signal Dispatcher" ID=5 RUNNABLE
[junit4:junit4] 2>
[junit4:junit4] 2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3aae4a
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> - waiting on java.lang.ref.ReferenceQueue$Lock@3aae4a
[junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
[junit4:junit4] 2> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
[junit4:junit4] 2> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
[junit4:junit4] 2>
[junit4:junit4] 2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@152f172
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> - waiting on java.lang.ref.Reference$Lock@152f172
[junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4:junit4] 2> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:142)
[junit4:junit4] 2>
[junit4:junit4] 2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@421c14
[junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1261)
[junit4:junit4] 2> at java.lang.Thread.join(Thread.java:1335)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:502)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:399)
[junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:180)
[junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:275)
[junit4:junit4] 2> at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
[junit4:junit4] 2>
[junit4:junit4] 2> ^^==============================================
[junit4:junit4] 2>
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.method=testRandom -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 7199s J1 | GroupFacetCollectorTest.testRandom <<<
[junit4:junit4] > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
[junit4:junit4] 2> mar 18, 2013 9:36:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4:junit4] 2> Avvertenza: Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4:junit4] 2> Grave: 1 thread leaked from SUITE scope at org.apache.lucene.search.grouping.GroupFacetCollectorTest:
[junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.newPair(PairOutputs.java:74)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:134)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:135)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
[junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
[junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> mar 18, 2013 9:37:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4:junit4] 2> Informazioni: Starting to interrupt leaked threads:
[junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
[junit4:junit4] 2> mar 18, 2013 9:37:03 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4:junit4] 2> Grave: There are still zombie threads that couldn't be terminated:
[junit4:junit4] 2> 1) Thread[id=10, name=TEST-GroupFacetCollectorTest.testRandom-seed#[66116EFB07D97440], state=RUNNABLE, group=TGRP-GroupFacetCollectorTest]
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.valid(PairOutputs.java:93)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:133)
[junit4:junit4] 2> at org.apache.lucene.util.fst.PairOutputs.add(PairOutputs.java:31)
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.pushFirst(FSTEnum.java:488)
[junit4:junit4] 2> at org.apache.lucene.util.fst.FSTEnum.doNext(FSTEnum.java:111)
[junit4:junit4] 2> at org.apache.lucene.util.fst.BytesRefFSTEnum.next(BytesRefFSTEnum.java:57)
[junit4:junit4] 2> at org.apache.lucene.codecs.simpletext.SimpleTextFieldsReader$SimpleTextTermsEnum.next(SimpleTextFieldsReader.java:156)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$OrdWrappedTermsEnum.seekExact(DocTermOrds.java:736)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds.lookupTerm(DocTermOrds.java:761)
[junit4:junit4] 2> at org.apache.lucene.index.DocTermOrds$Iterator.lookupOrd(DocTermOrds.java:870)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValues.lookupTerm(SortedSetDocValues.java:106)
[junit4:junit4] 2> at org.apache.lucene.index.SortedSetDocValuesTermsEnum.seekExact(SortedSetDocValuesTermsEnum.java:65)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.setNextReader(TermGroupFacetCollector.java:305)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:602)
[junit4:junit4] 2> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:309)
[junit4:junit4] 2> at org.apache.lucene.search.grouping.GroupFacetCollectorTest.testRandom(GroupFacetCollectorTest.java:379)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4] 2> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4] 2> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4] 2> at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4] 2> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4] 2> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4] 2> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {group=PostingsFormat(name=SimpleText), hotel=Lucene41(blocksize=128), facet=PostingsFormat(name=SimpleText), content=PostingsFormat(name=NestedPulsing), x=Lucene41(blocksize=128), airport=PostingsFormat(name=NestedPulsing)}, docValues:{facet_dv=DocValuesFormat(name=SimpleText), group_dv=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=it_IT, timezone=America/Jujuy
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=2,free=12003920,total=25985024
[junit4:junit4] 2> NOTE: All tests run in this JVM: [GroupFacetCollectorTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=GroupFacetCollectorTest -Dtests.seed=66116EFB07D97440 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=it_IT -Dtests.timezone=America/Jujuy -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR 0.00s J1 | GroupFacetCollectorTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([66116EFB07D97440]:0)
[junit4:junit4] Completed on J1 in 7223.36s, 2 tests, 2 errors <<< FAILURES!
[...truncated 14 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:548: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1757: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 6 suites, 11 tests, 1 suite-level error, 1 error, 1 ignored
Total time: 151 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.8.0-ea-b79 -client -XX:+UseConcMarkSweepGC
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_15) - Build # 4725
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4725/
Java: 64bit/jdk1.7.0_15 -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=871, name=coreLoadExecutor-448-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=871, name=coreLoadExecutor-448-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1567)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:980)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@65a23497 lockFactory=org.apache.lucene.store.NativeFSLockFactory@73034b6b; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375))
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=873, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=873, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1567)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:980)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375))
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9259 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 0 T812 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 4 T812 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363636664322
[junit4:junit4] 2> 4 T812 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 5 T813 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 5 T813 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T813 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 6 T813 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 6 T813 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 9 T813 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363636664322/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 105 T812 oasc.ZkTestServer.run start zk server on port:45835
[junit4:junit4] 2> 105 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@589dae25
[junit4:junit4] 2> 106 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 109 T818 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 110 T818 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 110 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52409
[junit4:junit4] 2> 110 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52409
[junit4:junit4] 2> 111 T816 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 119 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080000 with negotiated timeout 10000 for client /127.0.0.1:52409
[junit4:junit4] 2> 119 T818 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080000, negotiated timeout = 10000
[junit4:junit4] 2> 121 T819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@589dae25 name:ZooKeeperConnection Watcher:127.0.0.1:45835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 122 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 122 T812 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 126 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080000
[junit4:junit4] 2> 128 T819 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 128 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080000 closed
[junit4:junit4] 2> 130 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52409 which had sessionid 0x13d7f1438080000
[junit4:junit4] 2> 131 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@2e473824
[junit4:junit4] 2> 132 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 132 T820 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 133 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52410
[junit4:junit4] 2> 133 T820 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 133 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52410
[junit4:junit4] 2> 139 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080001 with negotiated timeout 10000 for client /127.0.0.1:52410
[junit4:junit4] 2> 139 T820 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080001, negotiated timeout = 10000
[junit4:junit4] 2> 139 T821 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e473824 name:ZooKeeperConnection Watcher:127.0.0.1:45835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 139 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 140 T812 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 143 T812 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 146 T812 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 149 T812 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 159 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 159 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 178 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 179 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 198 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 199 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 201 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 201 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 210 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 210 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 218 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 219 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 222 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 222 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 226 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 226 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 228 T812 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 228 T812 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 230 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080001
[junit4:junit4] 2> 230 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080001 closed
[junit4:junit4] 2> 231 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52410 which had sessionid 0x13d7f1438080001
[junit4:junit4] 2> 233 T821 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 288 T812 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 292 T812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44555
[junit4:junit4] 2> 292 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 293 T812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 293 T812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549
[junit4:junit4] 2> 293 T812 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/solr.xml
[junit4:junit4] 2> 293 T812 oasc.CoreContainer.<init> New CoreContainer 1787946593
[junit4:junit4] 2> 294 T812 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/'
[junit4:junit4] 2> 294 T812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/'
[junit4:junit4] 2> 312 T812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 312 T812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 312 T812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 313 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 313 T812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 313 T812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 314 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 314 T812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 314 T812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 315 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 322 T812 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 330 T812 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45835/solr
[junit4:junit4] 2> 331 T812 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 331 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@60303e2d
[junit4:junit4] 2> 332 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 332 T831 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 333 T831 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 333 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52411
[junit4:junit4] 2> 333 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52411
[junit4:junit4] 2> 339 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080002 with negotiated timeout 20000 for client /127.0.0.1:52411
[junit4:junit4] 2> 339 T831 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080002, negotiated timeout = 20000
[junit4:junit4] 2> 340 T832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60303e2d name:ZooKeeperConnection Watcher:127.0.0.1:45835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 340 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 341 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080002
[junit4:junit4] 2> 348 T832 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 348 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080002 closed
[junit4:junit4] 2> 348 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52411 which had sessionid 0x13d7f1438080002
[junit4:junit4] 2> 349 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 352 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@28596d13
[junit4:junit4] 2> 353 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 353 T833 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 353 T833 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 353 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52412
[junit4:junit4] 2> 354 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52412
[junit4:junit4] 2> 369 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080003 with negotiated timeout 20000 for client /127.0.0.1:52412
[junit4:junit4] 2> 369 T833 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080003, negotiated timeout = 20000
[junit4:junit4] 2> 369 T834 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28596d13 name:ZooKeeperConnection Watcher:127.0.0.1:45835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 370 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 379 T812 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 381 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x7 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 382 T812 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 384 T812 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 399 T812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44555_
[junit4:junit4] 2> 400 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:delete cxid:0x12 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:44555_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:44555_
[junit4:junit4] 2> 418 T812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44555_
[junit4:junit4] 2> 431 T812 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 438 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:delete cxid:0x20 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 440 T812 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 442 T812 oasc.Overseer.start Overseer (id=89367292433399811-127.0.0.1:44555_-n_0000000000) starting
[junit4:junit4] 2> 443 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x25 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 444 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x26 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 445 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x28 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 446 T812 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 448 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x2e zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 449 T836 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 449 T812 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 451 T812 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 452 T812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 453 T835 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 457 T837 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/collection1
[junit4:junit4] 2> 457 T837 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 458 T837 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 458 T837 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 459 T837 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/collection1/'
[junit4:junit4] 2> 459 T837 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/collection1/lib/README' to classloader
[junit4:junit4] 2> 459 T837 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 485 T837 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 510 T837 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 511 T837 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 514 T837 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 803 T837 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 808 T837 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 810 T837 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 821 T837 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 824 T837 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 827 T837 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 828 T837 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 828 T837 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 828 T837 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 829 T837 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 829 T837 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 830 T837 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 830 T837 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363636664549/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data/
[junit4:junit4] 2> 830 T837 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f57e182
[junit4:junit4] 2> 830 T837 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 831 T837 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data forceNew: false
[junit4:junit4] 2> 831 T837 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data/index/
[junit4:junit4] 2> 831 T837 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 832 T837 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data/index forceNew: false
[junit4:junit4] 2> 833 T837 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40865d5e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2793a9f0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 833 T837 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 835 T837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 835 T837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 835 T837 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 836 T837 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 836 T837 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 836 T837 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 837 T837 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 837 T837 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 837 T837 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 840 T837 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 842 T837 oass.SolrIndexSearcher.<init> Opening Searcher@3b2a90b4 main
[junit4:junit4] 2> 842 T837 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/control/data/tlog
[junit4:junit4] 2> 842 T837 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 843 T837 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 845 T838 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b2a90b4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 846 T837 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 846 T837 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 1957 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 1958 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44555_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44555"}
[junit4:junit4] 2> 1958 T835 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 1958 T835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 1961 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2848 T837 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2849 T837 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44555 collection:control_collection shard:shard1
[junit4:junit4] 2> 2850 T837 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2860 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:delete cxid:0x82 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2861 T837 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2861 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x83 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2863 T837 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2863 T837 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2863 T837 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44555/collection1/
[junit4:junit4] 2> 2864 T837 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2864 T837 oasc.SyncStrategy.syncToMe http://127.0.0.1:44555/collection1/ has no replicas
[junit4:junit4] 2> 2864 T837 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44555/collection1/
[junit4:junit4] 2> 2864 T837 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2867 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080003 type:create cxid:0x8e zxid:0x3a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3471 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3562 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 3570 T837 oasc.ZkController.register We are http://127.0.0.1:44555/collection1/ and leader is http://127.0.0.1:44555/collection1/
[junit4:junit4] 2> 3570 T837 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44555
[junit4:junit4] 2> 3570 T837 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 3571 T837 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 3571 T837 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 3572 T837 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 3573 T812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 3574 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 3574 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 3579 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 3580 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@40dffac8
[junit4:junit4] 2> 3581 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3582 T840 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3582 T840 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 3582 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52414
[junit4:junit4] 2> 3583 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52414
[junit4:junit4] 2> 3583 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080004 with negotiated timeout 10000 for client /127.0.0.1:52414
[junit4:junit4] 2> 3583 T840 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080004, negotiated timeout = 10000
[junit4:junit4] 2> 3584 T841 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40dffac8 name:ZooKeeperConnection Watcher:127.0.0.1:45835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3584 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3585 T812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 3587 T812 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 3666 T812 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 3669 T812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49479
[junit4:junit4] 2> 3670 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 3670 T812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 3671 T812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905
[junit4:junit4] 2> 3671 T812 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/solr.xml
[junit4:junit4] 2> 3671 T812 oasc.CoreContainer.<init> New CoreContainer 2072373846
[junit4:junit4] 2> 3672 T812 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/'
[junit4:junit4] 2> 3672 T812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/'
[junit4:junit4] 2> 3695 T812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 3696 T812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 3696 T812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 3696 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 3697 T812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 3697 T812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 3697 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 3697 T812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 3698 T812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 3698 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 3704 T812 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 3712 T812 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45835/solr
[junit4:junit4] 2> 3713 T812 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 3713 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4d874cf
[junit4:junit4] 2> 3714 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3714 T851 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3715 T851 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 3715 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52415
[junit4:junit4] 2> 3715 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52415
[junit4:junit4] 2> 3716 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080005 with negotiated timeout 20000 for client /127.0.0.1:52415
[junit4:junit4] 2> 3716 T851 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080005, negotiated timeout = 20000
[junit4:junit4] 2> 3717 T852 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d874cf name:ZooKeeperConnection Watcher:127.0.0.1:45835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3717 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3718 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080005
[junit4:junit4] 2> 3719 T852 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 3719 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080005 closed
[junit4:junit4] 2> 3719 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52415 which had sessionid 0x13d7f1438080005
[junit4:junit4] 2> 3719 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 3722 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4c079cef
[junit4:junit4] 2> 3723 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 3723 T853 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 3724 T853 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 3724 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52416
[junit4:junit4] 2> 3724 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52416
[junit4:junit4] 2> 3725 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080006 with negotiated timeout 20000 for client /127.0.0.1:52416
[junit4:junit4] 2> 3725 T853 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080006, negotiated timeout = 20000
[junit4:junit4] 2> 3725 T854 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c079cef name:ZooKeeperConnection Watcher:127.0.0.1:45835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 3726 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 3726 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:create cxid:0x1 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3727 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:create cxid:0x3 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3728 T812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4730 T812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49479_
[junit4:junit4] 2> 4731 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:delete cxid:0xd zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49479_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49479_
[junit4:junit4] 2> 4732 T812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49479_
[junit4:junit4] 2> 4734 T854 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4734 T841 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4734 T834 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 4734 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 4738 T855 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/collection1
[junit4:junit4] 2> 4738 T855 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 4739 T855 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 4739 T855 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 4740 T855 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/collection1/'
[junit4:junit4] 2> 4740 T855 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/collection1/lib/README' to classloader
[junit4:junit4] 2> 4740 T855 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 4763 T855 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 4801 T855 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 4802 T855 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 4805 T855 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 5084 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 5084 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44555__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44555_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44555"}
[junit4:junit4] 2> 5087 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5087 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5087 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5114 T855 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 5120 T855 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 5122 T855 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 5133 T855 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5136 T855 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 5139 T855 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5140 T855 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5142 T855 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5143 T855 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 5144 T855 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 5144 T855 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 5144 T855 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5145 T855 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363636667905/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1/
[junit4:junit4] 2> 5145 T855 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f57e182
[junit4:junit4] 2> 5146 T855 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 5146 T855 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1 forceNew: false
[junit4:junit4] 2> 5147 T855 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1/index/
[junit4:junit4] 2> 5147 T855 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 5147 T855 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1/index forceNew: false
[junit4:junit4] 2> 5149 T855 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@37d4ca92 lockFactory=org.apache.lucene.store.NativeFSLockFactory@407834e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 5150 T855 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 5152 T855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 5153 T855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 5153 T855 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 5154 T855 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 5155 T855 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 5155 T855 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 5155 T855 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 5156 T855 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 5156 T855 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 5159 T855 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 5162 T855 oass.SolrIndexSearcher.<init> Opening Searcher@704d769d main
[junit4:junit4] 2> 5163 T855 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty1/tlog
[junit4:junit4] 2> 5164 T855 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 5164 T855 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 5170 T856 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@704d769d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 5172 T855 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 5173 T855 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 6591 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 6592 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49479_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49479"}
[junit4:junit4] 2> 6592 T835 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 6592 T835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 6595 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6595 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6595 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7174 T855 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 7175 T855 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49479 collection:collection1 shard:shard1
[junit4:junit4] 2> 7176 T855 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 7181 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:delete cxid:0x4a zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 7182 T855 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 7182 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:create cxid:0x4b zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 7184 T855 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 7184 T855 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 7184 T855 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49479/collection1/
[junit4:junit4] 2> 7184 T855 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 7185 T855 oasc.SyncStrategy.syncToMe http://127.0.0.1:49479/collection1/ has no replicas
[junit4:junit4] 2> 7185 T855 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49479/collection1/
[junit4:junit4] 2> 7185 T855 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 7188 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080006 type:create cxid:0x56 zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8098 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8107 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8108 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8108 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8141 T855 oasc.ZkController.register We are http://127.0.0.1:49479/collection1/ and leader is http://127.0.0.1:49479/collection1/
[junit4:junit4] 2> 8141 T855 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49479
[junit4:junit4] 2> 8141 T855 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 8141 T855 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 8142 T855 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 8143 T855 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 8143 T812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 8144 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 8144 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 8251 T812 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 8261 T812 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49750
[junit4:junit4] 2> 8262 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 8262 T812 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 8263 T812 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468
[junit4:junit4] 2> 8263 T812 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/solr.xml
[junit4:junit4] 2> 8264 T812 oasc.CoreContainer.<init> New CoreContainer 1126178006
[junit4:junit4] 2> 8264 T812 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/'
[junit4:junit4] 2> 8265 T812 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/'
[junit4:junit4] 2> 8311 T812 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 8312 T812 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 8312 T812 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 8313 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 8314 T812 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 8314 T812 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 8314 T812 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 8317 T812 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 8317 T812 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 8318 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 8328 T812 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 8341 T812 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45835/solr
[junit4:junit4] 2> 8342 T812 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 8342 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@59bde227
[junit4:junit4] 2> 8344 T867 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8348 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8350 T867 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 8350 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52421
[junit4:junit4] 2> 8350 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52421
[junit4:junit4] 2> 8351 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080007 with negotiated timeout 20000 for client /127.0.0.1:52421
[junit4:junit4] 2> 8351 T867 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080007, negotiated timeout = 20000
[junit4:junit4] 2> 8352 T868 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59bde227 name:ZooKeeperConnection Watcher:127.0.0.1:45835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8352 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8353 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080007
[junit4:junit4] 2> 8354 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52421 which had sessionid 0x13d7f1438080007
[junit4:junit4] 2> 8354 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080007 closed
[junit4:junit4] 2> 8356 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 8361 T812 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45835/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@67beb94
[junit4:junit4] 2> 8362 T869 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45835. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4] 2> 8363 T869 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45835, initiating session
[junit4:junit4] 2> 8363 T812 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 8354 T868 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 8363 T814 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:52422
[junit4:junit4] 2> 8366 T814 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:52422
[junit4:junit4] 2> 8367 T816 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7f1438080008 with negotiated timeout 20000 for client /127.0.0.1:52422
[junit4:junit4] 2> 8367 T869 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45835, sessionid = 0x13d7f1438080008, negotiated timeout = 20000
[junit4:junit4] 2> 8368 T870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67beb94 name:ZooKeeperConnection Watcher:127.0.0.1:45835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 8368 T812 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 8369 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8371 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080008 type:create cxid:0x3 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8373 T812 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 9376 T812 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49750_
[junit4:junit4] 2> 9377 T817 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7f1438080008 type:delete cxid:0xd zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49750_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49750_
[junit4:junit4] 2> 9379 T812 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49750_
[junit4:junit4] 2> 9380 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9381 T834 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9381 T841 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9381 T870 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9381 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9381 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9383 T854 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 9386 T871 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/collection1
[junit4:junit4] 2> 9386 T871 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 9386 T871 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 9386 T871 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 9387 T871 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/collection1/'
[junit4:junit4] 2> 9388 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/collection1/lib/README' to classloader
[junit4:junit4] 2> 9388 T871 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 9412 T871 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4] 2> 9442 T871 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 9443 T871 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 9448 T871 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 9617 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9618 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49479__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49479_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49479"}
[junit4:junit4] 2> 9620 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9621 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9621 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9621 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 9787 T871 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 9793 T871 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 9795 T871 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 9806 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9809 T871 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 9812 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9813 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9814 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9814 T871 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 9817 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 9817 T871 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 9818 T871 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9818 T871 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363636672468/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/
[junit4:junit4] 2> 9818 T871 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@f57e182
[junit4:junit4] 2> 9819 T871 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 9819 T871 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2 forceNew: false
[junit4:junit4] 2> 9819 T871 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index/
[junit4:junit4] 2> 9819 T871 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 9820 T871 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index forceNew: false
[junit4:junit4] 2> 9822 T871 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4fc325a0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b57358d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 9822 T871 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 9824 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 9825 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 9825 T871 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 9826 T871 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 9826 T871 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 9826 T871 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 9827 T871 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 9827 T871 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 9828 T871 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 9832 T871 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 9835 T871 oass.SolrIndexSearcher.<init> Opening Searcher@6ce35d7c main
[junit4:junit4] 2> 9835 T871 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/tlog
[junit4:junit4] 2> 9836 T871 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 9836 T871 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 9840 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ce35d7c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 9842 T871 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 9842 T871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11124 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11125 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49750_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49750"}
[junit4:junit4] 2> 11125 T835 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 11125 T835 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 11138 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11138 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11138 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11138 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 11843 T871 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 11843 T871 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49750 collection:collection1 shard:shard1
[junit4:junit4] 2> 11846 T871 oasc.ZkController.register We are http://127.0.0.1:49750/collection1/ and leader is http://127.0.0.1:49479/collection1/
[junit4:junit4] 2> 11847 T871 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49750
[junit4:junit4] 2> 11847 T871 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 11847 T871 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@7ce9fc0a url=http://127.0.0.1:49750/collection1 node=127.0.0.1:49750_ C28_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49750_, base_url=http://127.0.0.1:49750}
[junit4:junit4] 2> 11847 T873 C28 P49750 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 11848 T871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 11848 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 11849 T873 C28 P49750 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 11849 T873 C28 P49750 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 11849 T812 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 11850 T812 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 11850 T812 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11850 T873 C28 P49750 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 11854 T812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@618574b6 url=http://127.0.0.1:44555/collection1 node=127.0.0.1:44555_ C29_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:44555_, base_url=http://127.0.0.1:44555, leader=true}
[junit4:junit4] 2> 11864 T830 C29 P44555 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@40865d5e lockFactory=org.apache.lucene.store.NativeFSLockFactory@2793a9f0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11865 T830 C29 P44555 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11869 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10001 (1429876691364216832)]} 0 6
[junit4:junit4] 2> 11869 T830 C29 P44555 /update {wt=javabin&version=2} {add=[1 (1429876691362119680)]} 0 7
[junit4:junit4] 2> ASYNC NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@77568a38 url=http://127.0.0.1:49479/collection1 node=127.0.0.1:49479_ C30_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49479_, base_url=http://127.0.0.1:49479, leader=true}
[junit4:junit4] 2> 11882 T848 C30 P49479 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@37d4ca92 lockFactory=org.apache.lucene.store.NativeFSLockFactory@407834e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 11883 T848 C30 P49479 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 11885 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10001 (1429876691380994048)]} 0 6
[junit4:junit4] 2> 11886 T849 C30 P49479 /update {wt=javabin&version=2} {add=[1 (1429876691382042624)]} 0 5
[junit4:junit4] 2> 11991 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10002 (1429876691495288832)]} 0 2
[junit4:junit4] 2> 11991 T829 C29 P44555 /update {wt=javabin&version=2} {add=[2 (1429876691495288833)]} 0 2
[junit4:junit4] 2> 11995 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10002 (1429876691500531712)]} 0 1
[junit4:junit4] 2> 11996 T847 C30 P49479 /update {wt=javabin&version=2} {add=[2 (1429876691500531713)]} 0 2
[junit4:junit4] 2> 12054 T830 C29 P44555 /update {wt=javabin&version=2} {add=[3 (1429876691562397696)]} 0 2
[junit4:junit4] 2> 12055 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10003 (1429876691562397697)]} 0 3
[junit4:junit4] 2> 12057 T849 C30 P49479 /update {wt=javabin&version=2} {add=[3 (1429876691566592000)]} 0 1
[junit4:junit4] 2> 12058 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10003 (1429876691566592001)]} 0 1
[junit4:junit4] 2> 12099 T827 C29 P44555 /update {wt=javabin&version=2} {add=[4 (1429876691609583616)]} 0 1
[junit4:junit4] 2> 12099 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10004 (1429876691609583617)]} 0 1
[junit4:junit4] 2> 12103 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10004 (1429876691613777921)]} 0 1
[junit4:junit4] 2> 12104 T847 C30 P49479 /update {wt=javabin&version=2} {add=[4 (1429876691613777920)]} 0 2
[junit4:junit4] 2> 12111 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[4 (-1429876691623215105)]} 0 1
[junit4:junit4] 2> 12111 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10004 (-1429876691623215104)]} 0 1
[junit4:junit4] 2> 12114 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10004 (-1429876691626360832)]} 0 0
[junit4:junit4] 2> 12114 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[4 (-1429876691626360833)]} 0 0
[junit4:junit4] 2> 12118 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10005 (1429876691629506560)]} 0 1
[junit4:junit4] 2> 12119 T829 C29 P44555 /update {wt=javabin&version=2} {add=[5 (1429876691629506561)]} 0 2
[junit4:junit4] 2> 12122 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10005 (1429876691633700864)]} 0 1
[junit4:junit4] 2> 12123 T848 C30 P49479 /update {wt=javabin&version=2} {add=[5 (1429876691633700865)]} 0 2
[junit4:junit4] 2> 12186 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10006 (1429876691700809728)]} 0 2
[junit4:junit4] 2> 12187 T828 C29 P44555 /update {wt=javabin&version=2} {add=[6 (1429876691700809729)]} 0 3
[junit4:junit4] 2> 12190 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10006 (1429876691705004032)]} 0 1
[junit4:junit4] 2> 12191 T847 C30 P49479 /update {wt=javabin&version=2} {add=[6 (1429876691705004033)]} 0 2
[junit4:junit4] 2> 12287 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10007 (1429876691806715904)]} 0 1
[junit4:junit4] 2> 12288 T829 C29 P44555 /update {wt=javabin&version=2} {add=[7 (1429876691806715905)]} 0 2
[junit4:junit4] 2> 12291 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10007 (1429876691810910208)]} 0 1
[junit4:junit4] 2> 12292 T849 C30 P49479 /update {wt=javabin&version=2} {add=[7 (1429876691810910209)]} 0 2
[junit4:junit4] 2> 12327 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10006 (-1429876691848658944)]} 0 1
[junit4:junit4] 2> 12327 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[6 (-1429876691849707520)]} 0 1
[junit4:junit4] 2> 12333 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[6 (-1429876691855998976)]} 0 0
[junit4:junit4] 2> 12333 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10006 (-1429876691855998977)]} 0 0
[junit4:junit4] 2> 12337 T827 C29 P44555 /update {wt=javabin&version=2} {add=[8 (1429876691859144704)]} 0 1
[junit4:junit4] 2> 12337 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10008 (1429876691859144705)]} 0 1
[junit4:junit4] 2> 12341 T849 C30 P49479 /update {wt=javabin&version=2} {add=[8 (1429876691863339008)]} 0 1
[junit4:junit4] 2> 12342 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10008 (1429876691863339009)]} 0 2
[junit4:junit4] 2> 12422 T830 C29 P44555 /update {wt=javabin&version=2} {add=[9 (1429876691948273665)]} 0 2
[junit4:junit4] 2> 12423 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10009 (1429876691948273664)]} 0 2
[junit4:junit4] 2> 12427 T847 C30 P49479 /update {wt=javabin&version=2} {add=[9 (1429876691952467968)]} 0 2
[junit4:junit4] 2> 12428 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10009 (1429876691953516544)]} 0 3
[junit4:junit4] 2> 12516 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10010 (1429876692046839809)]} 0 1
[junit4:junit4] 2> 12517 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10 (1429876692046839808)]} 0 2
[junit4:junit4] 2> 12520 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10 (1429876692051034113)]} 0 1
[junit4:junit4] 2> 12521 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10010 (1429876692051034112)]} 0 2
[junit4:junit4] 2> 12587 T830 C29 P44555 /update {wt=javabin&version=2} {add=[11 (1429876692121288705)]} 0 2
[junit4:junit4] 2> 12587 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10011 (1429876692121288704)]} 0 2
[junit4:junit4] 2> 12591 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10011 (1429876692125483008)]} 0 2
[junit4:junit4] 2> 12591 T847 C30 P49479 /update {wt=javabin&version=2} {add=[11 (1429876692125483009)]} 0 2
[junit4:junit4] 2> 12643 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12644 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49750__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49750_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49750"}
[junit4:junit4] 2> 12647 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12647 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12647 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12647 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12666 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10012 (1429876692204126208)]} 0 1
[junit4:junit4] 2> 12667 T828 C29 P44555 /update {wt=javabin&version=2} {add=[12 (1429876692204126209)]} 0 2
[junit4:junit4] 2> C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49750_, base_url=http://127.0.0.1:49750}
[junit4:junit4] 2> 12679 T866 C28 P49750 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4fc325a0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b57358d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12680 T866 C28 P49750 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12682 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10012 (1429876692209369088)]} 0 5
[junit4:junit4] 2> 12683 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[12 (1429876692210417664)]} 0 5
[junit4:junit4] 2> 12683 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10012 (1429876692209369088)]} 0 14
[junit4:junit4] 2> 12684 T849 C30 P49479 /update {wt=javabin&version=2} {add=[12 (1429876692210417664)]} 0 13
[junit4:junit4] 2> 12765 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[11 (-1429876692308983809)]} 0 0
[junit4:junit4] 2> 12765 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10011 (-1429876692308983808)]} 0 0
[junit4:junit4] 2> 12773 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10011 (-1429876692312129537)]} 0 1
[junit4:junit4] 2> 12773 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[11 (-1429876692312129536)]} 0 1
[junit4:junit4] 2> 12774 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10011 (-1429876692312129537)]} 0 6
[junit4:junit4] 2> 12774 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[11 (-1429876692312129536)]} 0 6
[junit4:junit4] 2> 12777 T829 C29 P44555 /update {wt=javabin&version=2} {add=[13 (1429876692320518144)]} 0 1
[junit4:junit4] 2> 12778 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10013 (1429876692321566720)]} 0 2
[junit4:junit4] 2> 12785 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[13 (1429876692324712448)]} 0 2
[junit4:junit4] 2> 12786 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10013 (1429876692324712449)]} 0 2
[junit4:junit4] 2> 12786 T849 C30 P49479 /update {wt=javabin&version=2} {add=[13 (1429876692324712448)]} 0 6
[junit4:junit4] 2> 12786 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10013 (1429876692324712449)]} 0 6
[junit4:junit4] 2> 12808 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10014 (1429876692351975424)]} 0 2
[junit4:junit4] 2> 12810 T827 C29 P44555 /update {wt=javabin&version=2} {add=[14 (1429876692351975425)]} 0 4
[junit4:junit4] 2> 12816 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1429876692357218304)]} 0 2
[junit4:junit4] 2> 12816 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10014 (1429876692357218304)]} 0 5
[junit4:junit4] 2> 12819 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1429876692359315456)]} 0 2
[junit4:junit4] 2> 12820 T849 C30 P49479 /update {wt=javabin&version=2} {add=[14 (1429876692359315456)]} 0 7
[junit4:junit4] 2> 12856 T850 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:49750__collection1&state=recovering&nodeName=127.0.0.1:49750_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 12865 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10014 (-1429876692412792832)]} 0 1
[junit4:junit4] 2> 12868 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[14 (-1429876692416987136)]} 0 0
[junit4:junit4] 2> 12870 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10014 (-1429876692416987136)]} 0 0
[junit4:junit4] 2> 12871 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10014 (-1429876692416987136)]} 0 3
[junit4:junit4] 2> 12873 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[14 (-1429876692420132864)]} 0 0
[junit4:junit4] 2> 12874 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[14 (-1429876692420132864)]} 0 3
[junit4:junit4] 2> 12875 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10015 (1429876692423278592)]} 0 2
[junit4:junit4] 2> 12878 T828 C29 P44555 /update {wt=javabin&version=2} {add=[15 (1429876692426424320)]} 0 2
[junit4:junit4] 2> 12882 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1429876692426424320)]} 0 1
[junit4:junit4] 2> 12883 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10015 (1429876692426424320)]} 0 6
[junit4:junit4] 2> 12886 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1429876692430618624)]} 0 2
[junit4:junit4] 2> 12887 T847 C30 P49479 /update {wt=javabin&version=2} {add=[15 (1429876692430618624)]} 0 6
[junit4:junit4] 2> 12974 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10016 (1429876692527087616)]} 0 2
[junit4:junit4] 2> 12977 T827 C29 P44555 /update {wt=javabin&version=2} {add=[16 (1429876692530233344)]} 0 1
[junit4:junit4] 2> 12982 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1429876692531281920)]} 0 2
[junit4:junit4] 2> 12982 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10016 (1429876692531281920)]} 0 5
[junit4:junit4] 2> 12986 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1429876692534427648)]} 0 2
[junit4:junit4] 2> 12987 T850 C30 P49479 /update {wt=javabin&version=2} {add=[16 (1429876692534427648)]} 0 7
[junit4:junit4] 2> 12991 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10015 (-1429876692545961984)]} 0 0
[junit4:junit4] 2> 12996 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[15 (-1429876692551204864)]} 0 0
[junit4:junit4] 2> 12996 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10015 (-1429876692549107712)]} 0 0
[junit4:junit4] 2> 12997 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10015 (-1429876692549107712)]} 0 3
[junit4:junit4] 2> 13001 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10017 (1429876692555399168)]} 0 1
[junit4:junit4] 2> 13002 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[15 (-1429876692554350592)]} 0 1
[junit4:junit4] 2> 13002 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[15 (-1429876692554350592)]} 0 3
[junit4:junit4] 2> 13006 T827 C29 P44555 /update {wt=javabin&version=2} {add=[17 (1429876692560642048)]} 0 1
[junit4:junit4] 2> 13010 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1429876692559593472)]} 0 2
[junit4:junit4] 2> 13011 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10017 (1429876692559593472)]} 0 7
[junit4:junit4] 2> 13015 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1429876692565884928)]} 0 2
[junit4:junit4] 2> 13016 T850 C30 P49479 /update {wt=javabin&version=2} {add=[17 (1429876692565884928)]} 0 7
[junit4:junit4] 2> 13035 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10018 (1429876692590002176)]} 0 1
[junit4:junit4] 2> 13039 T828 C29 P44555 /update {wt=javabin&version=2} {add=[18 (1429876692595245056)]} 0 1
[junit4:junit4] 2> 13043 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1429876692595245056)]} 0 2
[junit4:junit4] 2> 13044 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10018 (1429876692595245056)]} 0 6
[junit4:junit4] 2> 13047 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1429876692599439360)]} 0 1
[junit4:junit4] 2> 13048 T849 C30 P49479 /update {wt=javabin&version=2} {add=[18 (1429876692599439360)]} 0 6
[junit4:junit4] 2> 13122 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10016 (-1429876692683325440)]} 0 0
[junit4:junit4] 2> 13126 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[16 (-1429876692687519744)]} 0 0
[junit4:junit4] 2> 13128 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10016 (-1429876692686471168)]} 0 0
[junit4:junit4] 2> 13129 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10016 (-1429876692686471168)]} 0 4
[junit4:junit4] 2> 13131 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[16 (-1429876692690665472)]} 0 0
[junit4:junit4] 2> 13132 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[16 (-1429876692690665472)]} 0 4
[junit4:junit4] 2> 13132 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10019 (1429876692692762624)]} 0 1
[junit4:junit4] 2> 13135 T828 C29 P44555 /update {wt=javabin&version=2} {add=[19 (1429876692695908352)]} 0 1
[junit4:junit4] 2> 13139 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1429876692696956928)]} 0 1
[junit4:junit4] 2> 13141 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10019 (1429876692696956928)]} 0 6
[junit4:junit4] 2> 13143 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1429876692700102656)]} 0 2
[junit4:junit4] 2> 13144 T849 C30 P49479 /update {wt=javabin&version=2} {add=[19 (1429876692700102656)]} 0 6
[junit4:junit4] 2> 13157 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10020 (1429876692718977024)]} 0 1
[junit4:junit4] 2> 13159 T827 C29 P44555 /update {wt=javabin&version=2} {add=[20 (1429876692721074176)]} 0 1
[junit4:junit4] 2> 13164 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1429876692723171328)]} 0 0
[junit4:junit4] 2> 13164 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10020 (1429876692723171328)]} 0 4
[junit4:junit4] 2> 13166 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1429876692725268480)]} 0 2
[junit4:junit4] 2> 13166 T850 C30 P49479 /update {wt=javabin&version=2} {add=[20 (1429876692725268480)]} 0 5
[junit4:junit4] 2> 13222 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10021 (1429876692786085888)]} 0 2
[junit4:junit4] 2> 13224 T828 C29 P44555 /update {wt=javabin&version=2} {add=[21 (1429876692788183040)]} 0 2
[junit4:junit4] 2> 13231 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1429876692791328768)]} 0 2
[junit4:junit4] 2> 13232 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10021 (1429876692791328768)]} 0 7
[junit4:junit4] 2> 13232 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1429876692793425920)]} 0 2
[junit4:junit4] 2> 13233 T849 C30 P49479 /update {wt=javabin&version=2} {add=[21 (1429876692793425920)]} 0 6
[junit4:junit4] 2> 13317 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10022 (1429876692886749184)]} 0 1
[junit4:junit4] 2> 13318 T829 C29 P44555 /update {wt=javabin&version=2} {add=[22 (1429876692887797760)]} 0 1
[junit4:junit4] 2> 13324 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1429876692890943488)]} 0 1
[junit4:junit4] 2> 13325 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1429876692891992064)]} 0 1
[junit4:junit4] 2> 13325 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10022 (1429876692890943488)]} 0 5
[junit4:junit4] 2> 13326 T848 C30 P49479 /update {wt=javabin&version=2} {add=[22 (1429876692891992064)]} 0 6
[junit4:junit4] 2> 13390 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[17 (-1429876692964343809)]} 0 1
[junit4:junit4] 2> 13390 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[10017 (-1429876692964343808)]} 0 1
[junit4:junit4] 2> 13395 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10017 (-1429876692967489536)]} 0 0
[junit4:junit4] 2> 13395 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[17 (-1429876692967489537)]} 0 0
[junit4:junit4] 2> 13396 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10017 (-1429876692967489536)]} 0 3
[junit4:junit4] 2> 13396 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[17 (-1429876692967489537)]} 0 3
[junit4:junit4] 2> 13400 T830 C29 P44555 /update {wt=javabin&version=2} {add=[23 (1429876692973780992)]} 0 1
[junit4:junit4] 2> 13400 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10023 (1429876692973780993)]} 0 1
[junit4:junit4] 2> 13408 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1429876692977975296)]} 0 2
[junit4:junit4] 2> 13409 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1429876692977975297)]} 0 2
[junit4:junit4] 2> 13409 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10023 (1429876692977975296)]} 0 6
[junit4:junit4] 2> 13409 T848 C30 P49479 /update {wt=javabin&version=2} {add=[23 (1429876692977975297)]} 0 6
[junit4:junit4] 2> 13454 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[19 (-1429876693031452673)]} 0 1
[junit4:junit4] 2> 13454 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[10019 (-1429876693031452672)]} 0 1
[junit4:junit4] 2> 13459 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10019 (-1429876693034598401)]} 0 0
[junit4:junit4] 2> 13459 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[19 (-1429876693034598400)]} 0 0
[junit4:junit4] 2> 13460 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[19 (-1429876693034598400)]} 0 3
[junit4:junit4] 2> 13460 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10019 (-1429876693034598401)]} 0 3
[junit4:junit4] 2> 13464 T830 C29 P44555 /update {wt=javabin&version=2} {add=[24 (1429876693040889856)]} 0 1
[junit4:junit4] 2> 13464 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10024 (1429876693040889857)]} 0 1
[junit4:junit4] 2> 13471 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1429876693045084160)]} 0 1
[junit4:junit4] 2> 13472 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1429876693045084161)]} 0 2
[junit4:junit4] 2> 13472 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10024 (1429876693045084160)]} 0 6
[junit4:junit4] 2> 13472 T850 C30 P49479 /update {wt=javabin&version=2} {add=[24 (1429876693045084161)]} 0 6
[junit4:junit4] 2> 13476 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[10020 (-1429876693054521344)]} 0 0
[junit4:junit4] 2> 13476 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[20 (-1429876693054521345)]} 0 0
[junit4:junit4] 2> 13482 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[20 (-1429876693057667073)]} 0 1
[junit4:junit4] 2> 13482 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10020 (-1429876693057667072)]} 0 1
[junit4:junit4] 2> 13483 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[20 (-1429876693057667073)]} 0 4
[junit4:junit4] 2> 13483 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10020 (-1429876693057667072)]} 0 4
[junit4:junit4] 2> 13487 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10025 (1429876693065007104)]} 0 1
[junit4:junit4] 2> 13487 T829 C29 P44555 /update {wt=javabin&version=2} {add=[25 (1429876693065007105)]} 0 1
[junit4:junit4] 2> 13495 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1429876693069201409)]} 0 2
[junit4:junit4] 2> 13495 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1429876693069201408)]} 0 2
[junit4:junit4] 2> 13496 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10025 (1429876693069201409)]} 0 6
[junit4:junit4] 2> 13496 T850 C30 P49479 /update {wt=javabin&version=2} {add=[25 (1429876693069201408)]} 0 6
[junit4:junit4] 2> 13561 T828 C29 P44555 /update {wt=javabin&version=2} {add=[26 (1429876693142601728)]} 0 1
[junit4:junit4] 2> 13562 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10026 (1429876693142601729)]} 0 2
[junit4:junit4] 2> 13569 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1429876693147844608)]} 0 1
[junit4:junit4] 2> 13570 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1429876693147844609)]} 0 1
[junit4:junit4] 2> 13570 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10026 (1429876693147844608)]} 0 6
[junit4:junit4] 2> 13571 T847 C30 P49479 /update {wt=javabin&version=2} {add=[26 (1429876693147844609)]} 0 7
[junit4:junit4] 2> 13634 T830 C29 P44555 /update {wt=javabin&version=2} {add=[27 (1429876693219147777)]} 0 2
[junit4:junit4] 2> 13634 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10027 (1429876693219147776)]} 0 2
[junit4:junit4] 2> 13641 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1429876693223342081)]} 0 1
[junit4:junit4] 2> 13642 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1429876693223342080)]} 0 2
[junit4:junit4] 2> 13642 T848 C30 P49479 /update {wt=javabin&version=2} {add=[27 (1429876693223342081)]} 0 5
[junit4:junit4] 2> 13642 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10027 (1429876693223342080)]} 0 5
[junit4:junit4] 2> 13702 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10021 (-1429876693290450945)]} 0 1
[junit4:junit4] 2> 13702 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[21 (-1429876693290450944)]} 0 1
[junit4:junit4] 2> 13707 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10021 (-1429876693293596672)]} 0 0
[junit4:junit4] 2> 13707 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[21 (-1429876693294645248)]} 0 0
[junit4:junit4] 2> 13708 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[10021 (-1429876693293596672)]} 0 4
[junit4:junit4] 2> 13708 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[21 (-1429876693294645248)]} 0 4
[junit4:junit4] 2> 13712 T829 C29 P44555 /update {wt=javabin&version=2} {add=[28 (1429876693300936704)]} 0 1
[junit4:junit4] 2> 13712 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10028 (1429876693300936705)]} 0 1
[junit4:junit4] 2> 13720 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1429876693306179584)]} 0 1
[junit4:junit4] 2> 13721 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1429876693306179585)]} 0 2
[junit4:junit4] 2> 13721 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10028 (1429876693306179584)]} 0 6
[junit4:junit4] 2> 13722 T850 C30 P49479 /update {wt=javabin&version=2} {add=[28 (1429876693306179585)]} 0 7
[junit4:junit4] 2> 13781 T827 C29 P44555 /update {wt=javabin&version=2} {add=[29 (1429876693373288448)]} 0 1
[junit4:junit4] 2> 13781 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10029 (1429876693373288449)]} 0 1
[junit4:junit4] 2> 13788 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1429876693377482752)]} 0 2
[junit4:junit4] 2> 13789 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1429876693377482753)]} 0 2
[junit4:junit4] 2> 13789 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10029 (1429876693377482752)]} 0 5
[junit4:junit4] 2> 13789 T849 C30 P49479 /update {wt=javabin&version=2} {add=[29 (1429876693377482753)]} 0 5
[junit4:junit4] 2> 13857 T812 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 49750
[junit4:junit4] 2> 13857 T812 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1126178006
[junit4:junit4] 2> 13862 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10022 (-1429876693459271680)]} 0 0
[junit4:junit4] 2> 13862 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[22 (-1429876693459271681)]} 0 0
[junit4:junit4] 2> 13869 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1429876693462417409)]} 0 1
[junit4:junit4] 2> 13869 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1429876693462417408)]} 0 1
[junit4:junit4] 2> 13870 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[22 (-1429876693462417408)]} 0 5
[junit4:junit4] 2> 13870 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10022 (-1429876693462417409)]} 0 5
[junit4:junit4] 2> 13874 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10030 (1429876693470806016)]} 0 1
[junit4:junit4] 2> 13875 T827 C29 P44555 /update {wt=javabin&version=2} {add=[30 (1429876693470806017)]} 0 2
[junit4:junit4] 2> 13882 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1429876693475000321)]} 0 2
[junit4:junit4] 2> 13882 T847 C30 P49479 /update {wt=javabin&version=2} {add=[30 (1429876693475000321)]} 0 5
[junit4:junit4] 2> 13882 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1429876693475000320)]} 0 2
[junit4:junit4] 2> 13883 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10030 (1429876693475000320)]} 0 7
[junit4:junit4] 2> 13896 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10023 (-1429876693494923265)]} 0 1
[junit4:junit4] 2> 13896 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[23 (-1429876693494923264)]} 0 1
[junit4:junit4] 2> 13901 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1429876693498068992)]} 0 0
[junit4:junit4] 2> 13901 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1429876693498068993)]} 0 0
[junit4:junit4] 2> 13902 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10023 (-1429876693498068993)]} 0 3
[junit4:junit4] 2> 13902 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[23 (-1429876693498068992)]} 0 3
[junit4:junit4] 2> 13906 T828 C29 P44555 /update {wt=javabin&version=2} {add=[31 (1429876693504360449)]} 0 1
[junit4:junit4] 2> 13907 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10031 (1429876693504360448)]} 0 2
[junit4:junit4] 2> 13915 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1429876693509603328)]} 0 2
[junit4:junit4] 2> 13915 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1429876693508554752)]} 0 1
[junit4:junit4] 2> 13916 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10031 (1429876693509603328)]} 0 7
[junit4:junit4] 2> 13916 T847 C30 P49479 /update {wt=javabin&version=2} {add=[31 (1429876693508554752)]} 0 7
[junit4:junit4] 2> 13935 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10032 (1429876693534769152)]} 0 1
[junit4:junit4] 2> 13935 T829 C29 P44555 /update {wt=javabin&version=2} {add=[32 (1429876693534769153)]} 0 1
[junit4:junit4] 2> 13947 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1429876693540012032)]} 0 2
[junit4:junit4] 2> 13948 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1429876693540012033)]} 0 3
[junit4:junit4] 2> 13949 T848 C30 P49479 /update {wt=javabin&version=2} {add=[32 (1429876693540012033)]} 0 11
[junit4:junit4] 2> 13949 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10032 (1429876693540012032)]} 0 11
[junit4:junit4] 2> 13999 T828 C29 P44555 /update {wt=javabin&version=2} {add=[33 (1429876693600829441)]} 0 2
[junit4:junit4] 2> 13999 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10033 (1429876693600829440)]} 0 2
[junit4:junit4] 2> 14009 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1429876693606072321)]} 0 2
[junit4:junit4] 2> 14010 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10033 (1429876693606072321)]} 0 8
[junit4:junit4] 2> 14011 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1429876693606072320)]} 0 3
[junit4:junit4] 2> 14012 T849 C30 P49479 /update {wt=javabin&version=2} {add=[33 (1429876693606072320)]} 0 10
[junit4:junit4] 2> 14071 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10034 (1429876693677375488)]} 0 1
[junit4:junit4] 2> 14073 T829 C29 P44555 /update {wt=javabin&version=2} {add=[34 (1429876693679472640)]} 0 1
[junit4:junit4] 2> 14078 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1429876693681569792)]} 0 1
[junit4:junit4] 2> 14079 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10034 (1429876693681569792)]} 0 5
[junit4:junit4] 2> 14082 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1429876693683666944)]} 0 1
[junit4:junit4] 2> 14083 T847 C30 P49479 /update {wt=javabin&version=2} {add=[34 (1429876693683666944)]} 0 7
[junit4:junit4] 2> 14138 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[10024 (-1429876693748678656)]} 0 0
[junit4:junit4] 2> 14142 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[24 (-1429876693751824384)]} 0 1
[junit4:junit4] 2> 14143 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1429876693751824384)]} 0 0
[junit4:junit4] 2> 14144 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10024 (-1429876693751824384)]} 0 3
[junit4:junit4] 2> 14147 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1429876693756018688)]} 0 0
[junit4:junit4] 2> 14148 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[24 (-1429876693756018688)]} 0 4
[junit4:junit4] 2> 14148 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10035 (1429876693758115840)]} 0 1
[junit4:junit4] 2> 14150 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 14151 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49750__collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49750_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49750"}
[junit4:junit4] 2> 14152 T829 C29 P44555 /update {wt=javabin&version=2} {add=[35 (1429876693762310144)]} 0 2
[junit4:junit4] 2> 14155 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1429876693762310144)]} 0 1
[junit4:junit4] 2> 14156 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10035 (1429876693762310144)]} 0 6
[junit4:junit4] 2> 14159 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1429876693766504448)]} 0 1
[junit4:junit4] 2> 14161 T847 C30 P49479 /update {wt=javabin&version=2} {add=[35 (1429876693766504448)]} 0 7
[junit4:junit4] 2> 14163 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14163 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14163 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14163 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 14250 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10036 (1429876693866119168)]} 0 1
[junit4:junit4] 2> 14254 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10036 (1429876693869264896)]} 0 1
[junit4:junit4] 2> 14254 T830 C29 P44555 /update {wt=javabin&version=2} {add=[36 (1429876693869264896)]} 0 1
[junit4:junit4] 2> 14258 T850 C30 P49479 /update {wt=javabin&version=2} {add=[36 (1429876693873459200)]} 0 1
[junit4:junit4] 2> 14272 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10025 (-1429876693889187840)]} 0 0
[junit4:junit4] 2> 14276 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[25 (-1429876693892333568)]} 0 1
[junit4:junit4] 2> 14276 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10025 (-1429876693892333568)]} 0 1
[junit4:junit4] 2> 14279 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[25 (-1429876693896527872)]} 0 0
[junit4:junit4] 2> 14280 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10037 (1429876693896527872)]} 0 1
[junit4:junit4] 2> 14283 T830 C29 P44555 /update {wt=javabin&version=2} {add=[37 (1429876693899673600)]} 0 1
[junit4:junit4] 2> 14284 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10037 (1429876693900722176)]} 0 2
[junit4:junit4] 2> 14287 T847 C30 P49479 /update {wt=javabin&version=2} {add=[37 (1429876693903867904)]} 0 1
[junit4:junit4] 2> 14336 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10027 (-1429876693955248128)]} 0 1
[junit4:junit4] 2> 14339 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10027 (-1429876693958393856)]} 0 1
[junit4:junit4] 2> 14339 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[27 (-1429876693958393856)]} 0 1
[junit4:junit4] 2> 14342 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[27 (-1429876693961539584)]} 0 1
[junit4:junit4] 2> 14343 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10038 (1429876693962588160)]} 0 2
[junit4:junit4] 2> 14345 T830 C29 P44555 /update {wt=javabin&version=2} {add=[38 (1429876693964685312)]} 0 1
[junit4:junit4] 2> 14346 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10038 (1429876693965733888)]} 0 1
[junit4:junit4] 2> 14349 T847 C30 P49479 /update {wt=javabin&version=2} {add=[38 (1429876693968879616)]} 0 1
[junit4:junit4] 2> 14373 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10039 (1429876693994045440)]} 0 2
[junit4:junit4] 2> 14375 T827 C29 P44555 /update {wt=javabin&version=2} {add=[39 (1429876693997191168)]} 0 1
[junit4:junit4] 2> 14376 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10039 (1429876693997191168)]} 0 1
[junit4:junit4] 2> 14379 T849 C30 P49479 /update {wt=javabin&version=2} {add=[39 (1429876694000336896)]} 0 1
[junit4:junit4] 2> 14391 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10040 (1429876694012919808)]} 0 1
[junit4:junit4] 2> 14394 T830 C29 P44555 /update {wt=javabin&version=2} {add=[40 (1429876694016065536)]} 0 1
[junit4:junit4] 2> 14395 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10040 (1429876694017114112)]} 0 1
[junit4:junit4] 2> 14398 T847 C30 P49479 /update {wt=javabin&version=2} {add=[40 (1429876694020259840)]} 0 1
[junit4:junit4] 2> 14461 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10041 (1429876694086320128)]} 0 1
[junit4:junit4] 2> 14464 T827 C29 P44555 /update {wt=javabin&version=2} {add=[41 (1429876694089465856)]} 0 1
[junit4:junit4] 2> 14465 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10041 (1429876694090514432)]} 0 1
[junit4:junit4] 2> 14468 T849 C30 P49479 /update {wt=javabin&version=2} {add=[41 (1429876694093660160)]} 0 1
[junit4:junit4] 2> 14487 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10042 (1429876694113583104)]} 0 1
[junit4:junit4] 2> 14490 T830 C29 P44555 /update {wt=javabin&version=2} {add=[42 (1429876694116728832)]} 0 2
[junit4:junit4] 2> 14490 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10042 (1429876694117777408)]} 0 1
[junit4:junit4] 2> 14494 T848 C30 P49479 /update {wt=javabin&version=2} {add=[42 (1429876694120923136)]} 0 1
[junit4:junit4] 2> 14520 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10028 (-1429876694149234688)]} 0 1
[junit4:junit4] 2> 14523 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[28 (-1429876694152380416)]} 0 1
[junit4:junit4] 2> 14523 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10028 (-1429876694152380416)]} 0 1
[junit4:junit4] 2> 14526 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[28 (-1429876694155526144)]} 0 1
[junit4:junit4] 2> 14526 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10043 (1429876694155526144)]} 0 1
[junit4:junit4] 2> 14530 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10043 (1429876694158671872)]} 0 1
[junit4:junit4] 2> 14530 T830 C29 P44555 /update {wt=javabin&version=2} {add=[43 (1429876694158671872)]} 0 1
[junit4:junit4] 2> 14534 T848 C30 P49479 /update {wt=javabin&version=2} {add=[43 (1429876694162866176)]} 0 1
[junit4:junit4] 2> 14593 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10031 (-1429876694225780736)]} 0 0
[junit4:junit4] 2> 14596 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10031 (-1429876694228926464)]} 0 0
[junit4:junit4] 2> 14597 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[31 (-1429876694229975040)]} 0 0
[junit4:junit4] 2> 14600 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[31 (-1429876694233120768)]} 0 1
[junit4:junit4] 2> 14600 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10044 (1429876694232072192)]} 0 2
[junit4:junit4] 2> 14604 T830 C29 P44555 /update {wt=javabin&version=2} {add=[44 (1429876694236266496)]} 0 1
[junit4:junit4] 2> 14604 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10044 (1429876694236266496)]} 0 1
[junit4:junit4] 2> 14608 T848 C30 P49479 /update {wt=javabin&version=2} {add=[44 (1429876694240460800)]} 0 2
[junit4:junit4] 2> 14668 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10045 (1429876694304423936)]} 0 1
[junit4:junit4] 2> 14672 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10045 (1429876694307569664)]} 0 1
[junit4:junit4] 2> 14672 T829 C29 P44555 /update {wt=javabin&version=2} {add=[45 (1429876694308618240)]} 0 0
[junit4:junit4] 2> 14676 T847 C30 P49479 /update {wt=javabin&version=2} {add=[45 (1429876694311763968)]} 0 1
[junit4:junit4] 2> 14726 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10032 (-1429876694365241344)]} 0 1
[junit4:junit4] 2> 14729 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[32 (-1429876694368387072)]} 0 0
[junit4:junit4] 2> 14729 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10032 (-1429876694368387072)]} 0 0
[junit4:junit4] 2> 14732 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[32 (-1429876694371532800)]} 0 0
[junit4:junit4] 2> 14733 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10046 (1429876694371532800)]} 0 1
[junit4:junit4] 2> 14737 T830 C29 P44555 /update {wt=javabin&version=2} {add=[46 (1429876694375727104)]} 0 2
[junit4:junit4] 2> 14737 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10046 (1429876694375727104)]} 0 1
[junit4:junit4] 2> 14742 T847 C30 P49479 /update {wt=javabin&version=2} {add=[46 (1429876694380969984)]} 0 2
[junit4:junit4] 2> 14813 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10047 (1429876694456467456)]} 0 1
[junit4:junit4] 2> 14817 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10047 (1429876694460661760)]} 0 1
[junit4:junit4] 2> 14818 T827 C29 P44555 /update {wt=javabin&version=2} {add=[47 (1429876694460661760)]} 0 1
[junit4:junit4] 2> 14821 T848 C30 P49479 /update {wt=javabin&version=2} {add=[47 (1429876694464856064)]} 0 1
[junit4:junit4] 2> C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49750_, base_url=http://127.0.0.1:49750}
[junit4:junit4] 2> 14857 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49479/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 14857 T873 C28 P49750 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49750 START replicas=[http://127.0.0.1:49479/collection1/] nUpdates=100
[junit4:junit4] 2> 14859 T873 C28 P49750 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 14860 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 14860 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 14860 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 14860 T873 C28 P49750 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 14860 T847 C30 P49479 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 14860 T873 C28 P49750 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49479/collection1/. core=collection1
[junit4:junit4] 2> 14860 T873 C28 P49750 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 14865 T850 C30 P49479 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14880 T850 C30 P49479 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@37d4ca92 lockFactory=org.apache.lucene.store.NativeFSLockFactory@407834e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@37d4ca92 lockFactory=org.apache.lucene.store.NativeFSLockFactory@407834e),segFN=segments_2,generation=2,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene40_0.prx, _0.fdt, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14880 T850 C30 P49479 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene40_0.prx, _0.fdt, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14882 T850 C30 P49479 oass.SolrIndexSearcher.<init> Opening Searcher@77925ee1 realtime
[junit4:junit4] 2> 14882 T850 C30 P49479 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 14882 T850 C30 P49479 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 14883 T873 C28 P49750 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14883 T873 C28 P49750 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 14886 T849 C30 P49479 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 14886 T849 C30 P49479 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 14887 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 14887 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 14887 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 14889 T847 C30 P49479 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 14890 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 12
[junit4:junit4] 2> 14891 T873 C28 P49750 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index.20130318135759208 forceNew: false
[junit4:junit4] 2> 14891 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Starting download to BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)) fullCopy=true
[junit4:junit4] 2> 14893 T850 C30 P49479 REQ /replication {file=_0_Lucene40_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14900 T848 C30 P49479 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14903 T849 C30 P49479 REQ /replication {file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14907 T847 C30 P49479 REQ /replication {file=_0_nrm.cfs&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14910 T850 C30 P49479 REQ /replication {file=_0_Lucene40_0.prx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14911 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10033 (-1429876694558179328)]} 0 1
[junit4:junit4] 2> 14913 T848 C30 P49479 REQ /replication {file=_0_Lucene40_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14914 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[10033 (-1429876694561325056)]} 0 1
[junit4:junit4] 2> 14914 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[33 (-1429876694562373632)]} 0 0
[junit4:junit4] 2> 14916 T850 C30 P49479 REQ /replication {file=_0_Lucene40_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14918 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[33 (-1429876694565519360)]} 0 1
[junit4:junit4] 2> 14919 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10048 (1429876694565519360)]} 0 2
[junit4:junit4] 2> 14920 T848 C30 P49479 REQ /replication {file=_0_nrm.cfe&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14922 T827 C29 P44555 /update {wt=javabin&version=2} {add=[48 (1429876694568665088)]} 0 2
[junit4:junit4] 2> 14923 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10048 (1429876694570762240)]} 0 2
[junit4:junit4] 2> 14924 T850 C30 P49479 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14926 T847 C30 P49479 /update {wt=javabin&version=2} {add=[48 (1429876694573907968)]} 0 1
[junit4:junit4] 2> 14928 T849 C30 P49479 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 14931 T848 C30 P49479 REQ /replication {file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14934 T850 C30 P49479 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 14936 T873 C28 P49750 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 14936 T873 C28 P49750 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130318135759208
[junit4:junit4] 2> 14937 T873 C28 P49750 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 14937 T873 C28 P49750 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 14937 T873 C28 P49750 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4] 2> 14940 T873 C28 P49750 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index.20130318135759208
[junit4:junit4] 2> 14941 T873 C28 P49750 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)),segFN=segments_2,generation=2,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, segments_2, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14941 T873 C28 P49750 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, segments_2, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14942 T873 C28 P49750 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 14943 T873 C28 P49750 oass.SolrIndexSearcher.<init> Opening Searcher@46f9b6c9 main
[junit4:junit4] 2> 14944 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46f9b6c9 main{StandardDirectoryReader(segments_2:3:nrt _0(4.3):C94/36)}
[junit4:junit4] 2> 14944 T873 C28 P49750 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@4fc325a0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b57358d)
[junit4:junit4] 2> 14945 T873 C28 P49750 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/index
[junit4:junit4] 2> 14945 T873 C28 P49750 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@4fc325a0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5b57358d)
[junit4:junit4] 2> 14947 T873 C28 P49750 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 14947 T881 C28 P49750 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363636664322/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5443
[junit4:junit4] 2> 14949 T881 C28 P49750 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 14950 T881 C28 P49750 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)),segFN=segments_2,generation=2,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, segments_2, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, _0_Lucene40_0.tip]
[junit4:junit4] 2> commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)),segFN=segments_3,generation=3,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, segments_3, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14950 T881 C28 P49750 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, segments_3, _0_Lucene40_0.tip]
[junit4:junit4] 2> 14951 T881 C28 P49750 oass.SolrIndexSearcher.<init> Opening Searcher@70de6db3 main
[junit4:junit4] 2> 14951 T881 C28 P49750 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 14952 T872 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70de6db3 main{StandardDirectoryReader(segments_2:3:nrt _0(4.3):C94/36)}
[junit4:junit4] 2> 14953 T881 C28 P49750 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5443}
[junit4:junit4] 2> 14953 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 14953 T873 C28 P49750 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 14953 T873 C28 P49750 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 14954 T873 C28 P49750 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 14995 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10049 (1429876694646259712)]} 0 1
[junit4:junit4] 2> 14999 T830 C29 P44555 /update {wt=javabin&version=2} {add=[49 (1429876694650454016)]} 0 1
[junit4:junit4] 2> 15000 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10049 (1429876694651502592)]} 0 2
[junit4:junit4] 2> 15002 T849 C30 P49479 /update {wt=javabin&version=2} {add=[49 (1429876694654648320)]} 0 1
[junit4:junit4] 2> 15077 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10050 (1429876694732242944)]} 0 1
[junit4:junit4] 2> 15080 T829 C29 P44555 /update {wt=javabin&version=2} {add=[50 (1429876694734340096)]} 0 2
[junit4:junit4] 2> 15081 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10050 (1429876694736437248)]} 0 2
[junit4:junit4] 2> 15084 T850 C30 P49479 /update {wt=javabin&version=2} {add=[50 (1429876694739582976)]} 0 2
[junit4:junit4] 2> 15133 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10051 (1429876694792011776)]} 0 1
[junit4:junit4] 2> 15136 T828 C29 P44555 /update {wt=javabin&version=2} {add=[51 (1429876694795157504)]} 0 1
[junit4:junit4] 2> 15137 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10051 (1429876694795157504)]} 0 1
[junit4:junit4] 2> 15140 T849 C30 P49479 /update {wt=javabin&version=2} {add=[51 (1429876694798303232)]} 0 1
[junit4:junit4] 2> 15179 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10052 (1429876694840246272)]} 0 1
[junit4:junit4] 2> 15181 T827 C29 P44555 /update {wt=javabin&version=2} {add=[52 (1429876694841294848)]} 0 1
[junit4:junit4] 2> 15183 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10052 (1429876694843392000)]} 0 1
[junit4:junit4] 2> 15184 T850 C30 P49479 /update {wt=javabin&version=2} {add=[52 (1429876694844440576)]} 0 1
[junit4:junit4] 2> 15277 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10053 (1429876694941958144)]} 0 1
[junit4:junit4] 2> 15278 T828 C29 P44555 /update {wt=javabin&version=2} {add=[53 (1429876694943006720)]} 0 1
[junit4:junit4] 2> 15281 T849 C30 P49479 /update {wt=javabin&version=2} {add=[53 (1429876694947201024)]} 0 1
[junit4:junit4] 2> 15282 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10053 (1429876694947201025)]} 0 2
[junit4:junit4] 2> 15348 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[35 (-1429876695017455616)]} 0 1
[junit4:junit4] 2> 15350 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[10035 (-1429876695019552768)]} 0 1
[junit4:junit4] 2> 15351 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[35 (-1429876695020601344)]} 0 0
[junit4:junit4] 2> 15353 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10035 (-1429876695022698496)]} 0 1
[junit4:junit4] 2> 15355 T830 C29 P44555 /update {wt=javabin&version=2} {add=[54 (1429876695023747072)]} 0 1
[junit4:junit4] 2> 15356 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10054 (1429876695024795648)]} 0 1
[junit4:junit4] 2> 15360 T847 C30 P49479 /update {wt=javabin&version=2} {add=[54 (1429876695028989952)]} 0 1
[junit4:junit4] 2> 15361 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10054 (1429876695028989953)]} 0 2
[junit4:junit4] 2> 15435 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10055 (1429876695107633152)]} 0 1
[junit4:junit4] 2> 15435 T829 C29 P44555 /update {wt=javabin&version=2} {add=[55 (1429876695107633153)]} 0 2
[junit4:junit4] 2> 15439 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10055 (1429876695111827456)]} 0 1
[junit4:junit4] 2> 15440 T849 C30 P49479 /update {wt=javabin&version=2} {add=[55 (1429876695111827457)]} 0 2
[junit4:junit4] 2> 15503 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10056 (1429876695177887744)]} 0 1
[junit4:junit4] 2> 15503 T827 C29 P44555 /update {wt=javabin&version=2} {add=[56 (1429876695177887745)]} 0 2
[junit4:junit4] 2> 15507 T850 C30 P49479 /update {wt=javabin&version=2} {add=[56 (1429876695183130625)]} 0 2
[junit4:junit4] 2> 15507 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10056 (1429876695183130624)]} 0 2
[junit4:junit4] 2> 15560 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[10036 (-1429876695239753729)]} 0 0
[junit4:junit4] 2> 15560 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[36 (-1429876695239753728)]} 0 0
[junit4:junit4] 2> 15564 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[36 (-1429876695242899456)]} 0 1
[junit4:junit4] 2> 15564 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10036 (-1429876695242899457)]} 0 1
[junit4:junit4] 2> 15570 T830 C29 P44555 /update {wt=javabin&version=2} {add=[57 (1429876695247093760)]} 0 4
[junit4:junit4] 2> 15571 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10057 (1429876695247093761)]} 0 5
[junit4:junit4] 2> 15575 T850 C30 P49479 /update {wt=javabin&version=2} {add=[57 (1429876695254433792)]} 0 2
[junit4:junit4] 2> 15578 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10057 (1429876695257579520)]} 0 1
[junit4:junit4] 2> 15622 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[37 (-1429876695304765440)]} 0 0
[junit4:junit4] 2> 15624 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10037 (-1429876695306862592)]} 0 0
[junit4:junit4] 2> 15626 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[37 (-1429876695308959744)]} 0 1
[junit4:junit4] 2> 15628 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10037 (-1429876695311056896)]} 0 1
[junit4:junit4] 2> 15630 T827 C29 P44555 /update {wt=javabin&version=2} {add=[58 (1429876695312105472)]} 0 2
[junit4:junit4] 2> 15632 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10058 (1429876695314202624)]} 0 1
[junit4:junit4] 2> 15633 T850 C30 P49479 /update {wt=javabin&version=2} {add=[58 (1429876695315251200)]} 0 1
[junit4:junit4] 2> 15636 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10058 (1429876695318396928)]} 0 1
[junit4:junit4] 2> 15669 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15670 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49750__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49750_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49750"}
[junit4:junit4] 2> 15674 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15675 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15674 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15675 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15706 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[38 (-1429876695391797248)]} 0 1
[junit4:junit4] 2> 15709 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10038 (-1429876695395991552)]} 0 0
[junit4:junit4] 2> C28_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49750_, base_url=http://127.0.0.1:49750}
[junit4:junit4] 2> 15716 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[38 (-1429876695395991552)]} 0 5
[junit4:junit4] 2> 15717 T849 C30 P49479 /update {wt=javabin&version=2} {delete=[38 (-1429876695395991552)]} 0 8
[junit4:junit4] 2> 15719 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10038 (-1429876695399137280)]} 0 0
[junit4:junit4] 2> 15720 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10038 (-1429876695399137280)]} 0 8
[junit4:junit4] 2> 15721 T827 C29 P44555 /update {wt=javabin&version=2} {add=[59 (1429876695407525888)]} 0 1
[junit4:junit4] 2> 15725 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10059 (1429876695410671616)]} 0 2
[junit4:junit4] 2> 15732 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[59 (1429876695412768768)]} 0 3
[junit4:junit4] 2> 15734 T847 C30 P49479 /update {wt=javabin&version=2} {add=[59 (1429876695412768768)]} 0 10
[junit4:junit4] 2> 15734 T863 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10059 (1429876695415914496)]} 0 1
[junit4:junit4] 2> 15737 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10059 (1429876695415914496)]} 0 9
[junit4:junit4] 2> 15771 T828 C29 P44555 /update {wt=javabin&version=2} {add=[60 (1429876695459954688)]} 0 1
[junit4:junit4] 2> 15772 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10060 (1429876695461003264)]} 0 1
[junit4:junit4] 2> 15778 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[60 (1429876695464148992)]} 0 1
[junit4:junit4] 2> 15779 T849 C30 P49479 /update {wt=javabin&version=2} {add=[60 (1429876695464148992)]} 0 5
[junit4:junit4] 2> 15782 T866 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10060 (1429876695464148993)]} 0 2
[junit4:junit4] 2> 15783 T847 C30 P49479 /update {wt=javabin&version=2} {add=[10060 (1429876695464148993)]} 0 9
[junit4:junit4] 2> 15860 T812 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 15861 T812 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15865 T812 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7ce9fc0a
[junit4:junit4] 2> 15870 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[40 (-1429876695564812288)]} 0 1
[junit4:junit4] 2> 15875 T812 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=4,adds=4,deletesById=2,deletesByQuery=0,errors=0,cumulative_adds=52,cumulative_deletesById=24,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15878 T812 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 15879 T812 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 15879 T812 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 15880 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10040 (-1429876695575298048)]} 0 1
[junit4:junit4] 2> 15883 T865 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 1
[junit4:junit4] 2> 15894 T864 C28 P49750 /update {distrib.from=http://127.0.0.1:49479/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 1
[junit4:junit4] 2> 15894 T864 C28 P49750 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:283)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:456)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1031)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:717)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 15895 T812 C28 P49750 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)),segFN=segments_3,generation=3,filenames=[_0_Lucene40_0.tim, _0.fnm, _0_Lucene40_0.frq, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0_Lucene40_0.prx, _0.si, _0.fdt, segments_3, _0_Lucene40_0.tip]
[junit4:junit4] 2> commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375)),segFN=segments_4,generation=4,filenames=[_1_Lucene40_0.tim, _1_nrm.cfs, _0_Lucene40_0.tim, _0.fnm, _1_Lucene40_0.frq, _0_nrm.cfs, _0_Lucene40_0.prx, _1.fnm, _1_Lucene40_0.prx, _1_Lucene40_0.tip, _1_nrm.cfe, _0_Lucene40_0.tip, _1.fdx, _0_Lucene40_0.frq, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_2.del, _0.fdt, segments_4]
[junit4:junit4] 2> 15896 T812 C28 P49750 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_1_Lucene40_0.tim, _1_nrm.cfs, _0_Lucene40_0.tim, _0.fnm, _1_Lucene40_0.frq, _0_nrm.cfs, _0_Lucene40_0.prx, _1.fnm, _1_Lucene40_0.prx, _1_Lucene40_0.tip, _1_nrm.cfe, _0_Lucene40_0.tip, _1.fdx, _0_Lucene40_0.frq, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_2.del, _0.fdt, segments_4]
[junit4:junit4] 2> 15897 T865 C28 P49750 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:283)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:456)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1031)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:717)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 15898 T865 C28 P49750 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@7ce9fc0a
[junit4:junit4] 2> 15899 T865 C28 P49750 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=4,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=52,cumulative_deletesById=26,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 15900 T848 C30 P49479 oasc.SolrException.log SEVERE shard update error StdNode: http://127.0.0.1:49750/collection1/:org.apache.solr.common.SolrException: Server at http://127.0.0.1:49750/collection1 returned non ok status:503, message:Service Unavailable
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 15900 T865 C28 P49750 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 15901 T848 C30 P49479 oasup.DistributedUpdateProcessor.doFinish try and ask http://127.0.0.1:49750 to recover
[junit4:junit4] 2> 15901 T848 C30 P49479 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 15904 T850 C30 P49479 oasc.SolrException.log SEVERE shard update error StdNode: http://127.0.0.1:49750/collection1/:org.apache.solr.common.SolrException: Server at http://127.0.0.1:49750/collection1 returned non ok status:503, message:Service Unavailable
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 15905 T850 C30 P49479 oasup.DistributedUpdateProcessor.doFinish try and ask http://127.0.0.1:49750 to recover
[junit4:junit4] 2> 15905 T850 C30 P49479 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 15907 T812 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 15908 T817 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7f1438080008
[junit4:junit4] 2> 15909 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15909 T870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15909 T814 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:52422 which had sessionid 0x13d7f1438080008
[junit4:junit4] 2> 15910 T841 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15910 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15910 T812 oaz.ZooKeeper.close Session: 0x13d7f1438080008 closed
[junit4:junit4] 2> 15911 T854 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15911 T866 oasha.CoreAdminHandler.handleRequestRecoveryAction It has been requested that we recover
[junit4:junit4] 2> 15911 T863 oasha.CoreAdminHandler.handleRequestRecoveryAction It has been requested that we recover
[junit4:junit4] 2> 15910 T834 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 15913 T882 oasc.SolrException.log SEVERE Cound not find core to call recovery:collection1
[junit4:junit4] 2> 15913 T863 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {action=REQUESTRECOVERY&core=collection1&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 15913 T883 oasc.SolrException.log SEVERE Cound not find core to call recovery:collection1
[junit4:junit4] 2> 15913 T866 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {action=REQUESTRECOVERY&core=collection1&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 15913 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 15915 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10040 (-1429876695578443776)]} 0 33
[junit4:junit4] 2> 15917 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[40 (-1429876695575298048)]} 0 38
[junit4:junit4] 2> 15919 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10061 (1429876695615143936)]} 0 1
[junit4:junit4] 2> 15921 T828 C29 P44555 /update {wt=javabin&version=2} {add=[61 (1429876695617241088)]} 0 1
[junit4:junit4] 2> 15923 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10061 (1429876695619338240)]} 0 1
[junit4:junit4] 2> 15927 T812 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 15929 T847 C30 P49479 /update {wt=javabin&version=2} {add=[61 (1429876695622483968)]} 0 5
[junit4:junit4] 2> 15985 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10044 (-1429876695685398528)]} 0 0
[junit4:junit4] 2> 15989 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[44 (-1429876695689592832)]} 0 0
[junit4:junit4] 2> 15989 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10044 (-1429876695689592832)]} 0 0
[junit4:junit4] 2> 15992 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[44 (-1429876695692738560)]} 0 0
[junit4:junit4] 2> 15993 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10062 (1429876695692738560)]} 0 1
[junit4:junit4] 2> 15996 T828 C29 P44555 /update {wt=javabin&version=2} {add=[62 (1429876695696932864)]} 0 1
[junit4:junit4] 2> 15997 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10062 (1429876695696932864)]} 0 2
[junit4:junit4] 2> 16001 T847 C30 P49479 /update {wt=javabin&version=2} {add=[62 (1429876695700078592)]} 0 2
[junit4:junit4] 2> 16008 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[10045 (-1429876695709515776)]} 0 0
[junit4:junit4] 2> 16012 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[10045 (-1429876695713710080)]} 0 1
[junit4:junit4] 2> 16012 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[45 (-1429876695713710080)]} 0 0
[junit4:junit4] 2> 16015 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[45 (-1429876695716855808)]} 0 0
[junit4:junit4] 2> 16018 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10063 (1429876695716855808)]} 0 3
[junit4:junit4] 2> 16020 T828 C29 P44555 /update {wt=javabin&version=2} {add=[63 (1429876695721050112)]} 0 2
[junit4:junit4] 2> 16023 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10063 (1429876695724195840)]} 0 2
[junit4:junit4] 2> 16026 T847 C30 P49479 /update {wt=javabin&version=2} {add=[63 (1429876695727341568)]} 0 2
[junit4:junit4] 2> 16101 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10064 (1429876695804936192)]} 0 2
[junit4:junit4] 2> 16103 T827 C29 P44555 /update {wt=javabin&version=2} {add=[64 (1429876695808081920)]} 0 1
[junit4:junit4] 2> 16105 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10064 (1429876695810179072)]} 0 1
[junit4:junit4] 2> 16108 T850 C30 P49479 /update {wt=javabin&version=2} {add=[64 (1429876695813324800)]} 0 1
[junit4:junit4] 2> 16152 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10065 (1429876695859462144)]} 0 2
[junit4:junit4] 2> 16155 T828 C29 P44555 /update {wt=javabin&version=2} {add=[65 (1429876695862607872)]} 0 1
[junit4:junit4] 2> 16155 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10065 (1429876695862607872)]} 0 1
[junit4:junit4] 2> 16159 T848 C30 P49479 /update {wt=javabin&version=2} {add=[65 (1429876695866802176)]} 0 1
[junit4:junit4] 2> 16253 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10066 (1429876695965368320)]} 0 1
[junit4:junit4] 2> 16257 T827 C29 P44555 /update {wt=javabin&version=2} {add=[66 (1429876695969562624)]} 0 2
[junit4:junit4] 2> 16257 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10066 (1429876695969562624)]} 0 1
[junit4:junit4] 2> 16263 T847 C30 P49479 /update {wt=javabin&version=2} {add=[66 (1429876695973756928)]} 0 3
[junit4:junit4] 2> 16346 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10067 (1429876696062885888)]} 0 1
[junit4:junit4] 2> 16351 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10067 (1429876696068128768)]} 0 2
[junit4:junit4] 2> 16354 T828 C29 P44555 /update {wt=javabin&version=2} {add=[67 (1429876696071274496)]} 0 2
[junit4:junit4] 2> 16358 T850 C30 P49479 /update {wt=javabin&version=2} {add=[67 (1429876696075468800)]} 0 1
[junit4:junit4] 2> 16372 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10068 (1429876696090148864)]} 0 2
[junit4:junit4] 2> 16376 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10068 (1429876696094343168)]} 0 1
[junit4:junit4] 2> 16379 T827 C29 P44555 /update {wt=javabin&version=2} {add=[68 (1429876696097488896)]} 0 1
[junit4:junit4] 2> 16384 T847 C30 P49479 /update {wt=javabin&version=2} {add=[68 (1429876696101683200)]} 0 2
[junit4:junit4] 2> 16447 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10069 (1429876696168792064)]} 0 1
[junit4:junit4] 2> 16451 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10069 (1429876696172986368)]} 0 1
[junit4:junit4] 2> 16454 T829 C29 P44555 /update {wt=javabin&version=2} {add=[69 (1429876696176132096)]} 0 1
[junit4:junit4] 2> 16459 T850 C30 P49479 /update {wt=javabin&version=2} {add=[69 (1429876696180326400)]} 0 2
[junit4:junit4] 2> 16494 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10070 (1429876696219123712)]} 0 1
[junit4:junit4] 2> 16499 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10070 (1429876696223318016)]} 0 2
[junit4:junit4] 2> 16502 T828 C29 P44555 /update {wt=javabin&version=2} {add=[70 (1429876696226463744)]} 0 1
[junit4:junit4] 2> 16507 T847 C30 P49479 /update {wt=javabin&version=2} {add=[70 (1429876696230658048)]} 0 2
[junit4:junit4] 2> 16541 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10071 (1429876696267358208)]} 0 1
[junit4:junit4] 2> 16546 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10071 (1429876696272601088)]} 0 2
[junit4:junit4] 2> 16549 T829 C29 P44555 /update {wt=javabin&version=2} {add=[71 (1429876696275746816)]} 0 1
[junit4:junit4] 2> 16553 T850 C30 P49479 /update {wt=javabin&version=2} {add=[71 (1429876696279941120)]} 0 1
[junit4:junit4] 2> 16634 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10072 (1429876696364875776)]} 0 1
[junit4:junit4] 2> 16638 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10072 (1429876696369070080)]} 0 1
[junit4:junit4] 2> 16640 T828 C29 P44555 /update {wt=javabin&version=2} {add=[72 (1429876696372215808)]} 0 0
[junit4:junit4] 2> 16644 T847 C30 P49479 /update {wt=javabin&version=2} {add=[72 (1429876696375361536)]} 0 1
[junit4:junit4] 2> 16669 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10046 (-1429876696401575936)]} 0 1
[junit4:junit4] 2> 16673 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10046 (-1429876696405770240)]} 0 1
[junit4:junit4] 2> 16675 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[46 (-1429876696408915968)]} 0 0
[junit4:junit4] 2> 16676 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10073 (1429876696409964544)]} 0 0
[junit4:junit4] 2> 16678 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[46 (-1429876696412061696)]} 0 0
[junit4:junit4] 2> 16682 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10073 (1429876696415207424)]} 0 1
[junit4:junit4] 2> 16682 T830 C29 P44555 /update {wt=javabin&version=2} {add=[73 (1429876696415207424)]} 0 1
[junit4:junit4] 2> 16686 T849 C30 P49479 /update {wt=javabin&version=2} {add=[73 (1429876696419401728)]} 0 1
[junit4:junit4] 2> 16775 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10048 (-1429876696513773568)]} 0 0
[junit4:junit4] 2> 16778 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10048 (-1429876696516919296)]} 0 0
[junit4:junit4] 2> 16779 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[48 (-1429876696517967872)]} 0 1
[junit4:junit4] 2> 16781 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[48 (-1429876696520065024)]} 0 0
[junit4:junit4] 2> 16781 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10074 (1429876696520065024)]} 0 1
[junit4:junit4] 2> 16785 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10074 (1429876696523210752)]} 0 1
[junit4:junit4] 2> 16785 T830 C29 P44555 /update {wt=javabin&version=2} {add=[74 (1429876696523210752)]} 0 1
[junit4:junit4] 2> 16788 T849 C30 P49479 /update {wt=javabin&version=2} {add=[74 (1429876696526356480)]} 0 1
[junit4:junit4] 2> 16829 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10049 (-1429876696570396672)]} 0 1
[junit4:junit4] 2> 16832 T827 C29 P44555 /update {wt=javabin&version=2} {delete=[49 (-1429876696573542400)]} 0 1
[junit4:junit4] 2> 16832 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10049 (-1429876696573542400)]} 0 1
[junit4:junit4] 2> 16836 T828 C29 P44555 /update {wt=javabin&version=2} {add=[10075 (1429876696576688128)]} 0 1
[junit4:junit4] 2> 16838 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[49 (-1429876696575639552)]} 0 4
[junit4:junit4] 2> 16841 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10075 (1429876696580882432)]} 0 2
[junit4:junit4] 2> 16842 T830 C29 P44555 /update {wt=javabin&version=2} {add=[75 (1429876696582979584)]} 0 1
[junit4:junit4] 2> 16845 T849 C30 P49479 /update {wt=javabin&version=2} {add=[75 (1429876696587173888)]} 0 1
[junit4:junit4] 2> 16894 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10076 (1429876696637505536)]} 0 1
[junit4:junit4] 2> 16900 T828 C29 P44555 /update {wt=javabin&version=2} {add=[76 (1429876696643796992)]} 0 1
[junit4:junit4] 2> 16901 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10076 (1429876696645894144)]} 0 1
[junit4:junit4] 2> 16904 T847 C30 P49479 /update {wt=javabin&version=2} {add=[76 (1429876696647991296)]} 0 1
[junit4:junit4] 2> 16975 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10077 (1429876696723488768)]} 0 1
[junit4:junit4] 2> 16978 T827 C29 P44555 /update {wt=javabin&version=2} {add=[77 (1429876696725585920)]} 0 1
[junit4:junit4] 2> 16979 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10077 (1429876696726634496)]} 0 1
[junit4:junit4] 2> 16982 T849 C30 P49479 /update {wt=javabin&version=2} {add=[77 (1429876696729780224)]} 0 1
[junit4:junit4] 2> 17020 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10050 (-1429876696769626112)]} 0 1
[junit4:junit4] 2> 17022 T828 C29 P44555 /update {wt=javabin&version=2} {delete=[50 (-1429876696772771840)]} 0 0
[junit4:junit4] 2> 17022 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10050 (-1429876696772771840)]} 0 0
[junit4:junit4] 2> 17025 T847 C30 P49479 /update {wt=javabin&version=2} {delete=[50 (-1429876696775917568)]} 0 0
[junit4:junit4] 2> 17026 T830 C29 P44555 /update {wt=javabin&version=2} {add=[10078 (1429876696775917568)]} 0 1
[junit4:junit4] 2> 17031 T827 C29 P44555 /update {wt=javabin&version=2} {add=[78 (1429876696780111872)]} 0 3
[junit4:junit4] 2> 17032 T848 C30 P49479 /update {wt=javabin&version=2} {add=[10078 (1429876696782209024)]} 0 1
[junit4:junit4] 2> 17035 T849 C30 P49479 /update {wt=javabin&version=2} {add=[78 (1429876696785354752)]} 0 1
[junit4:junit4] 2> 17061 T829 C29 P44555 /update {wt=javabin&version=2} {delete=[10052 (-1429876696813666304)]} 0 0
[junit4:junit4] 2> 17065 T830 C29 P44555 /update {wt=javabin&version=2} {delete=[52 (-1429876696816812032)]} 0 1
[junit4:junit4] 2> 17065 T850 C30 P49479 /update {wt=javabin&version=2} {delete=[10052 (-1429876696816812032)]} 0 1
[junit4:junit4] 2> 17068 T848 C30 P49479 /update {wt=javabin&version=2} {delete=[52 (-1429876696821006336)]} 0 1
[junit4:junit4] 2> 17069 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10079 (1429876696821006336)]} 0 2
[junit4:junit4] 2> 17072 T828 C29 P44555 /update {wt=javabin&version=2} {add=[79 (1429876696824152064)]} 0 1
[junit4:junit4] 2> 17072 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10079 (1429876696824152064)]} 0 1
[junit4:junit4] 2> 17077 T847 C30 P49479 /update {wt=javabin&version=2} {add=[79 (1429876696829394944)]} 0 2
[junit4:junit4] 2> 17154 T829 C29 P44555 /update {wt=javabin&version=2} {add=[10080 (1429876696910135296)]} 0 1
[junit4:junit4] 2> 17158 T830 C29 P44555 /update {wt=javabin&version=2} {add=[80 (1429876696914329600)]} 0 1
[junit4:junit4] 2> 17158 T850 C30 P49479 /update {wt=javabin&version=2} {add=[10080 (1429876696914329600)]} 0 1
[junit4:junit4] 2> 17163 T848 C30 P49479 /update {wt=javabin&version=2} {add=[80 (1429876696919572480)]} 0 2
[junit4:junit4] 2> 17179 T835 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 17179 T827 C29 P44555 /update {wt=javabin&version=2} {add=[10081 (1429876696936349696)]} 0 1
[junit4:junit4] 2> 17179 T835 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:49750__collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:49750_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49750"}
[junit4:junit4] 2> 17183 T828 C29 P44555 /update {wt=javabin&version=2} {add=[81 (1429876696940544000)]} 0 1
[junit4:junit4] 2> 17183 T849 C30 P49479 /update {wt=javabin&version=2} {add=[10081 (1429876696940544000)]} 0 1
[junit4:junit4] 2> 17186 T847 C30 P49479 /update {wt=javabin&version=2} {add=[81 (1429876696944738304)]} 0 0
[junit4:junit4] 2> 17191 T854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 17191 T841 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 17191 T834 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 17209 T870 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4
[...truncated too long message...]
scc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 35866 T896 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 35866 T896 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 35866 T896 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_KW, timezone=America/Swift_Current
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.7.0_15 (64-bit)/cpus=8,threads=3,free=84480584,total=145555456
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestLRUCache, TestSearchPerf, OverseerTest, JsonLoaderTest, TestBadConfig, TestCharFilters, DateFieldTest, SuggesterTest, TestReversedWildcardFilterFactory, CSVRequestHandlerTest, DirectSolrConnectionTest, TestQueryUtils, TestSort, OverseerCollectionProcessorTest, MBeansHandlerTest, TestCollationKeyRangeQueries, SpellingQueryConverterTest, TestCoreContainer, TestPseudoReturnFields, PluginInfoTest, TestFastWriter, TestPHPSerializedResponseWriter, PrimitiveFieldTypeTest, SchemaVersionSpecificBehaviorTest, UnloadDistributedZkTest, LegacyHTMLStripCharFilterTest, TestValueSourceCache, SearchHandlerTest, SpatialFilterTest, SystemInfoHandlerTest, ScriptEngineTest, TestWriterPerf, TestPropInjectDefaults, TestCopyFieldCollectionResource, TermsComponentTest, RecoveryZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=7F2E5EEA9B9C5DF3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=America/Swift_Current -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=871, name=coreLoadExecutor-448-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:980)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@65a23497 lockFactory=org.apache.lucene.store.NativeFSLockFactory@73034b6b; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375))
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > ... 1 more
[junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=873, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:980)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4d6f119e lockFactory=org.apache.lucene.store.NativeFSLockFactory@387df920; maxCacheMB=0.23046875 maxMergeSizeMB=0.55859375))
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] Completed on J0 in 35.97s, 1 test, 2 errors <<< FAILURES!
[...truncated 572 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 2 suite-level errors, 13 ignored (7 assumptions)
Total time: 59 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.7.0_15 -XX:+UseConcMarkSweepGC
Email was triggered for: Failure
Sending email for trigger: Failure