You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/03/17 12:22:27 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.6.0_41) - Build #
4745 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4745/
Java: 32bit/jdk1.6.0_41 -server -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1090, name=coreLoadExecutor-563-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1090, name=coreLoadExecutor-563-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1479)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:284)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:236)
at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@1159c89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f2d2d3)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
... 1 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest
Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1092, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1092, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
at java.lang.Thread.getStackTrace(Thread.java:1479)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:284)
at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:236)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 8563 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] 2> 0 T1031 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /mah/nf
[junit4:junit4] 2> 3 T1031 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363518270255
[junit4:junit4] 2> 4 T1031 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 4 T1032 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 5 T1032 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4] 2> 5 T1032 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4] 2> 5 T1032 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4] 2> 5 T1032 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4] 2> 6 T1032 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363518270254/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4] 2> 104 T1031 oasc.ZkTestServer.run start zk server on port:55564
[junit4:junit4] 2> 105 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a68676
[junit4:junit4] 2> 106 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 106 T1037 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 107 T1037 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 107 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41321
[junit4:junit4] 2> 107 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41321
[junit4:junit4] 2> 107 T1035 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4] 2> 482 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310000 with negotiated timeout 10000 for client /127.0.0.1:41321
[junit4:junit4] 2> 482 T1037 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310000, negotiated timeout = 10000
[junit4:junit4] 2> 483 T1038 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a68676 name:ZooKeeperConnection Watcher:127.0.0.1:55564 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 483 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 483 T1031 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 491 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310000
[junit4:junit4] 2> 491 T1038 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 492 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41321 which had sessionid 0x13d7805ab310000
[junit4:junit4] 2> 491 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310000 closed
[junit4:junit4] 2> 492 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@a7bc81
[junit4:junit4] 2> 493 T1039 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 493 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 493 T1039 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 494 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41322
[junit4:junit4] 2> 494 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41322
[junit4:junit4] 2> 494 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310001 with negotiated timeout 10000 for client /127.0.0.1:41322
[junit4:junit4] 2> 494 T1039 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310001, negotiated timeout = 10000
[junit4:junit4] 2> 495 T1040 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a7bc81 name:ZooKeeperConnection Watcher:127.0.0.1:55564/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 495 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 495 T1031 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 500 T1031 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 503 T1031 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 505 T1031 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 507 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 507 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 520 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 521 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 526 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 527 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 529 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 530 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 532 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 532 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 534 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 535 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 537 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 537 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 540 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 540 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 545 T1031 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 546 T1031 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 549 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310001
[junit4:junit4] 2> 549 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41322 which had sessionid 0x13d7805ab310001
[junit4:junit4] 2> 549 T1040 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 549 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310001 closed
[junit4:junit4] 2> 630 T1031 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 636 T1031 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44725
[junit4:junit4] 2> 637 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 637 T1031 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 637 T1031 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801
[junit4:junit4] 2> 638 T1031 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/solr.xml
[junit4:junit4] 2> 638 T1031 oasc.CoreContainer.<init> New CoreContainer 17443015
[junit4:junit4] 2> 638 T1031 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/'
[junit4:junit4] 2> 638 T1031 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/'
[junit4:junit4] 2> 656 T1031 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 656 T1031 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 656 T1031 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 657 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 657 T1031 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 657 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 658 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 658 T1031 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 658 T1031 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 659 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 665 T1031 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 681 T1031 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:55564/solr
[junit4:junit4] 2> 682 T1031 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 682 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6b597e
[junit4:junit4] 2> 683 T1050 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 684 T1050 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 684 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 684 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41323
[junit4:junit4] 2> 685 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41323
[junit4:junit4] 2> 686 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310002 with negotiated timeout 20000 for client /127.0.0.1:41323
[junit4:junit4] 2> 686 T1050 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310002, negotiated timeout = 20000
[junit4:junit4] 2> 687 T1051 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b597e name:ZooKeeperConnection Watcher:127.0.0.1:55564 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 687 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 688 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310002
[junit4:junit4] 2> 689 T1051 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 689 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41323 which had sessionid 0x13d7805ab310002
[junit4:junit4] 2> 689 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310002 closed
[junit4:junit4] 2> 690 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 694 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1e9b5f1
[junit4:junit4] 2> 695 T1052 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 695 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 695 T1052 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 695 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41324
[junit4:junit4] 2> 696 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41324
[junit4:junit4] 2> 696 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310003 with negotiated timeout 20000 for client /127.0.0.1:41324
[junit4:junit4] 2> 696 T1052 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310003, negotiated timeout = 20000
[junit4:junit4] 2> 697 T1053 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e9b5f1 name:ZooKeeperConnection Watcher:127.0.0.1:55564/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 697 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 699 T1031 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 701 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x7 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 702 T1031 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 704 T1031 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 706 T1031 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44725_mah%2Fnf
[junit4:junit4] 2> 706 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:delete cxid:0x12 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:44725_mah%2Fnf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:44725_mah%2Fnf
[junit4:junit4] 2> 707 T1031 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44725_mah%2Fnf
[junit4:junit4] 2> 711 T1031 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 720 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:delete cxid:0x20 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4] 2> 721 T1031 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 723 T1031 oasc.Overseer.start Overseer (id=89359533359562755-127.0.0.1:44725_mah%2Fnf-n_0000000000) starting
[junit4:junit4] 2> 724 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x25 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 725 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x26 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 726 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x28 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 726 T1031 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 729 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x2e zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 730 T1055 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 730 T1031 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 733 T1031 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 734 T1031 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 736 T1054 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 739 T1056 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/collection1
[junit4:junit4] 2> 740 T1056 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 740 T1056 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 740 T1056 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 741 T1056 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/collection1/'
[junit4:junit4] 2> 741 T1056 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/collection1/lib/README' to classloader
[junit4:junit4] 2> 742 T1056 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 775 T1056 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 836 T1056 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 837 T1056 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 841 T1056 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 1329 T1056 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 1336 T1056 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 1342 T1056 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 1353 T1056 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1356 T1056 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 1359 T1056 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1360 T1056 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1361 T1056 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 1361 T1056 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 1362 T1056 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 1362 T1056 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 1362 T1056 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1363 T1056 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363518270801/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data/
[junit4:junit4] 2> 1363 T1056 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f0b7d3
[junit4:junit4] 2> 1363 T1056 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 1363 T1056 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data forceNew: false
[junit4:junit4] 2> 1364 T1056 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data/index/
[junit4:junit4] 2> 1364 T1056 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 1364 T1056 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data/index forceNew: false
[junit4:junit4] 2> 1365 T1056 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e4ebbb lockFactory=org.apache.lucene.store.NativeFSLockFactory@14c9b76),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 1365 T1056 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 1367 T1056 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 1367 T1056 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 1367 T1056 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 1368 T1056 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 1368 T1056 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 1368 T1056 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 1368 T1056 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 1369 T1056 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 1369 T1056 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 1371 T1056 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 1373 T1056 oass.SolrIndexSearcher.<init> Opening Searcher@41e20a main
[junit4:junit4] 2> 1373 T1056 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/control/data/tlog
[junit4:junit4] 2> 1373 T1056 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 1374 T1056 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 1376 T1057 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@41e20a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 1377 T1056 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 1377 T1056 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 2239 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 2240 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44725_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44725/mah/nf"}
[junit4:junit4] 2> 2241 T1054 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 2241 T1054 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 2300 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 2378 T1056 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 2379 T1056 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44725/mah/nf collection:control_collection shard:shard1
[junit4:junit4] 2> 2380 T1056 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 2460 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:delete cxid:0x82 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4] 2> 2480 T1056 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 2481 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x83 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 2520 T1056 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 2521 T1056 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 2521 T1056 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44725/mah/nf/collection1/
[junit4:junit4] 2> 2521 T1056 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 2521 T1056 oasc.SyncStrategy.syncToMe http://127.0.0.1:44725/mah/nf/collection1/ has no replicas
[junit4:junit4] 2> 2521 T1056 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44725/mah/nf/collection1/
[junit4:junit4] 2> 2521 T1056 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 2561 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310003 type:create cxid:0x8e zxid:0x3a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 3823 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4487 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 4505 T1056 oasc.ZkController.register We are http://127.0.0.1:44725/mah/nf/collection1/ and leader is http://127.0.0.1:44725/mah/nf/collection1/
[junit4:junit4] 2> 4505 T1056 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44725/mah/nf
[junit4:junit4] 2> 4506 T1056 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 4506 T1056 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 4506 T1056 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 4508 T1056 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 4508 T1031 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 4509 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 4509 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 4513 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 4514 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@12c9689
[junit4:junit4] 2> 4515 T1059 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 4515 T1059 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 4516 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41325
[junit4:junit4] 2> 4517 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 4517 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41325
[junit4:junit4] 2> 4518 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310004 with negotiated timeout 10000 for client /127.0.0.1:41325
[junit4:junit4] 2> 4518 T1059 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310004, negotiated timeout = 10000
[junit4:junit4] 2> 4518 T1060 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12c9689 name:ZooKeeperConnection Watcher:127.0.0.1:55564/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 4518 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 4520 T1031 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 4521 T1031 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 4590 T1031 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 4593 T1031 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59596
[junit4:junit4] 2> 4594 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 4594 T1031 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 4595 T1031 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772
[junit4:junit4] 2> 4595 T1031 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/solr.xml
[junit4:junit4] 2> 4595 T1031 oasc.CoreContainer.<init> New CoreContainer 19048294
[junit4:junit4] 2> 4595 T1031 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/'
[junit4:junit4] 2> 4596 T1031 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/'
[junit4:junit4] 2> 4612 T1031 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 4612 T1031 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 4613 T1031 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 4613 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 4613 T1031 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 4614 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 4614 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 4614 T1031 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 4614 T1031 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 4615 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 4621 T1031 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 4641 T1031 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:55564/solr
[junit4:junit4] 2> 4642 T1031 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 4643 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@448950
[junit4:junit4] 2> 4644 T1070 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 4644 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 4645 T1070 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 4645 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41326
[junit4:junit4] 2> 4645 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41326
[junit4:junit4] 2> 4646 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310005 with negotiated timeout 20000 for client /127.0.0.1:41326
[junit4:junit4] 2> 4646 T1070 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310005, negotiated timeout = 20000
[junit4:junit4] 2> 4647 T1071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@448950 name:ZooKeeperConnection Watcher:127.0.0.1:55564 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 4647 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 4648 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310005
[junit4:junit4] 2> 4649 T1071 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 4649 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41326 which had sessionid 0x13d7805ab310005
[junit4:junit4] 2> 4649 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310005 closed
[junit4:junit4] 2> 4649 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 4653 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1877352
[junit4:junit4] 2> 4655 T1072 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 4654 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 4655 T1072 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 4655 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41327
[junit4:junit4] 2> 4655 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41327
[junit4:junit4] 2> 4656 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310006 with negotiated timeout 20000 for client /127.0.0.1:41327
[junit4:junit4] 2> 4656 T1072 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310006, negotiated timeout = 20000
[junit4:junit4] 2> 4656 T1073 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1877352 name:ZooKeeperConnection Watcher:127.0.0.1:55564/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 4656 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 4657 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:create cxid:0x1 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 4658 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:create cxid:0x3 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 4660 T1031 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 5671 T1031 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59596_mah%2Fnf
[junit4:junit4] 2> 5671 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:delete cxid:0xd zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:59596_mah%2Fnf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:59596_mah%2Fnf
[junit4:junit4] 2> 5672 T1031 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59596_mah%2Fnf
[junit4:junit4] 2> 5674 T1060 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 5675 T1073 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 5675 T1053 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 5675 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5679 T1074 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/collection1
[junit4:junit4] 2> 5679 T1074 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 5680 T1074 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 5680 T1074 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 5681 T1074 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/collection1/'
[junit4:junit4] 2> 5681 T1074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/collection1/lib/README' to classloader
[junit4:junit4] 2> 5682 T1074 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 5716 T1074 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 5751 T1074 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 5752 T1074 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 5756 T1074 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 5992 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 5992 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:44725_mah%2Fnf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "node_name":"127.0.0.1:44725_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44725/mah/nf"}
[junit4:junit4] 2> 5995 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5995 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 5995 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 6198 T1074 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 6204 T1074 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 6209 T1074 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 6224 T1074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6228 T1074 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 6231 T1074 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6232 T1074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6232 T1074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 6233 T1074 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 6234 T1074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 6234 T1074 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 6234 T1074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 6234 T1074 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363518274772/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1/
[junit4:junit4] 2> 6235 T1074 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f0b7d3
[junit4:junit4] 2> 6235 T1074 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 6236 T1074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1 forceNew: false
[junit4:junit4] 2> 6236 T1074 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1/index/
[junit4:junit4] 2> 6236 T1074 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 6237 T1074 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1/index forceNew: false
[junit4:junit4] 2> 6238 T1074 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49a82c lockFactory=org.apache.lucene.store.NativeFSLockFactory@15d5a96),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 6238 T1074 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 6241 T1074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 6241 T1074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 6242 T1074 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 6243 T1074 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 6243 T1074 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 6243 T1074 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 6243 T1074 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 6244 T1074 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 6244 T1074 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 6247 T1074 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 6251 T1074 oass.SolrIndexSearcher.<init> Opening Searcher@15f4f58 main
[junit4:junit4] 2> 6252 T1074 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty1/tlog
[junit4:junit4] 2> 6252 T1074 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 6253 T1074 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 6259 T1075 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15f4f58 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 6261 T1074 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 6261 T1074 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 7499 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 7500 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59596_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59596/mah/nf"}
[junit4:junit4] 2> 7500 T1054 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4] 2> 7500 T1054 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 7573 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7573 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 7574 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 8263 T1074 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 8263 T1074 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59596/mah/nf collection:collection1 shard:shard1
[junit4:junit4] 2> 8264 T1074 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 8271 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:delete cxid:0x4a zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4] 2> 8271 T1074 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 8272 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:create cxid:0x4b zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 8274 T1074 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 8274 T1074 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 8275 T1074 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59596/mah/nf/collection1/
[junit4:junit4] 2> 8275 T1074 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 8275 T1074 oasc.SyncStrategy.syncToMe http://127.0.0.1:59596/mah/nf/collection1/ has no replicas
[junit4:junit4] 2> 8275 T1074 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59596/mah/nf/collection1/
[junit4:junit4] 2> 8275 T1074 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 8281 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310006 type:create cxid:0x56 zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9077 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9085 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9085 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9085 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 9136 T1074 oasc.ZkController.register We are http://127.0.0.1:59596/mah/nf/collection1/ and leader is http://127.0.0.1:59596/mah/nf/collection1/
[junit4:junit4] 2> 9136 T1074 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59596/mah/nf
[junit4:junit4] 2> 9137 T1074 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 9137 T1074 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 9138 T1074 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 9141 T1074 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 9143 T1031 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 9144 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 9145 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 9217 T1031 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4] 2> 9219 T1031 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53624
[junit4:junit4] 2> 9219 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 9220 T1031 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 9220 T1031 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402
[junit4:junit4] 2> 9220 T1031 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/solr.xml
[junit4:junit4] 2> 9221 T1031 oasc.CoreContainer.<init> New CoreContainer 28536120
[junit4:junit4] 2> 9221 T1031 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/'
[junit4:junit4] 2> 9221 T1031 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/'
[junit4:junit4] 2> 9237 T1031 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 9238 T1031 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 9238 T1031 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 9238 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 9239 T1031 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 9239 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 9239 T1031 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 9239 T1031 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 9240 T1031 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 9240 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 9246 T1031 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4] 2> 9262 T1031 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:55564/solr
[junit4:junit4] 2> 9262 T1031 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 9263 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a8689b
[junit4:junit4] 2> 9263 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9264 T1086 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 9264 T1086 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 9264 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41328
[junit4:junit4] 2> 9265 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41328
[junit4:junit4] 2> 9265 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310007 with negotiated timeout 20000 for client /127.0.0.1:41328
[junit4:junit4] 2> 9265 T1086 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310007, negotiated timeout = 20000
[junit4:junit4] 2> 9266 T1087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a8689b name:ZooKeeperConnection Watcher:127.0.0.1:55564 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9266 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9267 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310007
[junit4:junit4] 2> 9267 T1087 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 9267 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41328 which had sessionid 0x13d7805ab310007
[junit4:junit4] 2> 9267 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310007 closed
[junit4:junit4] 2> 9268 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 9271 T1031 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:55564/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@e26fc2
[junit4:junit4] 2> 9271 T1088 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:55564. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4] 2> 9271 T1031 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 9272 T1088 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:55564, initiating session
[junit4:junit4] 2> 9272 T1033 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:41329
[junit4:junit4] 2> 9272 T1033 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:41329
[junit4:junit4] 2> 9273 T1035 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d7805ab310008 with negotiated timeout 20000 for client /127.0.0.1:41329
[junit4:junit4] 2> 9273 T1088 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:55564, sessionid = 0x13d7805ab310008, negotiated timeout = 20000
[junit4:junit4] 2> 9273 T1089 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e26fc2 name:ZooKeeperConnection Watcher:127.0.0.1:55564/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 9273 T1031 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 9274 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9276 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310008 type:create cxid:0x3 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4] 2> 9277 T1031 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 10280 T1031 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53624_mah%2Fnf
[junit4:junit4] 2> 10281 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d7805ab310008 type:delete cxid:0xd zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53624_mah%2Fnf Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53624_mah%2Fnf
[junit4:junit4] 2> 10281 T1031 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53624_mah%2Fnf
[junit4:junit4] 2> 10284 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 10284 T1053 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 10284 T1060 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 10284 T1089 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 10284 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10284 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10286 T1073 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 10289 T1090 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/collection1
[junit4:junit4] 2> 10289 T1090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 10290 T1090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 10290 T1090 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 10291 T1090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/collection1/'
[junit4:junit4] 2> 10292 T1090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/collection1/lib/README' to classloader
[junit4:junit4] 2> 10292 T1090 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 10322 T1090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4] 2> 10357 T1090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 10358 T1090 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4] 2> 10362 T1090 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4] 2> 10590 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 10591 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:59596_mah%2Fnf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:59596_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59596/mah/nf"}
[junit4:junit4] 2> 10594 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10594 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10594 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10594 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 10775 T1090 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 10782 T1090 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 10787 T1090 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 10798 T1090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10800 T1090 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 10803 T1090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 10804 T1090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 10805 T1090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 10805 T1090 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 10806 T1090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 10806 T1090 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4] 2> 10806 T1090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 10806 T1090 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363518279402/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/
[junit4:junit4] 2> 10807 T1090 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f0b7d3
[junit4:junit4] 2> 10807 T1090 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4] 2> 10807 T1090 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2 forceNew: false
[junit4:junit4] 2> 10808 T1090 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index/
[junit4:junit4] 2> 10808 T1090 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 10808 T1090 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index forceNew: false
[junit4:junit4] 2> 10809 T1090 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d6854d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1139040),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 10809 T1090 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 10811 T1090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 10811 T1090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 10812 T1090 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 10812 T1090 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 10812 T1090 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 10813 T1090 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 10813 T1090 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 10813 T1090 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 10813 T1090 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 10815 T1090 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 10817 T1090 oass.SolrIndexSearcher.<init> Opening Searcher@ae0d89 main
[junit4:junit4] 2> 10818 T1090 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/tlog
[junit4:junit4] 2> 10818 T1090 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 10818 T1090 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 10822 T1091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ae0d89 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 10823 T1090 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 10823 T1090 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12098 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12099 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":null,
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53624_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53624/mah/nf"}
[junit4:junit4] 2> 12099 T1054 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4] 2> 12099 T1054 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4] 2> 12104 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12104 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12104 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12104 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 12825 T1090 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 12825 T1090 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53624/mah/nf collection:collection1 shard:shard1
[junit4:junit4] 2> 12827 T1090 oasc.ZkController.register We are http://127.0.0.1:53624/mah/nf/collection1/ and leader is http://127.0.0.1:59596/mah/nf/collection1/
[junit4:junit4] 2> 12828 T1090 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53624/mah/nf
[junit4:junit4] 2> 12828 T1090 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 12828 T1090 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C60 name=collection1 org.apache.solr.core.SolrCore@1677c5c url=http://127.0.0.1:53624/mah/nf/collection1 node=127.0.0.1:53624_mah%2Fnf C60_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53624_mah%2Fnf, base_url=http://127.0.0.1:53624/mah/nf}
[junit4:junit4] 2> 12828 T1092 C60 P53624 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 12829 T1090 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 12829 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 12829 T1092 C60 P53624 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 12829 T1092 C60 P53624 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 12830 T1031 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 12830 T1092 C60 P53624 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12830 T1031 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 12831 T1031 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 12835 T1031 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C61 name=collection1 org.apache.solr.core.SolrCore@10581f2 url=http://127.0.0.1:44725/mah/nf/collection1 node=127.0.0.1:44725_mah%2Fnf C61_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:44725_mah%2Fnf, base_url=http://127.0.0.1:44725/mah/nf, leader=true}
[junit4:junit4] 2> 12846 T1047 C61 P44725 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e4ebbb lockFactory=org.apache.lucene.store.NativeFSLockFactory@14c9b76),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12847 T1047 C61 P44725 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12849 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[1 (1429752547215671297)]} 0 4
[junit4:junit4] 2> 12849 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10001 (1429752547215671296)]} 0 4
[junit4:junit4] 2> ASYNC NEW_CORE C62 name=collection1 org.apache.solr.core.SolrCore@e6b3d9 url=http://127.0.0.1:59596/mah/nf/collection1 node=127.0.0.1:59596_mah%2Fnf C62_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:59596_mah%2Fnf, base_url=http://127.0.0.1:59596/mah/nf, leader=true}
[junit4:junit4] 2> 12858 T1067 C62 P59596 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49a82c lockFactory=org.apache.lucene.store.NativeFSLockFactory@15d5a96),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 12858 T1067 C62 P59596 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 12860 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10001 (1429752547228254208)]} 0 3
[junit4:junit4] 2> 12861 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[1 (1429752547229302784)]} 0 3
[junit4:junit4] 2> 12899 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10002 (1429752547271245824)]} 0 1
[junit4:junit4] 2> 12902 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[2 (1429752547275440128)]} 0 0
[junit4:junit4] 2> 12903 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10002 (1429752547275440128)]} 0 1
[junit4:junit4] 2> 12906 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[2 (1429752547278585856)]} 0 1
[junit4:junit4] 2> 12951 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10002 (-1429752547326820352)]} 0 0
[junit4:junit4] 2> 12953 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[2 (-1429752547328917504)]} 0 0
[junit4:junit4] 2> 12954 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10002 (-1429752547329966080)]} 0 0
[junit4:junit4] 2> 12956 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[2 (-1429752547332063232)]} 0 0
[junit4:junit4] 2> 12958 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10003 (1429752547333111808)]} 0 1
[junit4:junit4] 2> 12960 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[3 (1429752547335208960)]} 0 1
[junit4:junit4] 2> 12961 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10003 (1429752547336257536)]} 0 1
[junit4:junit4] 2> 12963 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[3 (1429752547338354688)]} 0 1
[junit4:junit4] 2> 12983 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10004 (1429752547359326208)]} 0 1
[junit4:junit4] 2> 12984 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[4 (1429752547360374784)]} 0 1
[junit4:junit4] 2> 12987 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[4 (1429752547363520512)]} 0 1
[junit4:junit4] 2> 12987 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10004 (1429752547363520513)]} 0 1
[junit4:junit4] 2> 13082 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[5 (1429752547464183808)]} 0 1
[junit4:junit4] 2> 13083 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10005 (1429752547464183809)]} 0 2
[junit4:junit4] 2> 13086 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[5 (1429752547467329536)]} 0 1
[junit4:junit4] 2> 13087 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10005 (1429752547469426688)]} 0 1
[junit4:junit4] 2> 13153 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[4 (-1429752547538632704)]} 0 0
[junit4:junit4] 2> 13155 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10004 (-1429752547539681280)]} 0 1
[junit4:junit4] 2> 13156 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[4 (-1429752547541778432)]} 0 0
[junit4:junit4] 2> 13157 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[10004 (-1429752547542827008)]} 0 0
[junit4:junit4] 2> 13159 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[6 (1429752547543875584)]} 0 1
[junit4:junit4] 2> 13160 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10006 (1429752547545972736)]} 0 1
[junit4:junit4] 2> 13161 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[6 (1429752547547021312)]} 0 0
[junit4:junit4] 2> 13163 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10006 (1429752547548069888)]} 0 1
[junit4:junit4] 2> 13189 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[6 (-1429752547576381440)]} 0 0
[junit4:junit4] 2> 13190 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10006 (-1429752547577430016)]} 0 0
[junit4:junit4] 2> 13192 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[6 (-1429752547579527168)]} 0 0
[junit4:junit4] 2> 13193 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10006 (-1429752547580575744)]} 0 1
[junit4:junit4] 2> 13195 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[7 (1429752547581624320)]} 0 1
[junit4:junit4] 2> 13195 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10007 (1429752547582672896)]} 0 0
[junit4:junit4] 2> 13198 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[7 (1429752547584770048)]} 0 1
[junit4:junit4] 2> 13198 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10007 (1429752547585818624)]} 0 1
[junit4:junit4] 2> 13227 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10008 (1429752547615178753)]} 0 1
[junit4:junit4] 2> 13227 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[8 (1429752547615178752)]} 0 1
[junit4:junit4] 2> 13230 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10008 (1429752547619373057)]} 0 1
[junit4:junit4] 2> 13230 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[8 (1429752547619373056)]} 0 1
[junit4:junit4] 2> 13324 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10008 (-1429752547717939200)]} 0 0
[junit4:junit4] 2> 13324 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[8 (-1429752547717939201)]} 0 0
[junit4:junit4] 2> 13327 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[8 (-1429752547721084928)]} 0 1
[junit4:junit4] 2> 13327 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10008 (-1429752547721084929)]} 0 0
[junit4:junit4] 2> 13330 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[9 (1429752547723182080)]} 0 1
[junit4:junit4] 2> 13330 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10009 (1429752547723182081)]} 0 1
[junit4:junit4] 2> 13332 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[9 (1429752547726327808)]} 0 0
[junit4:junit4] 2> 13333 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10009 (1429752547726327809)]} 0 1
[junit4:junit4] 2> 13395 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10 (1429752547791339520)]} 0 1
[junit4:junit4] 2> 13397 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10010 (1429752547793436672)]} 0 1
[junit4:junit4] 2> 13398 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10 (1429752547795533824)]} 0 0
[junit4:junit4] 2> 13400 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10010 (1429752547796582400)]} 0 1
[junit4:junit4] 2> 13402 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[9 (-1429752547799728128)]} 0 0
[junit4:junit4] 2> 13403 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[10009 (-1429752547800776704)]} 0 0
[junit4:junit4] 2> 13404 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[9 (-1429752547801825280)]} 0 0
[junit4:junit4] 2> 13405 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10009 (-1429752547802873856)]} 0 0
[junit4:junit4] 2> 13407 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[11 (1429752547804971008)]} 0 1
[junit4:junit4] 2> 13408 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10011 (1429752547806019584)]} 0 0
[junit4:junit4] 2> 13410 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[11 (1429752547808116736)]} 0 0
[junit4:junit4] 2> 13411 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10011 (1429752547808116737)]} 0 1
[junit4:junit4] 2> 13488 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10012 (1429752547888857088)]} 0 1
[junit4:junit4] 2> 13489 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[12 (1429752547888857089)]} 0 2
[junit4:junit4] 2> 13492 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10012 (1429752547893051392)]} 0 1
[junit4:junit4] 2> 13492 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[12 (1429752547893051393)]} 0 1
[junit4:junit4] 2> 13558 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10013 (1429752547963305984)]} 0 1
[junit4:junit4] 2> 13559 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[13 (1429752547963305985)]} 0 2
[junit4:junit4] 2> 13561 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10013 (1429752547966451712)]} 0 0
[junit4:junit4] 2> 13562 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[13 (1429752547966451713)]} 0 1
[junit4:junit4] 2> 13608 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 13608 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53624_mah%2Fnf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53624_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53624/mah/nf"}
[junit4:junit4] 2> 13613 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13613 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13613 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13613 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 13662 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[14 (1429752548071309312)]} 0 1
[junit4:junit4] 2> 13662 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10014 (1429752548071309313)]} 0 1
[junit4:junit4] 2> C60_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:53624_mah%2Fnf, base_url=http://127.0.0.1:53624/mah/nf}
[junit4:junit4] 2> 13677 T1084 C60 P53624 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d6854d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1139040),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> 13677 T1084 C60 P53624 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4] 2> 13679 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[14 (1429752548078649345)]} 0 4
[junit4:junit4] 2> 13679 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10014 (1429752548078649344)]} 0 4
[junit4:junit4] 2> 13680 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10014 (1429752548078649344)]} 0 13
[junit4:junit4] 2> 13680 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[14 (1429752548078649345)]} 0 13
[junit4:junit4] 2> 13760 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10010 (-1429752548175118336)]} 0 0
[junit4:junit4] 2> 13760 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10 (-1429752548175118337)]} 0 0
[junit4:junit4] 2> 13766 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10 (-1429752548177215489)]} 0 0
[junit4:junit4] 2> 13766 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10010 (-1429752548177215488)]} 0 0
[junit4:junit4] 2> 13767 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10010 (-1429752548177215488)]} 0 5
[junit4:junit4] 2> 13767 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10 (-1429752548177215489)]} 0 5
[junit4:junit4] 2> 13770 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[15 (1429752548185604096)]} 0 1
[junit4:junit4] 2> 13770 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10015 (1429752548185604097)]} 0 1
[junit4:junit4] 2> 13775 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10015 (1429752548187701248)]} 0 1
[junit4:junit4] 2> 13776 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1429752548187701249)]} 0 1
[junit4:junit4] 2> 13776 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10015 (1429752548187701248)]} 0 4
[junit4:junit4] 2> 13776 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[15 (1429752548187701249)]} 0 4
[junit4:junit4] 2> 13836 T1069 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:53624_mah%252Fnf_collection1&state=recovering&nodeName=127.0.0.1:53624_mah%252Fnf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4] 2> 13847 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10016 (1429752548266344448)]} 0 1
[junit4:junit4] 2> 13848 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[16 (1429752548266344449)]} 0 1
[junit4:junit4] 2> 13853 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1429752548269490176)]} 0 1
[junit4:junit4] 2> 13853 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1429752548269490177)]} 0 1
[junit4:junit4] 2> 13854 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10016 (1429752548269490176)]} 0 4
[junit4:junit4] 2> 13854 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[16 (1429752548269490177)]} 0 4
[junit4:junit4] 2> 13909 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[14 (-1429752548330307584)]} 0 1
[junit4:junit4] 2> 13909 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[10014 (-1429752548330307585)]} 0 1
[junit4:junit4] 2> 13914 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[14 (-1429752548333453312)]} 0 1
[junit4:junit4] 2> 13914 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10014 (-1429752548333453313)]} 0 1
[junit4:junit4] 2> 13915 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[14 (-1429752548333453312)]} 0 4
[junit4:junit4] 2> 13915 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10014 (-1429752548333453313)]} 0 4
[junit4:junit4] 2> 13921 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10017 (1429752548342890497)]} 0 1
[junit4:junit4] 2> 13921 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[17 (1429752548342890496)]} 0 1
[junit4:junit4] 2> 13927 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1429752548347084801)]} 0 1
[junit4:junit4] 2> 13928 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1429752548347084800)]} 0 1
[junit4:junit4] 2> 13928 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10017 (1429752548347084800)]} 0 4
[junit4:junit4] 2> 13928 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[17 (1429752548347084801)]} 0 4
[junit4:junit4] 2> 13934 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10015 (-1429752548357570561)]} 0 0
[junit4:junit4] 2> 13934 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[15 (-1429752548357570560)]} 0 0
[junit4:junit4] 2> 13939 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10015 (-1429752548360716289)]} 0 1
[junit4:junit4] 2> 13939 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[15 (-1429752548360716288)]} 0 1
[junit4:junit4] 2> 13939 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10015 (-1429752548360716289)]} 0 2
[junit4:junit4] 2> 13939 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[15 (-1429752548360716288)]} 0 2
[junit4:junit4] 2> 13943 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10018 (1429752548365959168)]} 0 1
[junit4:junit4] 2> 13943 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[18 (1429752548365959169)]} 0 1
[junit4:junit4] 2> 13949 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1429752548370153473)]} 0 1
[junit4:junit4] 2> 13950 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1429752548370153472)]} 0 2
[junit4:junit4] 2> 13950 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[18 (1429752548370153472)]} 0 5
[junit4:junit4] 2> 13950 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10018 (1429752548370153473)]} 0 5
[junit4:junit4] 2> 13974 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10018 (-1429752548399513601)]} 0 0
[junit4:junit4] 2> 13974 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[18 (-1429752548399513600)]} 0 0
[junit4:junit4] 2> 13980 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1429752548402659329)]} 0 1
[junit4:junit4] 2> 13980 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1429752548402659328)]} 0 0
[junit4:junit4] 2> 13981 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10018 (-1429752548402659329)]} 0 4
[junit4:junit4] 2> 13981 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[18 (-1429752548402659328)]} 0 4
[junit4:junit4] 2> 13984 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10019 (1429752548408950784)]} 0 1
[junit4:junit4] 2> 13984 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[19 (1429752548408950785)]} 0 1
[junit4:junit4] 2> 13989 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1429752548412096512)]} 0 1
[junit4:junit4] 2> 13990 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1429752548412096513)]} 0 1
[junit4:junit4] 2> 13990 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10019 (1429752548412096512)]} 0 4
[junit4:junit4] 2> 13990 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[19 (1429752548412096513)]} 0 4
[junit4:junit4] 2> 14067 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10020 (1429752548495982592)]} 0 1
[junit4:junit4] 2> 14067 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[20 (1429752548495982593)]} 0 1
[junit4:junit4] 2> 14072 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1429752548499128321)]} 0 1
[junit4:junit4] 2> 14072 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1429752548499128320)]} 0 0
[junit4:junit4] 2> 14073 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[20 (1429752548499128320)]} 0 4
[junit4:junit4] 2> 14078 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10020 (1429752548499128321)]} 0 9
[junit4:junit4] 2> 14100 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[21 (1429752548531634176)]} 0 0
[junit4:junit4] 2> 14105 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10021 (1429752548535828480)]} 0 1
[junit4:junit4] 2> 14105 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1429752548534779904)]} 0 0
[junit4:junit4] 2> 14106 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[21 (1429752548534779904)]} 0 3
[junit4:junit4] 2> 14110 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1429752548538974208)]} 0 1
[junit4:junit4] 2> 14110 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10021 (1429752548538974208)]} 0 3
[junit4:junit4] 2> 14136 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[22 (1429752548569382912)]} 0 0
[junit4:junit4] 2> 14140 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10022 (1429752548572528640)]} 0 1
[junit4:junit4] 2> 14142 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1429752548572528640)]} 0 1
[junit4:junit4] 2> 14142 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[22 (1429752548572528640)]} 0 3
[junit4:junit4] 2> 14145 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1429752548575674368)]} 0 0
[junit4:junit4] 2> 14146 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10022 (1429752548575674368)]} 0 4
[junit4:junit4] 2> 14228 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[23 (1429752548664803328)]} 0 1
[junit4:junit4] 2> 14231 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10023 (1429752548668997632)]} 0 1
[junit4:junit4] 2> 14234 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1429752548668997632)]} 0 1
[junit4:junit4] 2> 14235 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[23 (1429752548668997632)]} 0 5
[junit4:junit4] 2> 14236 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1429752548671094784)]} 0 1
[junit4:junit4] 2> 14237 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10023 (1429752548671094784)]} 0 4
[junit4:junit4] 2> 14267 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[20 (-1429752548706746368)]} 0 0
[junit4:junit4] 2> 14269 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10020 (-1429752548708843520)]} 0 0
[junit4:junit4] 2> 14272 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[20 (-1429752548709892096)]} 0 0
[junit4:junit4] 2> 14273 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[20 (-1429752548709892096)]} 0 3
[junit4:junit4] 2> 14274 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10020 (-1429752548711989248)]} 0 1
[junit4:junit4] 2> 14274 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10020 (-1429752548711989248)]} 0 3
[junit4:junit4] 2> 14276 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[24 (1429752548716183552)]} 0 1
[junit4:junit4] 2> 14277 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10024 (1429752548716183553)]} 0 1
[junit4:junit4] 2> 14282 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1429752548719329280)]} 0 1
[junit4:junit4] 2> 14282 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1429752548719329281)]} 0 1
[junit4:junit4] 2> 14282 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[24 (1429752548719329280)]} 0 3
[junit4:junit4] 2> 14283 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10024 (1429752548719329281)]} 0 4
[junit4:junit4] 2> 14305 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[25 (1429752548745543681)]} 0 1
[junit4:junit4] 2> 14305 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10025 (1429752548745543680)]} 0 1
[junit4:junit4] 2> 14315 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1429752548748689408)]} 0 1
[junit4:junit4] 2> 14315 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1429752548748689409)]} 0 1
[junit4:junit4] 2> 14316 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[25 (1429752548748689408)]} 0 9
[junit4:junit4] 2> 14316 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10025 (1429752548748689409)]} 0 9
[junit4:junit4] 2> 14344 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[22 (-1429752548787486720)]} 0 0
[junit4:junit4] 2> 14344 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10022 (-1429752548787486721)]} 0 0
[junit4:junit4] 2> 14349 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10022 (-1429752548790632448)]} 0 1
[junit4:junit4] 2> 14349 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[22 (-1429752548790632449)]} 0 1
[junit4:junit4] 2> 14349 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10022 (-1429752548790632448)]} 0 2
[junit4:junit4] 2> 14349 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[22 (-1429752548790632449)]} 0 2
[junit4:junit4] 2> 14352 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10026 (1429752548794826752)]} 0 1
[junit4:junit4] 2> 14352 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[26 (1429752548795875328)]} 0 1
[junit4:junit4] 2> 14357 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1429752548797972480)]} 0 0
[junit4:junit4] 2> 14357 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1429752548799021056)]} 0 0
[junit4:junit4] 2> 14358 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10026 (1429752548797972480)]} 0 4
[junit4:junit4] 2> 14358 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[26 (1429752548799021056)]} 0 3
[junit4:junit4] 2> 14405 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10027 (1429752548850401280)]} 0 1
[junit4:junit4] 2> 14405 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[27 (1429752548851449856)]} 0 0
[junit4:junit4] 2> 14410 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1429752548853547008)]} 0 0
[junit4:junit4] 2> 14410 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1429752548853547009)]} 0 0
[junit4:junit4] 2> 14411 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[27 (1429752548853547009)]} 0 4
[junit4:junit4] 2> 14411 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10027 (1429752548853547008)]} 0 4
[junit4:junit4] 2> 14428 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10028 (1429752548874518528)]} 0 1
[junit4:junit4] 2> 14428 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[28 (1429752548874518529)]} 0 1
[junit4:junit4] 2> 14433 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1429752548877664256)]} 0 0
[junit4:junit4] 2> 14434 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1429752548877664257)]} 0 1
[junit4:junit4] 2> 14434 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[28 (1429752548877664256)]} 0 4
[junit4:junit4] 2> 14434 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10028 (1429752548877664257)]} 0 4
[junit4:junit4] 2> 14504 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10029 (1429752548954210305)]} 0 1
[junit4:junit4] 2> 14504 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[29 (1429752548954210304)]} 0 1
[junit4:junit4] 2> 14512 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1429752548958404609)]} 0 1
[junit4:junit4] 2> 14512 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1429752548958404608)]} 0 1
[junit4:junit4] 2> 14512 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10029 (1429752548958404609)]} 0 5
[junit4:junit4] 2> 14512 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[29 (1429752548958404608)]} 0 5
[junit4:junit4] 2> 14518 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[30 (1429752548969938944)]} 0 0
[junit4:junit4] 2> 14519 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10030 (1429752548969938945)]} 0 1
[junit4:junit4] 2> 14524 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1429752548973084672)]} 0 1
[junit4:junit4] 2> 14524 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1429752548973084673)]} 0 1
[junit4:junit4] 2> 14524 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[30 (1429752548973084672)]} 0 3
[junit4:junit4] 2> 14524 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10030 (1429752548973084673)]} 0 3
[junit4:junit4] 2> 14570 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10031 (1429752549024464896)]} 0 0
[junit4:junit4] 2> 14570 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[31 (1429752549024464897)]} 0 0
[junit4:junit4] 2> 14576 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1429752549027610625)]} 0 1
[junit4:junit4] 2> 14576 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1429752549027610624)]} 0 1
[junit4:junit4] 2> 14576 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10031 (1429752549027610625)]} 0 3
[junit4:junit4] 2> 14577 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[31 (1429752549027610624)]} 0 5
[junit4:junit4] 2> 14662 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[23 (-1429752549120933888)]} 0 0
[junit4:junit4] 2> 14664 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10023 (-1429752549123031040)]} 0 0
[junit4:junit4] 2> 14667 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[23 (-1429752549124079616)]} 0 0
[junit4:junit4] 2> 14667 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[23 (-1429752549124079616)]} 0 3
[junit4:junit4] 2> 14668 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10023 (-1429752549125128192)]} 0 0
[junit4:junit4] 2> 14669 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10023 (-1429752549125128192)]} 0 3
[junit4:junit4] 2> 14670 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[32 (1429752549129322496)]} 0 1
[junit4:junit4] 2> 14672 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10032 (1429752549130371072)]} 0 1
[junit4:junit4] 2> 14675 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1429752549131419648)]} 0 1
[junit4:junit4] 2> 14676 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[32 (1429752549131419648)]} 0 4
[junit4:junit4] 2> 14677 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1429752549133516800)]} 0 1
[junit4:junit4] 2> 14677 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10032 (1429752549133516800)]} 0 3
[junit4:junit4] 2> 14699 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[24 (-1429752549159731200)]} 0 0
[junit4:junit4] 2> 14701 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10024 (-1429752549161828352)]} 0 0
[junit4:junit4] 2> 14704 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[24 (-1429752549162876928)]} 0 0
[junit4:junit4] 2> 14705 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[24 (-1429752549162876928)]} 0 3
[junit4:junit4] 2> 14705 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10024 (-1429752549164974080)]} 0 0
[junit4:junit4] 2> 14706 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10024 (-1429752549164974080)]} 0 3
[junit4:junit4] 2> 14707 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[33 (1429752549168119808)]} 0 0
[junit4:junit4] 2> 14708 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10033 (1429752549169168384)]} 0 0
[junit4:junit4] 2> 14712 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[33 (1429752549170216960)]} 0 1
[junit4:junit4] 2> 14713 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[33 (1429752549170216960)]} 0 4
[junit4:junit4] 2> 14713 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10033 (1429752549172314112)]} 0 0
[junit4:junit4] 2> 14714 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10033 (1429752549172314112)]} 0 3
[junit4:junit4] 2> 14723 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[25 (-1429752549184897024)]} 0 0
[junit4:junit4] 2> 14724 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10025 (-1429752549185945600)]} 0 0
[junit4:junit4] 2> 14727 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[25 (-1429752549186994176)]} 0 0
[junit4:junit4] 2> 14728 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[25 (-1429752549186994176)]} 0 3
[junit4:junit4] 2> 14728 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10025 (-1429752549188042752)]} 0 0
[junit4:junit4] 2> 14729 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10025 (-1429752549188042752)]} 0 3
[junit4:junit4] 2> 14731 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[34 (1429752549192237056)]} 0 1
[junit4:junit4] 2> 14731 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10034 (1429752549193285632)]} 0 0
[junit4:junit4] 2> 14736 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[34 (1429752549195382784)]} 0 1
[junit4:junit4] 2> 14736 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[34 (1429752549195382784)]} 0 3
[junit4:junit4] 2> 14736 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10034 (1429752549195382785)]} 0 0
[junit4:junit4] 2> 14737 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10034 (1429752549195382785)]} 0 4
[junit4:junit4] 2> 14823 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10035 (1429752549289754625)]} 0 0
[junit4:junit4] 2> 14823 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[35 (1429752549289754624)]} 0 0
[junit4:junit4] 2> 14829 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10035 (1429752549292900352)]} 0 1
[junit4:junit4] 2> 14829 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[35 (1429752549292900353)]} 0 1
[junit4:junit4] 2> 14829 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10035 (1429752549292900352)]} 0 3
[junit4:junit4] 2> 14829 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[35 (1429752549292900353)]} 0 3
[junit4:junit4] 2> 14838 T1031 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53624
[junit4:junit4] 2> 14838 T1031 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=28536120
[junit4:junit4] 2> 14894 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[36 (1429752549363154944)]} 0 1
[junit4:junit4] 2> 14894 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10036 (1429752549363154945)]} 0 1
[junit4:junit4] 2> 14901 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[36 (1429752549368397824)]} 0 1
[junit4:junit4] 2> 14902 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[36 (1429752549368397824)]} 0 5
[junit4:junit4] 2> 14903 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10036 (1429752549370494976)]} 0 0
[junit4:junit4] 2> 14904 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10036 (1429752549370494976)]} 0 4
[junit4:junit4] 2> 14966 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[37 (1429752549438652416)]} 0 1
[junit4:junit4] 2> 14967 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10037 (1429752549440749568)]} 0 1
[junit4:junit4] 2> 14972 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[37 (1429752549441798144)]} 0 1
[junit4:junit4] 2> 14973 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[37 (1429752549441798144)]} 0 5
[junit4:junit4] 2> 14973 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10037 (1429752549443895296)]} 0 1
[junit4:junit4] 2> 14974 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10037 (1429752549443895296)]} 0 4
[junit4:junit4] 2> 14978 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[38 (1429752549452283904)]} 0 1
[junit4:junit4] 2> 14979 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10038 (1429752549452283905)]} 0 1
[junit4:junit4] 2> 14985 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10038 (1429752549455429633)]} 0 1
[junit4:junit4] 2> 14986 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10038 (1429752549455429633)]} 0 5
[junit4:junit4] 2> 14986 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[38 (1429752549455429632)]} 0 1
[junit4:junit4] 2> 14987 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[38 (1429752549455429632)]} 0 6
[junit4:junit4] 2> 15088 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10039 (1429752549567627264)]} 0 0
[junit4:junit4] 2> 15089 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[39 (1429752549567627265)]} 0 1
[junit4:junit4] 2> 15094 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10039 (1429752549570772992)]} 0 1
[junit4:junit4] 2> 15095 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10039 (1429752549570772992)]} 0 4
[junit4:junit4] 2> 15103 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[39 (1429752549570772993)]} 0 1
[junit4:junit4] 2> 15104 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[39 (1429752549570772993)]} 0 13
[junit4:junit4] 2> 15117 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 15118 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53624_mah%2Fnf_collection1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53624_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53624/mah/nf"}
[junit4:junit4] 2> 15122 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15122 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15122 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15122 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 15143 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10040 (1429752549624250368)]} 0 1
[junit4:junit4] 2> 15146 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10040 (1429752549628444672)]} 0 1
[junit4:junit4] 2> 15151 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[40 (1429752549633687552)]} 0 1
[junit4:junit4] 2> 15155 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[40 (1429752549636833280)]} 0 1
[junit4:junit4] 2> 15226 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10030 (-1429752549712330752)]} 0 0
[junit4:junit4] 2> 15229 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10030 (-1429752549715476480)]} 0 0
[junit4:junit4] 2> 15232 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10041 (1429752549717573632)]} 0 1
[junit4:junit4] 2> 15235 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[30 (-1429752549721767936)]} 0 0
[junit4:junit4] 2> 15236 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10041 (1429752549721767936)]} 0 1
[junit4:junit4] 2> 15239 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[30 (-1429752549724913664)]} 0 1
[junit4:junit4] 2> 15242 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[41 (1429752549728059392)]} 0 1
[junit4:junit4] 2> 15245 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[41 (1429752549731205120)]} 0 1
[junit4:junit4] 2> 15260 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10042 (1429752549746933760)]} 0 1
[junit4:junit4] 2> 15263 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10042 (1429752549751128064)]} 0 1
[junit4:junit4] 2> 15269 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[42 (1429752549756370944)]} 0 1
[junit4:junit4] 2> 15272 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[42 (1429752549759516672)]} 0 1
[junit4:junit4] 2> 15273 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10031 (-1429752549761613824)]} 0 0
[junit4:junit4] 2> 15275 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10031 (-1429752549763710976)]} 0 0
[junit4:junit4] 2> 15278 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10043 (1429752549766856704)]} 0 0
[junit4:junit4] 2> 15282 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10043 (1429752549770002432)]} 0 0
[junit4:junit4] 2> 15282 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[31 (-1429752549771051008)]} 0 0
[junit4:junit4] 2> 15284 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[31 (-1429752549773148160)]} 0 0
[junit4:junit4] 2> 15288 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[43 (1429752549776293888)]} 0 1
[junit4:junit4] 2> 15292 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[43 (1429752549780488192)]} 0 1
[junit4:junit4] 2> 15354 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10044 (1429752549846548480)]} 0 1
[junit4:junit4] 2> 15358 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10044 (1429752549849694208)]} 0 1
[junit4:junit4] 2> 15364 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[44 (1429752549857034240)]} 0 0
[junit4:junit4] 2> 15380 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[44 (1429752549860179968)]} 0 13
[junit4:junit4] 2> 15381 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10033 (-1429752549874860032)]} 0 0
[junit4:junit4] 2> 15384 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10033 (-1429752549878005760)]} 0 1
[junit4:junit4] 2> 15387 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10045 (1429752549880102912)]} 0 1
[junit4:junit4] 2> 15390 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10045 (1429752549884297216)]} 0 0
[junit4:junit4] 2> 15400 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[33 (-1429752549894782976)]} 0 0
[junit4:junit4] 2> 15403 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[33 (-1429752549897928704)]} 0 1
[junit4:junit4] 2> 15405 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[45 (1429752549900025856)]} 0 0
[junit4:junit4] 2> 15408 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[45 (1429752549903171584)]} 0 0
[junit4:junit4] 2> 15493 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10046 (1429752549991251968)]} 0 1
[junit4:junit4] 2> 15496 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10046 (1429752549995446272)]} 0 0
[junit4:junit4] 2> 15511 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[46 (1429752550010126336)]} 0 1
[junit4:junit4] 2> 15514 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[46 (1429752550013272064)]} 0 1
[junit4:junit4] 2> 15527 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10034 (-1429752550027952128)]} 0 1
[junit4:junit4] 2> 15529 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10034 (-1429752550030049280)]} 0 0
[junit4:junit4] 2> 15532 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10047 (1429752550033195008)]} 0 1
[junit4:junit4] 2> 15535 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10047 (1429752550035292160)]} 0 1
[junit4:junit4] 2> 15544 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[34 (-1429752550045777920)]} 0 0
[junit4:junit4] 2> 15547 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[34 (-1429752550048923648)]} 0 0
[junit4:junit4] 2> 15550 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[47 (1429752550052069376)]} 0 1
[junit4:junit4] 2> 15553 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[47 (1429752550054166528)]} 0 1
[junit4:junit4] 2> 15635 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10048 (1429752550140149760)]} 0 1
[junit4:junit4] 2> 15637 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10048 (1429752550143295488)]} 0 0
[junit4:junit4] 2> 15652 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[48 (1429752550159024128)]} 0 0
[junit4:junit4] 2> 15655 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[48 (1429752550162169856)]} 0 0
[junit4:junit4] 2> 15721 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10037 (-1429752550231375872)]} 0 0
[junit4:junit4] 2> 15725 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10037 (-1429752550235570176)]} 0 1
[junit4:junit4] 2> 15728 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10049 (1429752550237667328)]} 0 1
[junit4:junit4] 2> 15731 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10049 (1429752550240813056)]} 0 1
[junit4:junit4] 2> 15739 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[37 (-1429752550250250240)]} 0 0
[junit4:junit4] 2> 15741 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[37 (-1429752550252347392)]} 0 0
[junit4:junit4] 2> 15744 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[49 (1429752550255493120)]} 0 0
[junit4:junit4] 2> 15747 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[49 (1429752550258638848)]} 0 1
[junit4:junit4] 2> 15829 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10038 (-1429752550344622080)]} 0 1
[junit4:junit4] 2> 15831 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10038 (-1429752550346719232)]} 0 0
[junit4:junit4] 2> 15834 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10050 (1429752550348816384)]} 0 1
[junit4:junit4] 2> 15837 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10050 (1429752550351962112)]} 0 1
[junit4:junit4] 2> C60_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53624_mah%2Fnf, base_url=http://127.0.0.1:53624/mah/nf}
[junit4:junit4] 2> 15837 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59596/mah/nf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 15837 T1092 C60 P53624 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53624/mah/nf START replicas=[http://127.0.0.1:59596/mah/nf/collection1/] nUpdates=100
[junit4:junit4] 2> 15839 T1092 C60 P53624 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 15840 T1066 C62 P59596 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 15840 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 15840 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 15840 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 15841 T1092 C60 P53624 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4] 2> 15841 T1092 C60 P53624 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59596/mah/nf/collection1/. core=collection1
[junit4:junit4] 2> 15841 T1092 C60 P53624 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 15846 T1069 C62 P59596 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 15846 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[38 (-1429752550362447872)]} 0 0
[junit4:junit4] 2> 15850 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[38 (-1429752550365593600)]} 0 1
[junit4:junit4] 2> 15853 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[50 (1429752550368739328)]} 0 1
[junit4:junit4] 2> 15853 T1069 C62 P59596 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49a82c lockFactory=org.apache.lucene.store.NativeFSLockFactory@15d5a96),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@49a82c lockFactory=org.apache.lucene.store.NativeFSLockFactory@15d5a96),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, segments_2, _0.fdx, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4] 2> 15854 T1069 C62 P59596 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, segments_2, _0.fdx, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4] 2> 15855 T1069 C62 P59596 oass.SolrIndexSearcher.<init> Opening Searcher@19ece02 realtime
[junit4:junit4] 2> 15855 T1069 C62 P59596 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 15856 T1069 C62 P59596 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 15857 T1092 C60 P53624 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 15857 T1092 C60 P53624 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 15857 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[50 (1429752550372933632)]} 0 2
[junit4:junit4] 2> 15859 T1067 C62 P59596 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 15859 T1067 C62 P59596 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 15860 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 15860 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 15860 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 15860 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10039 (-1429752550377127936)]} 0 0
[junit4:junit4] 2> 15862 T1069 C62 P59596 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 15863 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10039 (-1429752550380273664)]} 0 1
[junit4:junit4] 2> 15863 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 14
[junit4:junit4] 2> 15864 T1092 C60 P53624 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index.20130317070446114 forceNew: false
[junit4:junit4] 2> 15864 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e) fullCopy=true
[junit4:junit4] 2> 15866 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[10051 (1429752550382370816)]} 0 1
[junit4:junit4] 2> 15866 T1066 C62 P59596 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15869 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10051 (1429752550385516544)]} 0 1
[junit4:junit4] 2> 15870 T1069 C62 P59596 REQ /replication {file=_0_Lucene42_0.dvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15873 T1068 C62 P59596 REQ /replication {file=_0_Lucene41_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15880 T1066 C62 P59596 REQ /replication {file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15881 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[39 (-1429752550399148032)]} 0 0
[junit4:junit4] 2> 15883 T1067 C62 P59596 REQ /replication {file=_0_Lucene42_0.dvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15883 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[39 (-1429752550401245184)]} 0 0
[junit4:junit4] 2> 15886 T1068 C62 P59596 REQ /replication {file=_0.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15887 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[51 (1429752550404390912)]} 0 1
[junit4:junit4] 2> 15889 T1069 C62 P59596 REQ /replication {file=_0_Lucene41_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15890 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[51 (1429752550407536640)]} 0 0
[junit4:junit4] 2> 15892 T1067 C62 P59596 REQ /replication {file=_0.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15896 T1068 C62 P59596 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15903 T1066 C62 P59596 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15906 T1069 C62 P59596 REQ /replication {file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15910 T1067 C62 P59596 REQ /replication {file=_0_Lucene41_0.tim&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15913 T1068 C62 P59596 REQ /replication {file=_0_Lucene41_0.tip&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 15917 T1066 C62 P59596 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1
[junit4:junit4] 2> 15918 T1092 C60 P53624 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 15918 T1092 C60 P53624 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130317070446114
[junit4:junit4] 2> 15918 T1092 C60 P53624 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 15918 T1092 C60 P53624 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 15919 T1092 C60 P53624 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4] 2> 15920 T1092 C60 P53624 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index.20130317070446114
[junit4:junit4] 2> 15921 T1092 C60 P53624 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, segments_2, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4] 2> 15921 T1092 C60 P53624 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, segments_2, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4] 2> 15922 T1092 C60 P53624 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 15923 T1092 C60 P53624 oass.SolrIndexSearcher.<init> Opening Searcher@11442e6 main
[junit4:junit4] 2> 15923 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[10040 (-1429752550443188224)]} 0 0
[junit4:junit4] 2> 15924 T1091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11442e6 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C99/39)}
[junit4:junit4] 2> 15924 T1092 C60 P53624 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d6854d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1139040)
[junit4:junit4] 2> 15924 T1092 C60 P53624 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/index
[junit4:junit4] 2> 15925 T1092 C60 P53624 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@1d6854d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1139040)
[junit4:junit4] 2> 15926 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10040 (-1429752550446333952)]} 0 0
[junit4:junit4] 2> 15927 T1092 C60 P53624 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4] 2> 15927 T1100 C60 P53624 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363518270254/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=5757
[junit4:junit4] 2> 15932 T1100 C60 P53624 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 15933 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10052 (1429752550452625408)]} 0 1
[junit4:junit4] 2> 15933 T1100 C60 P53624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, segments_2, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, segments_3, _0.fdt]
[junit4:junit4] 2> 15934 T1100 C60 P53624 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, segments_3, _0.fdt]
[junit4:junit4] 2> 15934 T1100 C60 P53624 oass.SolrIndexSearcher.<init> Opening Searcher@3f25dc main
[junit4:junit4] 2> 15934 T1100 C60 P53624 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 15935 T1091 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f25dc main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C99/39)}
[junit4:junit4] 2> 15935 T1100 C60 P53624 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=5757}
[junit4:junit4] 2> 15936 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 15936 T1092 C60 P53624 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 15936 T1092 C60 P53624 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 15936 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10052 (1429752550456819712)]} 0 0
[junit4:junit4] 2> 15937 T1092 C60 P53624 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 15940 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[40 (-1429752550461014016)]} 0 0
[junit4:junit4] 2> 15944 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[10041 (-1429752550464159744)]} 0 1
[junit4:junit4] 2> 15944 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[40 (-1429752550464159744)]} 0 1
[junit4:junit4] 2> 15946 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[10041 (-1429752550467305472)]} 0 0
[junit4:junit4] 2> 15947 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[52 (1429752550467305472)]} 0 1
[junit4:junit4] 2> 15949 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10053 (1429752550470451200)]} 0 1
[junit4:junit4] 2> 15949 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[52 (1429752550470451200)]} 0 0
[junit4:junit4] 2> 15952 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10053 (1429752550472548352)]} 0 1
[junit4:junit4] 2> 15956 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[41 (-1429752550477791232)]} 0 0
[junit4:junit4] 2> 15958 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[41 (-1429752550479888384)]} 0 0
[junit4:junit4] 2> 15961 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[53 (1429752550481985536)]} 0 1
[junit4:junit4] 2> 15964 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[53 (1429752550485131264)]} 0 1
[junit4:junit4] 2> 16029 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10042 (-1429752550553288704)]} 0 1
[junit4:junit4] 2> 16031 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10042 (-1429752550556434432)]} 0 0
[junit4:junit4] 2> 16035 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10054 (1429752550559580160)]} 0 1
[junit4:junit4] 2> 16038 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10054 (1429752550562725888)]} 0 1
[junit4:junit4] 2> 16040 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[42 (-1429752550565871616)]} 0 0
[junit4:junit4] 2> 16043 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[42 (-1429752550569017344)]} 0 0
[junit4:junit4] 2> 16045 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[54 (1429752550571114496)]} 0 0
[junit4:junit4] 2> 16048 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[54 (1429752550574260224)]} 0 0
[junit4:junit4] 2> 16073 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10045 (-1429752550599426048)]} 0 0
[junit4:junit4] 2> 16075 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10045 (-1429752550602571776)]} 0 0
[junit4:junit4] 2> 16078 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10055 (1429752550604668928)]} 0 1
[junit4:junit4] 2> 16081 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10055 (1429752550607814656)]} 0 1
[junit4:junit4] 2> 16083 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[45 (-1429752550610960384)]} 0 0
[junit4:junit4] 2> 16085 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[45 (-1429752550613057536)]} 0 0
[junit4:junit4] 2> 16089 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[55 (1429752550616203264)]} 0 1
[junit4:junit4] 2> 16092 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[55 (1429752550619348992)]} 0 1
[junit4:junit4] 2> 16120 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10046 (-1429752550649757696)]} 0 1
[junit4:junit4] 2> 16123 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10046 (-1429752550652903424)]} 0 0
[junit4:junit4] 2> 16126 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10056 (1429752550656049152)]} 0 0
[junit4:junit4] 2> 16129 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10056 (1429752550659194880)]} 0 0
[junit4:junit4] 2> 16131 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[46 (-1429752550661292032)]} 0 1
[junit4:junit4] 2> 16133 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[46 (-1429752550663389184)]} 0 0
[junit4:junit4] 2> 16136 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[56 (1429752550665486336)]} 0 1
[junit4:junit4] 2> 16138 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[56 (1429752550668632064)]} 0 0
[junit4:junit4] 2> 16226 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10047 (-1429752550760906752)]} 0 0
[junit4:junit4] 2> 16229 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10047 (-1429752550764052480)]} 0 0
[junit4:junit4] 2> 16232 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10057 (1429752550766149632)]} 0 1
[junit4:junit4] 2> 16234 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[47 (-1429752550769295360)]} 0 0
[junit4:junit4] 2> 16235 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10057 (1429752550769295360)]} 0 1
[junit4:junit4] 2> 16237 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[47 (-1429752550772441088)]} 0 1
[junit4:junit4] 2> 16239 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[57 (1429752550774538240)]} 0 0
[junit4:junit4] 2> 16242 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[57 (1429752550777683968)]} 0 0
[junit4:junit4] 2> 16248 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10058 (1429752550782926848)]} 0 1
[junit4:junit4] 2> 16251 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10058 (1429752550786072576)]} 0 1
[junit4:junit4] 2> 16256 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[58 (1429752550791315456)]} 0 1
[junit4:junit4] 2> 16258 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[58 (1429752550794461184)]} 0 0
[junit4:junit4] 2> 16313 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[10050 (-1429752550852132864)]} 0 1
[junit4:junit4] 2> 16316 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[10050 (-1429752550855278592)]} 0 1
[junit4:junit4] 2> 16319 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10059 (1429752550857375744)]} 0 1
[junit4:junit4] 2> 16320 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[50 (-1429752550859472896)]} 0 0
[junit4:junit4] 2> 16322 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10059 (1429752550860521472)]} 0 1
[junit4:junit4] 2> 16322 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[50 (-1429752550861570048)]} 0 0
[junit4:junit4] 2> 16325 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[59 (1429752550864715776)]} 0 1
[junit4:junit4] 2> 16328 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[59 (1429752550867861504)]} 0 0
[junit4:junit4] 2> 16365 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10060 (1429752550906658816)]} 0 1
[junit4:junit4] 2> 16368 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[10060 (1429752550909804544)]} 0 1
[junit4:junit4] 2> 16371 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10061 (1429752550911901696)]} 0 1
[junit4:junit4] 2> 16372 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[60 (1429752550912950272)]} 0 1
[junit4:junit4] 2> 16374 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10061 (1429752550915047424)]} 0 1
[junit4:junit4] 2> 16374 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[60 (1429752550916096000)]} 0 0
[junit4:junit4] 2> 16377 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[61 (1429752550919241728)]} 0 0
[junit4:junit4] 2> 16380 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[61 (1429752550921338880)]} 0 1
[junit4:junit4] 2> 16439 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10051 (-1429752550984253440)]} 0 0
[junit4:junit4] 2> 16442 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[10051 (-1429752550987399168)]} 0 0
[junit4:junit4] 2> 16444 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[51 (-1429752550989496321)]} 0 0
[junit4:junit4] 2> 16444 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10062 (1429752550989496320)]} 0 0
[junit4:junit4] 2> 16447 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[51 (-1429752550992642048)]} 0 0
[junit4:junit4] 2> 16447 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10062 (1429752550992642049)]} 0 0
[junit4:junit4] 2> 16450 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[62 (1429752550994739200)]} 0 1
[junit4:junit4] 2> 16452 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[62 (1429752550997884928)]} 0 0
[junit4:junit4] 2> 16494 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10053 (-1429752551041925120)]} 0 0
[junit4:junit4] 2> 16497 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[10053 (-1429752551045070848)]} 0 0
[junit4:junit4] 2> 16499 T1046 C61 P44725 /update {wt=javabin&version=2} {delete=[53 (-1429752551047168000)]} 0 0
[junit4:junit4] 2> 16500 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10063 (1429752551047168001)]} 0 1
[junit4:junit4] 2> 16502 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[53 (-1429752551050313728)]} 0 1
[junit4:junit4] 2> 16503 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10063 (1429752551050313729)]} 0 1
[junit4:junit4] 2> 16504 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[63 (1429752551052410880)]} 0 0
[junit4:junit4] 2> 16507 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[63 (1429752551055556608)]} 0 0
[junit4:junit4] 2> 16557 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10064 (1429752551107985408)]} 0 1
[junit4:junit4] 2> 16560 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10064 (1429752551111131136)]} 0 1
[junit4:junit4] 2> 16561 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[64 (1429752551112179712)]} 0 0
[junit4:junit4] 2> 16565 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[64 (1429752551115325440)]} 0 1
[junit4:junit4] 2> 16629 T1054 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 16629 T1054 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"127.0.0.1:53624_mah%2Fnf_collection1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:53624_mah%2Fnf",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53624/mah/nf"}
[junit4:junit4] 2> 16658 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10054 (-1429752551213891584)]} 0 0
[junit4:junit4] 2> 16661 T1068 C62 P59596 /update {wt=javabin&version=2} {delete=[10054 (-1429752551217037312)]} 0 0
[junit4:junit4] 2> 16662 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[54 (-1429752551218085888)]} 0 0
[junit4:junit4] 2> 16664 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10065 (1429752551219134464)]} 0 1
[junit4:junit4] 2> 16665 T1069 C62 P59596 /update {wt=javabin&version=2} {delete=[54 (-1429752551221231616)]} 0 0
[junit4:junit4] 2> 16667 T1067 C62 P59596 /update {wt=javabin&version=2} {add=[10065 (1429752551222280192)]} 0 1
[junit4:junit4] 2> 16668 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[65 (1429752551224377344)]} 0 0
[junit4:junit4] 2> 16671 T1066 C62 P59596 /update {wt=javabin&version=2} {add=[65 (1429752551227523072)]} 0 1
[junit4:junit4] 2> 16720 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16720 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16720 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16720 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 16724 T1049 C61 P44725 /update {wt=javabin&version=2} {add=[10066 (1429752551282049024)]} 0 1
[junit4:junit4] 2> 16727 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[66 (1429752551285194752)]} 0 1
[junit4:junit4] 2> C60_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53624_mah%2Fnf, base_url=http://127.0.0.1:53624/mah/nf}
[junit4:junit4] 2> 16731 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10066 (1429752551285194752)]} 0 2
[junit4:junit4] 2> 16731 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10066 (1429752551285194752)]} 0 5
[junit4:junit4] 2> 16734 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[66 (1429752551289389056)]} 0 1
[junit4:junit4] 2> 16735 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[66 (1429752551289389056)]} 0 5
[junit4:junit4] 2> 16797 T1047 C61 P44725 /update {wt=javabin&version=2} {delete=[10058 (-1429752551359643648)]} 0 0
[junit4:junit4] 2> 16801 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[58 (-1429752551362789376)]} 0 1
[junit4:junit4] 2> 16802 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10058 (-1429752551362789376)]} 0 0
[junit4:junit4] 2> 16803 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10058 (-1429752551362789376)]} 0 3
[junit4:junit4] 2> 16805 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[58 (-1429752551365935104)]} 0 0
[junit4:junit4] 2> 16806 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[58 (-1429752551365935104)]} 0 3
[junit4:junit4] 2> 16806 T1048 C61 P44725 /update {wt=javabin&version=2} {add=[10067 (1429752551368032256)]} 0 1
[junit4:junit4] 2> 16809 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[67 (1429752551371177984)]} 0 1
[junit4:junit4] 2> 16812 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10067 (1429752551371177984)]} 0 1
[junit4:junit4] 2> 16812 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[10067 (1429752551371177984)]} 0 4
[junit4:junit4] 2> 16814 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[67 (1429752551374323712)]} 0 1
[junit4:junit4] 2> 16815 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[67 (1429752551374323712)]} 0 4
[junit4:junit4] 2> 16885 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10059 (-1429752551451918336)]} 0 0
[junit4:junit4] 2> 16888 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[59 (-1429752551455064064)]} 0 0
[junit4:junit4] 2> 16890 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10059 (-1429752551455064064)]} 0 0
[junit4:junit4] 2> 16891 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10059 (-1429752551455064064)]} 0 3
[junit4:junit4] 2> 16892 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[59 (-1429752551457161216)]} 0 0
[junit4:junit4] 2> 16893 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[59 (-1429752551457161216)]} 0 3
[junit4:junit4] 2> 16894 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10068 (1429752551460306944)]} 0 1
[junit4:junit4] 2> 16896 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[68 (1429752551462404096)]} 0 1
[junit4:junit4] 2> 16899 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10068 (1429752551463452672)]} 0 1
[junit4:junit4] 2> 16900 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10068 (1429752551463452672)]} 0 4
[junit4:junit4] 2> 16902 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[68 (1429752551466598400)]} 0 1
[junit4:junit4] 2> 16902 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[68 (1429752551466598400)]} 0 4
[junit4:junit4] 2> 16918 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10060 (-1429752551486521344)]} 0 1
[junit4:junit4] 2> 16920 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[60 (-1429752551487569920)]} 0 1
[junit4:junit4] 2> 16922 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10060 (-1429752551488618496)]} 0 0
[junit4:junit4] 2> 16923 T1067 C62 P59596 /update {wt=javabin&version=2} {delete=[10060 (-1429752551488618496)]} 0 3
[junit4:junit4] 2> 16924 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[60 (-1429752551490715648)]} 0 0
[junit4:junit4] 2> 16924 T1066 C62 P59596 /update {wt=javabin&version=2} {delete=[60 (-1429752551490715648)]} 0 2
[junit4:junit4] 2> 16926 T1047 C61 P44725 /update {wt=javabin&version=2} {add=[10069 (1429752551493861376)]} 0 1
[junit4:junit4] 2> 16927 T1046 C61 P44725 /update {wt=javabin&version=2} {add=[69 (1429752551495958528)]} 0 1
[junit4:junit4] 2> 16931 T1082 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10069 (1429752551497007104)]} 0 1
[junit4:junit4] 2> 16931 T1069 C62 P59596 /update {wt=javabin&version=2} {add=[10069 (1429752551497007104)]} 0 3
[junit4:junit4] 2> 16932 T1084 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[69 (1429752551498055680)]} 0 1
[junit4:junit4] 2> 16932 T1068 C62 P59596 /update {wt=javabin&version=2} {add=[69 (1429752551498055680)]} 0 3
[junit4:junit4] 2> 16980 T1031 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 16980 T1031 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 16987 T1031 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1677c5c
[junit4:junit4] 2> 16988 T1049 C61 P44725 /update {wt=javabin&version=2} {delete=[10061 (-1429752551559921664)]} 0 0
[junit4:junit4] 2> 16990 T1048 C61 P44725 /update {wt=javabin&version=2} {delete=[61 (-1429752551562018816)]} 0 0
[junit4:junit4] 2> 16992 T1031 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=8,adds=8,deletesById=6,deletesByQuery=0,errors=0,cumulative_adds=60,cumulative_deletesById=24,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 16993 T1031 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 16994 T1031 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 16994 T1085 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 0
[junit4:junit4] 2> 16994 T1031 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 16995 T1085 C60 P53624 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:288)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:456)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1031)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:717)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 16996 T1083 C60 P53624 /update {distrib.from=http://127.0.0.1:59596/mah/nf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 0
[junit4:junit4] 2> 16997 T1083 C60 P53624 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4] 2> at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4] 2> at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:288)
[junit4:junit4] 2> at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
[junit4:junit4] 2> at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:456)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1031)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:717)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
[junit4:junit4] 2> at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4] 2> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4] 2> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4] 2> at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 16997 T1031 C60 P53624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Lucene42_0.dvd, _0_Lucene41_0.pos, _0_1.del, _0_Lucene42_0.dvm, _0.nvm, _0_Lucene41_0.doc, _0.nvd, _0.fdx, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, segments_3, _0.fdt]
[junit4:junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e),segFN=segments_4,generation=4,filenames=[_1_Lucene41_0.doc, _0_Lucene42_0.dvd, _1_Lucene42_0.dvm, _0_Lucene41_0.pos, _0_Lucene42_0.dvm, _1_Lucene42_0.dvd, _1.fnm, _0.nvd, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1.nvm, _0.fnm, _1_Lucene41_0.tim, _1.nvd, _0_1.del, _1_Lucene41_0.tip, _0.nvm, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0.fdx, _0.fdt, segments_4]
[junit4:junit4] 2> 16998 T1083 C60 P53624 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1677c5c
[junit4:junit4] 2> 16998 T1067 C62 P59596 oasc.SolrException.log SEVERE shard update error StdNode: http://127.0.0.1:53624/mah/nf/collection1/:org.apache.solr.common.SolrException: Server at http://127.0.0.1:53624/mah/nf/collection1 returned non ok status:503, message:Service Unavailable
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 16998 T1031 C60 P53624 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_1_Lucene41_0.doc, _0_Lucene42_0.dvd, _1_Lucene42_0.dvm, _0_Lucene41_0.pos, _0_Lucene42_0.dvm, _1_Lucene42_0.dvd, _1.fnm, _0.nvd, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1.nvm, _0.fnm, _1_Lucene41_0.tim, _1.nvd, _0_1.del, _1_Lucene41_0.tip, _0.nvm, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0.fdx, _0.fdt, segments_4]
[junit4:junit4] 2> 16998 T1083 C60 P53624 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=8,deletesById=8,deletesByQuery=0,errors=0,cumulative_adds=60,cumulative_deletesById=26,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 16999 T1067 C62 P59596 oasup.DistributedUpdateProcessor.doFinish try and ask http://127.0.0.1:53624/mah/nf to recover
[junit4:junit4] 2> 17000 T1067 C62 P59596 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 17000 T1083 C60 P53624 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 17000 T1031 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 17002 T1036 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d7805ab310008
[junit4:junit4] 2> 17002 T1066 C62 P59596 oasc.SolrException.log SEVERE shard update error StdNode: http://127.0.0.1:53624/mah/nf/collection1/:org.apache.solr.common.SolrException: Server at http://127.0.0.1:53624/mah/nf/collection1 returned non ok status:503, message:Service Unavailable
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4] 2> at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
[junit4:junit4] 2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2>
[junit4:junit4] 2> 17002 T1066 C62 P59596 oasup.DistributedUpdateProcessor.doFinish try and ask http://127.0.0.1:53624/mah/nf to recover
[junit4:junit4] 2> 17002 T1073 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 17003 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:41329 which had sessionid 0x13d7805ab310008
[junit4:junit4] 2> 17003 T1066 C62 P59596 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 17003 T1089 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 17003 T1089 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 17003 T1089 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 17004 T1089 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> 17003 T1031 oaz.ZooKeeper.close Session: 0x13d7805ab310008 closed
[junit4:junit4] 2> 17005 T1053 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 17005 T1073 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 17004 T1060 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 17005 T1053 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 17005 T1060 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 17007 T1067 C62 P59596 oasup.DistributedUpdateProcessor.doFinish Could not tell a replica to recover org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:53624/mah/nf
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
[junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:440)
[junit4:junit4] 2> at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1086)
[junit4:junit4] 2> at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
[junit4:junit4] 2> at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
[junit4:junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4] 2> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1811)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:
[...truncated too long message...]
t talk to ZK
[junit4:junit4] 2> 37123 T1113 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=234), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=234)), sim=DefaultSimilarity, locale=es_PA, timezone=America/Kralendijk
[junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic i386/Sun Microsystems Inc. 1.6.0_41 (32-bit)/cpus=8,threads=1,free=78210952,total=151617536
[junit4:junit4] 2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, TestFastLRUCache, TestStressRecovery, TestPhraseSuggestions, QueryElevationComponentTest, TestCoreContainer, TestSolrIndexConfig, TestPerFieldSimilarity, ResponseLogComponentTest, SolrInfoMBeanTest, ZkControllerTest, RegexBoostProcessorTest, UUIDFieldTest, TestJoin, DistributedTermsComponentTest, SimpleFacetsTest, DocumentBuilderTest, TestWriterPerf, DistributedQueryElevationComponentTest, SliceStateTest, TestCopyFieldCollectionResource, TestFieldCollectionResource, ExternalFileFieldSortTest, ResourceLoaderTest, TestLMDirichletSimilarityFactory, SchemaVersionSpecificBehaviorTest, RAMDirectoryFactoryTest, MBeansHandlerTest, ScriptEngineTest, UnloadDistributedZkTest, LukeRequestHandlerTest, TestSearchPerf, MinimalSchemaTest, TestPluginEnable, TestRemoteStreaming, TestCharFilters, TestRandomDVFaceting, TestGroupingSearch, TestTrie, SystemInfoHandlerTest, TestFaceting, QueryParsingTest, ZkCLITest, WordBreakSolrSpellCheckerTest, TestSolrCoreProperties, TestSweetSpotSimilarityFactory, DirectSolrConnectionTest, SOLR749Test, TestPseudoReturnFields, PrimUtilsTest, MoreLikeThisHandlerTest, SpellingQueryConverterTest, RecoveryZkTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=RecoveryZkTest -Dtests.seed=EB048B570D5B71D3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=America/Kralendijk -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1090, name=coreLoadExecutor-563-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1479)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:284)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:236)
[junit4:junit4] > at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4] > at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
[junit4:junit4] > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
[junit4:junit4] > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
[junit4:junit4] > at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
[junit4:junit4] > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit4:junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[junit4:junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@1159c89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f2d2d3)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > ... 1 more
[junit4:junit4] > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1092, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4] > at java.lang.Thread.getStackTrace(Thread.java:1479)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
[junit4:junit4] > at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
[junit4:junit4] > at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:284)
[junit4:junit4] > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:236)
[junit4:junit4] > at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4] > at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4] > at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4] > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@16760fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c6248e)
[junit4:junit4] > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4] > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4] > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] Completed on J0 in 37.22s, 1 test, 2 errors <<< FAILURES!
[...truncated 492 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 2 suite-level errors, 13 ignored (7 assumptions)
Total time: 40 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.6.0_41 -server -XX:+UseConcMarkSweepGC
Email was triggered for: Failure
Sending email for trigger: Failure