You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/10/07 15:38:28 UTC
[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build #
6164 - Failure!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6164/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.test
Error Message:
There are still nodes recoverying - waited for 120 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 120 seconds
at __randomizedtesting.SeedInfo.seed([C7DCD9F18D9D42EB:4F88E62B23612F13]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:181)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:862)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1418)
at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11669 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\init-core-data-001
[junit4] 2> 1500886 INFO (SUITE-RecoveryZkTest-seed#[C7DCD9F18D9D42EB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1500888 INFO (SUITE-RecoveryZkTest-seed#[C7DCD9F18D9D42EB]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 1500891 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1500891 INFO (Thread-2989) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1500891 INFO (Thread-2989) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1500991 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ZkTestServer start zk server on port:51236
[junit4] 2> 1501017 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1501022 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1501025 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1501027 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1501032 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1501035 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1501037 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1501039 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1501044 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1501046 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1501049 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractZkTestCase put C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test-files\solr\collection1\conf\synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1501052 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1579fa61d340001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1501733 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\cores\collection1
[junit4] 2> 1501736 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 1501737 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@33690423{/,null,AVAILABLE}
[junit4] 2> 1501738 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.ServerConnector Started ServerConnector@6a55dd5d{HTTP/1.1,[http/1.1]}{127.0.0.1:51243}
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server Started @1506431ms
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\tempDir-001/control/data, hostContext=/, hostPort=51243, coreRootDirectory=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\cores}
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1501740 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-07T14:57:27.168Z
[junit4] 2> 1501746 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1501746 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\solr.xml
[junit4] 2> 1501765 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51236/solr
[junit4] 2> 1501802 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51243_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:51243_
[junit4] 2> 1501804 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51243_ ] o.a.s.c.Overseer Overseer (id=96721452815089668-127.0.0.1:51243_-n_0000000000) starting
[junit4] 2> 1501815 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51243_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51243_
[junit4] 2> 1501820 INFO (OverseerStateUpdate-96721452815089668-127.0.0.1:51243_-n_0000000000) [n:127.0.0.1:51243_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1501839 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51243_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\cores
[junit4] 2> 1501839 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51243_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1501843 INFO (OverseerStateUpdate-96721452815089668-127.0.0.1:51243_-n_0000000000) [n:127.0.0.1:51243_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1502867 WARN (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1502869 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1502890 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1503008 WARN (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1503009 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1503042 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1503043 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\cores\collection1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\control-001\cores\collection1\data\]
[junit4] 2> 1503043 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@23f82ab3
[junit4] 2> 1503044 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1503048 WARN (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
[junit4] 2> 1503062 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1503062 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1503064 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1503064 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1503064 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=43.9794921875, floorSegmentMB=1.3837890625, forceMergeDeletesPctAllowed=29.289365871401326, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3264400438938047
[junit4] 2> 1503065 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@694239f8[collection1] main]
[junit4] 2> 1503066 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1503067 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1503067 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1503068 INFO (searcherExecutor-7294-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@694239f8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1503068 INFO (coreLoadExecutor-7293-thread-1-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547543247318941696
[junit4] 2> 1503081 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1503081 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1503081 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:51243/collection1/
[junit4] 2> 1503081 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1503081 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:51243/collection1/ has no replicas
[junit4] 2> 1503090 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:51243/collection1/ shard1
[junit4] 2> 1503244 INFO (coreZkRegister-7286-thread-1-processing-n:127.0.0.1:51243_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1503349 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1503351 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1503351 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
[junit4] 2> 1503358 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1579fa61d340006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1503870 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\cores\collection1
[junit4] 2> 1503873 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001
[junit4] 2> 1503874 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 1503875 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@70075bc6{/,null,AVAILABLE}
[junit4] 2> 1503876 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.ServerConnector Started ServerConnector@54d7f00{HTTP/1.1,[http/1.1]}{127.0.0.1:51276}
[junit4] 2> 1503877 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server Started @1508568ms
[junit4] 2> 1503877 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=51276, coreRootDirectory=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\cores}
[junit4] 2> 1503877 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1503878 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1503878 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1503878 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-07T14:57:29.306Z
[junit4] 2> 1503883 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1503883 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\solr.xml
[junit4] 2> 1503901 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51236/solr
[junit4] 2> 1503916 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1503921 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51276_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51276_
[junit4] 2> 1503924 INFO (zkCallback-10363-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1503925 INFO (zkCallback-10359-thread-4-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1503925 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1503941 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51276_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\cores
[junit4] 2> 1503941 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51276_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1503944 INFO (OverseerStateUpdate-96721452815089668-127.0.0.1:51243_-n_0000000000) [n:127.0.0.1:51243_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1504047 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1504961 WARN (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1504962 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1504976 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1505066 WARN (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1505068 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1505095 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1505096 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\cores\collection1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-1-001\cores\collection1\data\]
[junit4] 2> 1505096 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@23f82ab3
[junit4] 2> 1505096 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1505100 WARN (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
[junit4] 2> 1505116 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1505116 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1505117 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1505118 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1505119 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=43.9794921875, floorSegmentMB=1.3837890625, forceMergeDeletesPctAllowed=29.289365871401326, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3264400438938047
[junit4] 2> 1505119 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1a76f302[collection1] main]
[junit4] 2> 1505120 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1505120 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1505120 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1505121 INFO (searcherExecutor-7305-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1a76f302[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1505121 INFO (coreLoadExecutor-7304-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547543249471668224
[junit4] 2> 1505133 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1505133 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1505133 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:51276/collection1/
[junit4] 2> 1505133 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1505133 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:51276/collection1/ has no replicas
[junit4] 2> 1505140 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:51276/collection1/ shard1
[junit4] 2> 1505246 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1505293 INFO (coreZkRegister-7299-thread-1-processing-n:127.0.0.1:51276_ x:collection1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1505398 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1506018 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.SolrTestCaseJ4 Writing core.properties file to C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\cores\collection1
[junit4] 2> 1506022 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001
[junit4] 2> 1506024 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 1506025 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@909f67c{/,null,AVAILABLE}
[junit4] 2> 1506026 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.ServerConnector Started ServerConnector@547f2148{HTTP/1.1,[http/1.1]}{127.0.0.1:51300}
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.e.j.s.Server Started @1510718ms
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/, hostPort=51300, coreRootDirectory=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\cores}
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1506028 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-07T14:57:31.456Z
[junit4] 2> 1506036 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1579fa61d34000a, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1506036 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1506037 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\solr.xml
[junit4] 2> 1506055 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51236/solr
[junit4] 2> 1506076 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51300_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1506083 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51300_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51300_
[junit4] 2> 1506085 INFO (zkCallback-10363-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1506087 INFO (zkCallback-10359-thread-4-processing-n:127.0.0.1:51243_) [n:127.0.0.1:51243_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1506087 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1506088 INFO (zkCallback-10375-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1506104 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51300_ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\cores
[junit4] 2> 1506104 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [n:127.0.0.1:51300_ ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
[junit4] 2> 1506109 INFO (OverseerStateUpdate-96721452815089668-127.0.0.1:51243_-n_0000000000) [n:127.0.0.1:51243_ ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
[junit4] 2> 1506214 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1506214 INFO (zkCallback-10375-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1507125 WARN (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 1507128 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 1507142 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
[junit4] 2> 1507238 WARN (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 1507239 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1507264 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1507264 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\cores\collection1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\..\..\..\..\..\..\..\..\..\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001\shard-2-001\cores\collection1\data\]
[junit4] 2> 1507264 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@23f82ab3
[junit4] 2> 1507265 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1507268 WARN (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
[junit4] 2> 1507283 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1507283 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1507286 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1507286 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1507287 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=43.9794921875, floorSegmentMB=1.3837890625, forceMergeDeletesPctAllowed=29.289365871401326, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3264400438938047
[junit4] 2> 1507287 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6888b2d2[collection1] main]
[junit4] 2> 1507288 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1507289 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1507289 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1507290 INFO (searcherExecutor-7316-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6888b2d2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1507290 INFO (coreLoadExecutor-7315-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547543251746029568
[junit4] 2> 1507297 INFO (coreZkRegister-7310-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.ZkController Core needs to recover:collection1
[junit4] 2> 1507297 INFO (updateExecutor-10372-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1507298 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1507298 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1507298 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1]
[junit4] 2> 1507298 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1507298 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Publishing state of core [collection1] as recovering, leader is [http://127.0.0.1:51276/collection1/] and I am [http://127.0.0.1:51300/collection1/]
[junit4] 2> 1507301 INFO (recoveryExecutor-10373-thread-1-processing-n:127.0.0.1:51300_ x:collection1 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:51276]; [WaitForState: action=PREPRECOVERY&core=collection1&nodeName=127.0.0.1:51300_&coreNodeName=core_node2&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 1507303 INFO (qtp1646218296-42301) [n:127.0.0.1:51276_ ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 1507304 INFO (qtp1646218296-42301) [n:127.0.0.1:51276_ ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 1507304 INFO (qtp1646218296-42301) [n:127.0.0.1:51276_ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard1, thisCore=collection1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:51300_, coreNodeName=core_node2, onlyIfActiveCheckResult=false, nodeProps: core_node2:{"core":"collection1","base_url":"http://127.0.0.1:51300","node_name":"127.0.0.1:51300_","state":"down"}
[junit4] 2> 1507405 INFO (zkCallback-10375-thread-1-processing-n:127.0.0.1:51300_) [n:127.0.0.1:51300_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1507405 INFO (zkCallback-10369-thread-1-processing-n:127.0.0.1:51276_) [n:127.0.0.1:51276_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1507612 INFO (TEST-RecoveryZkTest.test-seed#[C7DCD9F18D9D42EB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 1507621 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-0 (1547543252088913920)]} 0 4
[junit4] 2> 1507621 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-0 (1547543252088913921)]} 0 4
[junit4] 2> 1507635 INFO (qtp704931942-42330) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-0 (1547543252097302529)]} 0 1
[junit4] 2> 1507635 INFO (qtp704931942-42329) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-0 (1547543252097302528)]} 0 1
[junit4] 2> 1507636 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-0 (1547543252097302529)]} 0 11
[junit4] 2> 1507636 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-0 (1547543252097302528)]} 0 11
[junit4] 2> 1507637 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-1 (1547543252108836864)]} 0 0
[junit4] 2> 1507637 INFO (qtp1566556360-42261) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-1 (1547543252109885440)]} 0 0
[junit4] 2> 1507639 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-1 (1547543252110934016)]} 0 0
[junit4] 2> 1507639 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-1 (1547543252110934016)]} 0 1
[junit4] 2> 1507640 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-0 (-1547543252111982592)]} 0 0
[junit4] 2> 1507640 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-1 (1547543252111982592)]} 0 0
[junit4] 2> 1507640 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-1 (1547543252111982592)]} 0 1
[junit4] 2> 1507641 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-0 (-1547543252114079744)]} 0 0
[junit4] 2> 1507641 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[1-0 (-1547543252114079744)]} 0 0
[junit4] 2> 1507641 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-0 (-1547543252114079744)]} 0 0
[junit4] 2> 1507642 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[2-0 (-1547543252115128320)]} 0 0
[junit4] 2> 1507642 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-2 (1547543252115128320)]} 0 0
[junit4] 2> 1507642 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-0 (-1547543252115128320)]} 0 0
[junit4] 2> 1507645 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-2 (1547543252118274048)]} 0 0
[junit4] 2> 1507646 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-2 (1547543252118274048)]} 0 0
[junit4] 2> 1507646 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-2 (1547543252118274048)]} 0 1
[junit4] 2> 1507646 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-2 (1547543252119322624)]} 0 0
[junit4] 2> 1507646 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-3 (1547543252119322624)]} 0 0
[junit4] 2> 1507646 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-2 (1547543252119322624)]} 0 1
[junit4] 2> 1507647 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-3 (1547543252120371200)]} 0 0
[junit4] 2> 1507649 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-3 (1547543252120371200)]} 0 0
[junit4] 2> 1507649 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-3 (1547543252120371200)]} 0 0
[junit4] 2> 1507650 INFO (qtp704931942-42329) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-3 (1547543252122468352)]} 0 0
[junit4] 2> 1507650 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-3 (1547543252122468352)]} 0 1
[junit4] 2> 1507650 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-4 (1547543252123516928)]} 0 0
[junit4] 2> 1507651 INFO (qtp1566556360-42261) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-4 (1547543252124565504)]} 0 0
[junit4] 2> 1507652 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-4 (1547543252125614080)]} 0 0
[junit4] 2> 1507652 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-4 (1547543252124565504)]} 0 0
[junit4] 2> 1507653 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-4 (1547543252125614080)]} 0 0
[junit4] 2> 1507653 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-4 (1547543252124565504)]} 0 1
[junit4] 2> 1507653 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-5 (1547543252126662656)]} 0 0
[junit4] 2> 1507654 INFO (qtp1566556360-42260) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-5 (1547543252126662657)]} 0 0
[junit4] 2> 1507656 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-5 (1547543252127711232)]} 0 0
[junit4] 2> 1507656 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-5 (1547543252127711232)]} 0 1
[junit4] 2> 1507656 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-5 (1547543252129808384)]} 0 0
[junit4] 2> 1507656 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-5 (1547543252129808384)]} 0 0
[junit4] 2> 1507657 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-6 (1547543252130856960)]} 0 0
[junit4] 2> 1507657 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-6 (1547543252130856961)]} 0 0
[junit4] 2> 1507658 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-6 (1547543252131905536)]} 0 0
[junit4] 2> 1507658 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-6 (1547543252131905537)]} 0 0
[junit4] 2> 1507658 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-6 (1547543252131905536)]} 0 0
[junit4] 2> 1507659 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-6 (1547543252131905537)]} 0 1
[junit4] 2> 1507659 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-1 (-1547543252132954112)]} 0 0
[junit4] 2> 1507659 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-1 (-1547543252132954113)]} 0 0
[junit4] 2> 1507661 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[1-1 (-1547543252134002689)]} 0 0
[junit4] 2> 1507661 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[2-1 (-1547543252134002688)]} 0 0
[junit4] 2> 1507661 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-1 (-1547543252134002689)]} 0 0
[junit4] 2> 1507661 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-1 (-1547543252134002688)]} 0 1
[junit4] 2> 1507662 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-7 (1547543252136099840)]} 0 0
[junit4] 2> 1507662 INFO (qtp1566556360-42261) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-7 (1547543252136099841)]} 0 0
[junit4] 2> 1507663 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-7 (1547543252137148416)]} 0 0
[junit4] 2> 1507663 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-7 (1547543252137148416)]} 0 1
[junit4] 2> 1507664 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-7 (1547543252137148417)]} 0 0
[junit4] 2> 1507664 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-7 (1547543252137148417)]} 0 0
[junit4] 2> 1507664 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-7 (-1547543252138196992)]} 0 0
[junit4] 2> 1507665 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-7 (-1547543252139245568)]} 0 0
[junit4] 2> 1507666 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[1-7 (-1547543252139245568)]} 0 0
[junit4] 2> 1507666 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-7 (-1547543252139245568)]} 0 0
[junit4] 2> 1507666 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[2-7 (-1547543252140294144)]} 0 0
[junit4] 2> 1507667 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-7 (-1547543252140294144)]} 0 1
[junit4] 2> 1507668 INFO (qtp1566556360-42260) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-8 (1547543252141342720)]} 0 1
[junit4] 2> 1507668 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-8 (1547543252141342721)]} 0 0
[junit4] 2> 1507669 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-8 (1547543252142391296)]} 0 0
[junit4] 2> 1507669 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-8 (1547543252142391296)]} 0 0
[junit4] 2> 1507669 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-8 (1547543252143439872)]} 0 0
[junit4] 2> 1507670 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-8 (1547543252143439872)]} 0 1
[junit4] 2> 1507670 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-9 (1547543252144488448)]} 0 0
[junit4] 2> 1507671 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-9 (1547543252145537024)]} 0 0
[junit4] 2> 1507672 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-9 (1547543252145537024)]} 0 0
[junit4] 2> 1507672 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-9 (1547543252145537024)]} 0 0
[junit4] 2> 1507673 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-9 (1547543252146585600)]} 0 0
[junit4] 2> 1507673 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-9 (1547543252146585600)]} 0 1
[junit4] 2> 1507673 INFO (qtp1566556360-42265) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-10 (1547543252147634176)]} 0 0
[junit4] 2> 1507674 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-10 (1547543252147634177)]} 0 0
[junit4] 2> 1507674 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-10 (1547543252148682752)]} 0 0
[junit4] 2> 1507674 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-10 (1547543252148682752)]} 0 0
[junit4] 2> 1507674 INFO (qtp1566556360-42261) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-11 (1547543252148682752)]} 0 0
[junit4] 2> 1507675 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-10 (1547543252148682753)]} 0 0
[junit4] 2> 1507675 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-10 (1547543252148682753)]} 0 1
[junit4] 2> 1507676 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-11 (1547543252150779904)]} 0 0
[junit4] 2> 1507676 INFO (qtp704931942-42329) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-11 (1547543252150779904)]} 0 0
[junit4] 2> 1507677 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-11 (1547543252150779904)]} 0 1
[junit4] 2> 1507677 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-12 (1547543252151828480)]} 0 0
[junit4] 2> 1507678 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-11 (1547543252151828480)]} 0 0
[junit4] 2> 1507678 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-11 (1547543252151828480)]} 0 1
[junit4] 2> 1507680 INFO (qtp704931942-42332) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-12 (1547543252152877056)]} 0 0
[junit4] 2> 1507680 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-12 (1547543252152877056)]} 0 1
[junit4] 2> 1507680 INFO (qtp1566556360-42260) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-12 (1547543252154974208)]} 0 0
[junit4] 2> 1507680 INFO (qtp1566556360-42258) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-13 (1547543252154974209)]} 0 0
[junit4] 2> 1507681 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-12 (1547543252154974208)]} 0 0
[junit4] 2> 1507681 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-12 (1547543252154974208)]} 0 1
[junit4] 2> 1507682 INFO (qtp704931942-42330) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-13 (1547543252156022784)]} 0 0
[junit4] 2> 1507682 INFO (qtp1566556360-42265) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-13 (1547543252157071360)]} 0 0
[junit4] 2> 1507682 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-13 (1547543252156022784)]} 0 1
[junit4] 2> 1507683 INFO (qtp1566556360-42263) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-14 (1547543252158119936)]} 0 0
[junit4] 2> 1507683 INFO (qtp704931942-42334) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-13 (1547543252158119936)]} 0 0
[junit4] 2> 1507684 INFO (qtp1646218296-42299) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-13 (1547543252158119936)]} 0 1
[junit4] 2> 1507684 INFO (qtp704931942-42327) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-14 (1547543252159168512)]} 0 0
[junit4] 2> 1507685 INFO (qtp1646218296-42304) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-14 (1547543252159168512)]} 0 0
[junit4] 2> 1507685 INFO (qtp1566556360-42261) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-14 (1547543252160217088)]} 0 0
[junit4] 2> 1507685 INFO (qtp1566556360-42264) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-10 (-1547543252160217089)]} 0 0
[junit4] 2> 1507686 INFO (qtp704931942-42329) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[2-14 (1547543252161265664)]} 0 0
[junit4] 2> 1507687 INFO (qtp1646218296-42306) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[2-14 (1547543252161265664)]} 0 1
[junit4] 2> 1507687 INFO (qtp704931942-42333) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[1-10 (-1547543252161265665)]} 0 0
[junit4] 2> 1507687 INFO (qtp1646218296-42303) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[1-10 (-1547543252161265665)]} 0 0
[junit4] 2> 1507687 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[2-10 (-1547543252162314240)]} 0 0
[junit4] 2> 1507688 INFO (qtp1566556360-42260) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[1-15 (1547543252163362816)]} 0 0
[junit4] 2> 1507689 INFO (qtp704931942-42332) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{delete=[2-10 (-1547543252163362816)]} 0 0
[junit4] 2> 1507690 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core_node2 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:51276/collection1/&wt=javabin&version=2}{add=[1-15 (1547543252164411392)]} 0 0
[junit4] 2> 1507690 INFO (qtp1646218296-42305) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{delete=[2-10 (-1547543252163362816)]} 0 1
[junit4] 2> 1507692 INFO (qtp1646218296-42302) [n:127.0.0.1:51276_ c:collection1 s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2}{add=[1-15 (1547543252164411392)]} 0 2
[junit4] 2> 1507692 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{add=[2-15 (1547543252167557120)]} 0 0
[junit4] 2> 1507692 INFO (qtp1566556360-42262) [n:127.0.0.1:51243_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1] webapp= path=/update params={wt=javabin&version=2&CONTROL=TRUE}{delete=[1-11 (-1547543252167557121)]} 0 0
[junit4] 2> 1507695 INFO (qtp704931942-42331) [n:127.0.0.1:51300_ c:collection1 s:shard1 r:core
[...truncated too long message...]
-Dtests.seed=C7DCD9F18D9D42EB -Dtests.slow=true -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP -Dtests.timezone=Etc/GMT-3 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 139s J0 | RecoveryZkTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 120 seconds
[junit4] > at __randomizedtesting.SeedInfo.seed([C7DCD9F18D9D42EB:4F88E62B23612F13]:0)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:181)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:862)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1418)
[junit4] > at org.apache.solr.cloud.RecoveryZkTest.test(RecoveryZkTest.java:105)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 1639426 INFO (SUITE-RecoveryZkTest-seed#[C7DCD9F18D9D42EB]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.RecoveryZkTest_C7DCD9F18D9D42EB-001
[junit4] 2> Oct 07, 2016 2:59:44 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=PostingsFormat(name=MockRandom), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i=PostingsFormat(name=MockRandom), id=PostingsFormat(name=MockRandom)}, docValues:{}, maxPointsInLeafNode=418, maxMBSortInHeap=5.038962343410876, sim=ClassicSimilarity, locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=Etc/GMT-3
[junit4] 2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=193164800,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestCSVResponseWriter, TestReloadDeadlock, OverseerTest, OutputWriterTest, FileUtilsTest, HdfsLockFactoryTest, TestExceedMaxTermLength, SpatialHeatmapFacetsTest, CollectionStateFormat2Test, TestStressLiveNodes, TestCursorMarkWithoutUniqueKey, PingRequestHandlerTest, TestOnReconnectListenerSupport, TestCloudDeleteByQuery, WrapperMergePolicyFactoryTest, DistributedQueryComponentCustomSortTest, CdcrReplicationDistributedZkTest, TestHashQParserPlugin, TestJsonRequest, TestSolrCloudWithDelegationTokens, CopyFieldTest, SolrCLIZkUtilsTest, TestTrackingShardHandlerFactory, TestSchemaSimilarityResource, TestLockTree, DocValuesTest, TestFieldCacheSortRandom, PrimitiveFieldTypeTest, HdfsRestartWhileUpdatingTest, HardAutoCommitTest, PeerSyncTest, SearchHandlerTest, DeleteLastCustomShardedReplicaTest, TestAddFieldRealTimeGet, ConnectionManagerTest, TestPostingsSolrHighlighter, RestartWhileUpdatingTest, AnalyticsMergeStrategyTest, SpatialFilterTest, TestCSVLoader, TestUninvertingReader, DistributedSuggestComponentTest, TestSurroundQueryParser, TestSolrDeletionPolicy1, TestSchemaNameResource, TestMinMaxOnMultiValuedField, SimplePostToolTest, TestCustomDocTransformer, AnalysisAfterCoreReloadTest, TestConfigOverlay, FastVectorHighlighterTest, TestFastLRUCache, TestPerFieldSimilarity, TestEmbeddedSolrServerConstructors, TestReRankQParserPlugin, MigrateRouteKeyTest, TermsComponentTest, HdfsCollectionsAPIDistributedZkTest, SchemaVersionSpecificBehaviorTest, TestManagedStopFilterFactory, TestLRUCache, DirectSolrSpellCheckerTest, TestMacros, DistributedFacetPivotSmallTest, TestCollapseQParserPlugin, XsltUpdateRequestHandlerTest, PreAnalyzedUpdateProcessorTest, TestDFISimilarityFactory, PKIAuthenticationIntegrationTest, TestXIncludeConfig, TestCoreContainer, SuggestComponentTest, CloudExitableDirectoryReaderTest, SolrRequestParserTest, BufferStoreTest, HdfsUnloadDistributedZkTest, TestLegacyNumericRangeQueryBuilder, TestSha256AuthenticationProvider, DeleteStatusTest, TestRandomDVFaceting, ActionThrottleTest, SSLMigrationTest, TestLRUStatsCache, TestSortingResponseWriter, TestHdfsCloudBackupRestore, ShardSplitTest, OpenExchangeRatesOrgProviderTest, TestDefaultStatsCache, BlockJoinFacetDistribTest, TestLazyCores, InfoHandlerTest, TestReplicaProperties, TestOverriddenPrefixQueryForCustomFieldType, DistanceUnitsTest, CloneFieldUpdateProcessorFactoryTest, TestPKIAuthenticationPlugin, PrimUtilsTest, TestWordDelimiterFilterFactory, TestSort, ExitableDirectoryReaderTest, SolrXmlInZkTest, SolrPluginUtilsTest, DirectUpdateHandlerTest, DistributedSpellCheckComponentTest, ZkStateReaderTest, TestCollationField, TestRawTransformer, ShardRoutingTest, SoftAutoCommitTest, TestShortCircuitedRequests, TestStandardQParsers, TestTrieFacet, TestUseDocValuesAsStored, TestInfoStreamLogging, PolyFieldTest, SuggesterFSTTest, TestFaceting, TestDynamicLoading, IndexSchemaRuntimeFieldTest, TestGroupingSearch, SynonymTokenizerTest, HighlighterTest, SolrCoreTest, TestFieldCacheSanityChecker, TestTolerantUpdateProcessorRandomCloud, DistributedFacetPivotLongTailTest, TestSubQueryTransformerDistrib, VersionInfoTest, TestFieldCacheWithThreads, AssignTest, TestIBSimilarityFactory, TestFieldCache, OverseerCollectionConfigSetProcessorTest, RecoveryAfterSoftCommitTest, TestDownShardTolerantSearch, HighlighterMaxOffsetTest, TestRequestStatusCollectionAPI, TestSortByMinMaxFunction, HdfsBasicDistributedZk2Test, TestHdfsBackupRestoreCore, SystemInfoHandlerTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSerializedLuceneMatchVersion, CursorPagingTest, ConjunctionSolrSpellCheckerTest, CurrencyFieldXmlFileTest, QueryElevationComponentTest, HdfsThreadLeakTest, TestRemoteStreaming, TestReqParamsAPI, TestRTGBase, UniqFieldsUpdateProcessorFactoryTest, DateFieldTest, BlockJoinFacetSimpleTest, TestSolrJ, TestUpdate, BasicDistributedZk2Test, SyncSliceTest, BasicZkTest, RecoveryZkTest]
[junit4] Completed [327/635 (1!)] on J0 in 138.82s, 1 test, 1 failure <<< FAILURES!
[...truncated 41688 lines...]
-ecj-javadoc-lint-src:
[mkdir] Created dir: C:\Users\jenkins\AppData\Local\Temp\ecj413951656
[ecj-lint] Compiling 127 source files to C:\Users\jenkins\AppData\Local\Temp\ecj413951656
[ecj-lint] ----------
[ecj-lint] 1. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 21)
[ecj-lint] import java.util.Collection;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.Collection is never used
[ecj-lint] ----------
[ecj-lint] 2. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 22)
[ecj-lint] import java.util.HashSet;
[ecj-lint] ^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.HashSet is never used
[ecj-lint] ----------
[ecj-lint] 3. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 24)
[ecj-lint] import java.util.List;
[ecj-lint] ^^^^^^^^^^^^^^
[ecj-lint] The import java.util.List is never used
[ecj-lint] ----------
[ecj-lint] 4. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 25)
[ecj-lint] import java.util.Set;
[ecj-lint] ^^^^^^^^^^^^^
[ecj-lint] The import java.util.Set is never used
[ecj-lint] ----------
[ecj-lint] 5. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 27)
[ecj-lint] import org.apache.lucene.analysis.Analyzer;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.analysis.Analyzer is never used
[ecj-lint] ----------
[ecj-lint] 6. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 33)
[ecj-lint] import org.apache.lucene.index.IndexableField;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.index.IndexableField is never used
[ecj-lint] ----------
[ecj-lint] 6 problems (6 errors)
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:765: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:101: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build.xml:204: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2177: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1992: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2031: Compile failed; see the compiler error output for details.
Total time: 88 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_102) - Build #
6170 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6170/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics
Error Message:
Could not find collection:testkerberoscollection
Stack Trace:
java.lang.AssertionError: Could not find collection:testkerberoscollection
at __randomizedtesting.SeedInfo.seed([26501AA142AE6761:1B88B48D7A403911]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:153)
at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testCollectionCreateSearchDelete(TestSolrCloudWithKerberosAlt.java:206)
at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics(TestSolrCloudWithKerberosAlt.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12210 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt
[junit4] 2> 2819810 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.d.s.c.DefaultDirectoryService You didn't change the admin password of directory service instance 'DefaultKrbServer'. Please update the admin password as soon as possible to prevent a possible security breach.
[junit4] 2> 2820886 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.TestSolrCloudWithKerberosAlt Enable delegation token: true
[junit4] 2> 2820888 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-002
[junit4] 2> 2820888 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2820889 INFO (Thread-5571) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2820889 INFO (Thread-5571) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2820989 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.ZkTestServer start zk server on port:53879
[junit4] 2> 2820990 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2820990 WARN (zkCallback-9820-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@9c9d0f name: ZooKeeperConnection Watcher:127.0.0.1:53879 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2820991 WARN (zkCallback-9820-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2821013 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 2821014 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@cfe696{/solr,null,AVAILABLE}
[junit4] 2> 2821015 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.ServerConnector Started ServerConnector@c48d73{HTTP/1.1,[http/1.1]}{127.0.0.1:53883}
[junit4] 2> 2821015 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.Server Started @2827271ms
[junit4] 2> 2821015 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53883}
[junit4] 2> 2821016 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 2821016 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2821016 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2821016 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-10T04:20:20.987Z
[junit4] 2> 2821017 WARN (jetty-launcher-9819-thread-1-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2821019 WARN (zkCallback-9821-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@ffbe08 name: ZooKeeperConnection Watcher:127.0.0.1:53879/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2821019 WARN (zkCallback-9821-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2821021 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2821034 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157acd1eb000001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2821041 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53879/solr
[junit4] 2> 2821042 WARN (jetty-launcher-9819-thread-1-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2821044 WARN (zkCallback-9824-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19f9d6e name: ZooKeeperConnection Watcher:127.0.0.1:53879 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2821044 WARN (zkCallback-9824-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2821045 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157acd1eb000002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2821047 WARN (jetty-launcher-9819-thread-1-SendThread(127.0.0.1:53879)) [n:127.0.0.1:53883_solr ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2821048 WARN (zkCallback-9825-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19ec545 name: ZooKeeperConnection Watcher:127.0.0.1:53879/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2821048 WARN (zkCallback-9825-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2821110 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53883_solr
[junit4] 2> 2821113 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.Overseer Overseer (id=96735934602543107-127.0.0.1:53883_solr-n_0000000000) starting
[junit4] 2> 2821390 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53883_solr
[junit4] 2> 2821394 INFO (OverseerStateUpdate-96735934602543107-127.0.0.1:53883_solr-n_0000000000) [n:127.0.0.1:53883_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2821404 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.security.KerberosPlugin
[junit4] 2> 2821405 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder Setting up SPNego auth with config: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf
[junit4] 2> 2821419 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.s.KerberosPlugin Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, kerberos.keytab=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\keytabs, cookie.domain=127.0.0.1, token.validity=36000, type=kerberos, delegation-token.token-kind=solr-dt, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, kerberos.name.rules=RULE:[1:$1@$0](.*EXAMPLE.COM)s/@.*//
[junit4] 2> RULE:[2:$2@$0](.*EXAMPLE.COM)s/@.*//
[junit4] 2> DEFAULT, signer.secret.provider=zookeeper}
[junit4] 2> 2821419 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2821421 WARN (jetty-launcher-9819-thread-1-SendThread(127.0.0.1:53879)) [n:127.0.0.1:53883_solr ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2821422 ERROR (jetty-launcher-9819-thread-1-EventThread) [n:127.0.0.1:53883_solr ] o.a.c.ConnectionState Authentication failed
[junit4] 2> 2821423 INFO (jetty-launcher-9819-thread-1-EventThread) [n:127.0.0.1:53883_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2822514 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder Setting up SPNego auth with config: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf
[junit4] 2> 2822517 INFO (jetty-launcher-9819-thread-1) [n:127.0.0.1:53883_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-002\node1\.
[junit4] 2> 2822521 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2822525 WARN (zkCallback-9827-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@13f6dda name: ZooKeeperConnection Watcher:127.0.0.1:53879 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2822525 WARN (zkCallback-9827-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2822534 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157acd1eb000005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2822560 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2822561 WARN (zkCallback-9829-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ec0529 name: ZooKeeperConnection Watcher:127.0.0.1:53879/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2822561 WARN (zkCallback-9829-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2823526 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2823528 WARN (zkCallback-9831-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33bf53 name: ZooKeeperConnection Watcher:127.0.0.1:53879/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2823528 WARN (zkCallback-9831-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2823531 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 1> Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\keytabs refreshKrb5Config is true principal is solr tryFirstPass is false useFirstPass is false storePass is false clearPass is false
[junit4] 1> Refreshing Kerberos configuration
[junit4] 1> principal is solr@EXAMPLE.COM
[junit4] 1> Will use keytab
[junit4] 1> Commit Succeeded
[junit4] 1>
[junit4] 2> 2825345 INFO (qtp8667766-44972) [n:127.0.0.1:53883_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&property.solr.tests.ramBufferSizeMB=100&property.solr.tests.mergePolicyFactory=org.apache.solr.index.TieredMergePolicyFactory&property.solr.tests.useMergePolicyFactory=true&property.solr.tests.useMergePolicy=false&version=2&property.solr.tests.mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler&property.config=solrconfig-tlog.xml&property.solr.tests.maxBufferedDocs=100000&name=testkerberoscollection&action=CREATE&numShards=1&property.solr.directoryFactory=solr.RAMDirectoryFactory&wt=javabin and sendToOCPQueue=true
[junit4] 2> 2825359 INFO (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.CreateCollectionCmd Create collection testkerberoscollection
[junit4] 2> 2825360 INFO (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:solrCloudCollectionConfig
[junit4] 2> 2825479 ERROR (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:53883/solr
[junit4] 2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53883/solr: Expected mime type application/octet-stream but got text/html. <html>
[junit4] 2> <head>
[junit4] 2> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
[junit4] 2> <title>Error 401 </title>
[junit4] 2> </head>
[junit4] 2> <body>
[junit4] 2> <h2>HTTP ERROR: 401</h2>
[junit4] 2> <p>Problem accessing /solr/admin/cores. Reason:
[junit4] 2> <pre> Authentication required</pre></p>
[junit4] 2> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
[junit4] 2> </body>
[junit4] 2> </html>
[junit4] 2>
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] 2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:195)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2825479 ERROR (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [testkerberoscollection].
[junit4] 2> 2825480 INFO (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 2826183 INFO (OverseerThreadFactory-12820-thread-1-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [testkerberoscollection]
[junit4] 2> 2826187 INFO (qtp8667766-44972) [n:127.0.0.1:53883_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 2827361 INFO (OverseerCollectionConfigSetProcessor-96735934602543107-127.0.0.1:53883_solr-n_0000000000) [n:127.0.0.1:53883_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2856209 ERROR (qtp8667766-44972) [n:127.0.0.1:53883_solr ] o.a.s.h.a.CollectionsHandler Timed out waiting for new collection's replicas to become ACTIVE with timeout=30
[junit4] 2> 2856209 INFO (qtp8667766-44972) [n:127.0.0.1:53883_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&property.solr.tests.ramBufferSizeMB=100&property.solr.tests.mergePolicyFactory=org.apache.solr.index.TieredMergePolicyFactory&property.solr.tests.useMergePolicyFactory=true&property.solr.tests.useMergePolicy=false&version=2&property.solr.tests.mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler&property.config=solrconfig-tlog.xml&property.solr.tests.maxBufferedDocs=100000&name=testkerberoscollection&action=CREATE&numShards=1&property.solr.directoryFactory=solr.RAMDirectoryFactory&wt=javabin} status=0 QTime=30863
[junit4] 2> 2856210 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=solr&p=solr@EXAMPLE.COM&t=kerberos-dt&e=1476109225316&s=OJ2+PzX0ncfJRoqumH04Kespjxw="; Path=/; Domain=127.0.0.1; Expires=ma, 10-okt-2016 14:20:25 GMT; HttpOnly". Invalid 'expires' attribute: ma, 10-okt-2016 14:20:25 GMT
[junit4] 2> 2856211 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]-SendThread(127.0.0.1:53879)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 2856212 WARN (zkCallback-9832-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@119f3f9 name: ZooKeeperConnection Watcher:127.0.0.1:53879/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 2856212 WARN (zkCallback-9832-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 2856215 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2856216 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testkerberoscollection failOnTimeout:true timeout (sec):330
[junit4] 1> -
[junit4] 2> 2856219 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.ServerConnector Stopped ServerConnector@c48d73{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2856219 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=32038384
[junit4] 2> 2856222 INFO (jetty-launcher-9819-thread-1) [ ] o.a.s.c.Overseer Overseer (id=96735934602543107-127.0.0.1:53883_solr-n_0000000000) closing
[junit4] 2> 2856222 INFO (OverseerStateUpdate-96735934602543107-127.0.0.1:53883_solr-n_0000000000) [n:127.0.0.1:53883_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:53883_solr
[junit4] 2> 2857725 WARN (zkCallback-9825-thread-2-processing-n:127.0.0.1:53883_solr) [n:127.0.0.1:53883_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2857725 ERROR (Thread[Thread-5580,5,TGRP-TestSolrCloudWithKerberosAlt]) [n:127.0.0.1:53883_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2857729 INFO (jetty-launcher-9819-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@cfe696{/solr,null,UNAVAILABLE}
[junit4] 2> 2857731 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[26501AA142AE6761]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53879 53879
[junit4] 2> 2857732 INFO (Thread-5571) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:53879 53879
[junit4] 2> 2857733 WARN (Thread-5571) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 33 /solr/overseer/collection-queue-work
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics -Dtests.seed=26501AA142AE6761 -Dtests.slow=true -Dtests.locale=no-NO -Dtests.timezone=Asia/Yerevan -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 60.3s J0 | TestSolrCloudWithKerberosAlt.testBasics <<<
[junit4] > Throwable #1: java.lang.AssertionError: Could not find collection:testkerberoscollection
[junit4] > at __randomizedtesting.SeedInfo.seed([26501AA142AE6761:1B88B48D7A403911]:0)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:153)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testCollectionCreateSearchDelete(TestSolrCloudWithKerberosAlt.java:206)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics(TestSolrCloudWithKerberosAlt.java:152)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithKerberosAlt_26501AA142AE6761-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=no-NO, timezone=Asia/Yerevan
[junit4] 2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_102 (32-bit)/cpus=3,threads=1,free=182138536,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestSQLHandler, HdfsCollectionsAPIDistributedZkTest, OverseerRolesTest, TestQuerySenderNoQuery, MBeansHandlerTest, DateFieldTest, TestTolerantSearch, TestJettySolrRunner, FastVectorHighlighterTest, CurrencyFieldXmlFileTest, TestSolrDeletionPolicy1, AnalysisErrorHandlingTest, TestFastLRUCache, TestSha256AuthenticationProvider, TestTrieFacet, DistanceUnitsTest, DistributedSuggestComponentTest, HLLSerializationTest, TestRemoteStreaming, TestMinMaxOnMultiValuedField, TestJmxIntegration, PKIAuthenticationIntegrationTest, TestManagedSchema, TestStressReorder, ReplaceNodeTest, TestRTGBase, TestSimpleQParserPlugin, TestFastOutputStream, MultiTermTest, SolrCLIZkUtilsTest, DeleteReplicaTest, TestFaceting, HdfsBasicDistributedZk2Test, MoreLikeThisHandlerTest, TestRTimerTree, TestTolerantUpdateProcessorRandomCloud, XmlUpdateRequestHandlerTest, TestElisionMultitermQuery, SuggesterFSTTest, TestCloudPivotFacet, ConnectionManagerTest, BinaryUpdateRequestHandlerTest, TestExceedMaxTermLength, HardAutoCommitTest, CursorPagingTest, TestFieldCacheSortRandom, TestDefaultStatsCache, TestNumericTerms32, UniqFieldsUpdateProcessorFactoryTest, TestCursorMarkWithoutUniqueKey, SharedFSAutoReplicaFailoverUtilsTest, HighlighterTest, TestIntervalFaceting, DisMaxRequestHandlerTest, SolrPluginUtilsTest, RequiredFieldsTest, PrimUtilsTest, SuggestComponentTest, TestConfigSetProperties, TestGroupingSearch, SoftAutoCommitTest, TestRandomDVFaceting, TestReloadDeadlock, DirectSolrConnectionTest, TestSolrDynamicMBean, TestSolrCloudWithSecureImpersonation, TestFieldCache, TestOverriddenPrefixQueryForCustomFieldType, DocValuesTest, TestXIncludeConfig, TestPartialUpdateDeduplication, TestConfigSetsAPIExclusivity, SpatialHeatmapFacetsTest, DistributedSpellCheckComponentTest, DistributedFacetPivotLongTailTest, TestBulkSchemaAPI, TestExactSharedStatsCache, FileUtilsTest, SolrRequestParserTest, TestStressLiveNodes, SynonymTokenizerTest, SchemaVersionSpecificBehaviorTest, TestDynamicFieldCollectionResource, DistributedQueueTest, TestBlobHandler, OutOfBoxZkACLAndCredentialsProvidersTest, QueryEqualityTest, TestMergePolicyConfig, RecoveryZkTest, UnloadDistributedZkTest, ForceLeaderTest, TestSurroundQueryParser, DeleteStatusTest, HdfsThreadLeakTest, SOLR749Test, PeerSyncTest, TestMacros, TestUpdate, AnalyticsMergeStrategyTest, ZkCLITest, LeaderFailoverAfterPartitionTest, AnalysisAfterCoreReloadTest, DocValuesMultiTest, PreAnalyzedUpdateProcessorTest, ExitableDirectoryReaderTest, HdfsUnloadDistributedZkTest, TestManagedStopFilterFactory, CustomCollectionTest, TolerantUpdateProcessorTest, BlockJoinFacetSimpleTest, TestAddFieldRealTimeGet, TestGraphMLResponseWriter, TestCSVLoader, TestSortingResponseWriter, CdcrReplicationDistributedZkTest, TestSolrCloudWithDelegationTokens, TestSortByMinMaxFunction, TestFieldCacheSanityChecker, HdfsChaosMonkeyNothingIsSafeTest, TestLRUCache, OpenExchangeRatesOrgProviderTest, TestPhraseSuggestions, DistributedQueryComponentCustomSortTest, SolrXmlInZkTest, BlockCacheTest, TestPKIAuthenticationPlugin, TestCollectionAPI, TestPseudoReturnFields, TestRequestStatusCollectionAPI, TestSubQueryTransformer, HighlighterMaxOffsetTest, SegmentsInfoRequestHandlerTest, BaseCdcrDistributedZkTest, ZkControllerTest, TestDynamicLoading, TestLockTree, SolrTestCaseJ4Test, DocValuesMissingTest, BasicDistributedZkTest, PingRequestHandlerTest, TestPerFieldSimilarity, TestHdfsCloudBackupRestore, BigEndianAscendingWordSerializerTest, TestMissingGroups, SolrIndexConfigTest, TestReplicationHandler, TestInfoStreamLogging, CloudExitableDirectoryReaderTest, SimpleFacetsTest, VersionInfoTest, PathHierarchyTokenizerFactoryTest, TestHighlightDedupGrouping, BlockJoinFacetDistribTest, DistributedQueryElevationComponentTest, IndexSchemaRuntimeFieldTest, TestSweetSpotSimilarityFactory, ChaosMonkeySafeLeaderTest, SyncSliceTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, LeaderElectionTest, TestDistributedSearch, AutoCommitTest, TestRecovery, TestHashPartitioner, TermVectorComponentDistributedTest, TestReload, TestStressVersions, DistributedTermsComponentTest, TestCoreContainer, TestSolr4Spatial, SolrCoreTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, BadIndexSchemaTest, TestSort, TestFiltering, TestFunctionQuery, TestLazyCores, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, CoreAdminHandlerTest, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterTest, TestStressLucene, TestTrie, SpatialFilterTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TermsComponentTest, DocumentBuilderTest, TermVectorComponentTest, TestIndexingPerformance, IndexSchemaTest, TestCollationField, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, TestAnalyzedSuggestions, JsonLoaderTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, SearchHandlerTest, TestLFUCache, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, CopyFieldTest, BadComponentTest, TestStressRecovery, SampleTest, TestBinaryField, NumericFieldsTest, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestCharFilters, TestCodecSupport, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, URLClassifyProcessorTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, DistributedIntervalFacetingTest, AliasIntegrationTest, CdcrReplicationHandlerTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, DeleteNodeTest, DistribCursorPagingTest, DistribDocExpirationUpdateProcessorTest, LeaderInitiatedRecoveryOnCommitTest, MultiThreadedOCPTest, OverseerTaskQueueTest, RestartWhileUpdatingTest, RollingRestartTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SimpleCollectionCreateDeleteTest, SolrCloudExampleTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudPseudoReturnFields, TestConfigSetsAPI, TestCryptoKeys, TestDownShardTolerantSearch, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestLeaderRecoverFromLogOnStartup, TestOnReconnectListenerSupport, TestRandomFlRTGCloud, TestRandomRequestDistribution, TestSSLRandomization, TestShortCircuitedRequests, TestSolrCloudWithKerberosAlt]
[junit4] Completed [487/635 (1!)] on J0 in 60.29s, 1 test, 1 failure <<< FAILURES!
[...truncated 53088 lines...]
[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_102) - Build #
6169 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6169/
Java: 32bit/jdk1.8.0_102 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics
Error Message:
Could not find collection:testkerberoscollection
Stack Trace:
java.lang.AssertionError: Could not find collection:testkerberoscollection
at __randomizedtesting.SeedInfo.seed([9FF77C9C6E00EE4E:A22FD2B056EEB03E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:153)
at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testCollectionCreateSearchDelete(TestSolrCloudWithKerberosAlt.java:206)
at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics(TestSolrCloudWithKerberosAlt.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10795 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithKerberosAlt
[junit4] 2> 328557 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.d.s.c.DefaultDirectoryService You didn't change the admin password of directory service instance 'DefaultKrbServer'. Please update the admin password as soon as possible to prevent a possible security breach.
[junit4] 2> 329387 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.TestSolrCloudWithKerberosAlt Enable delegation token: true
[junit4] 2> 329390 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-002
[junit4] 2> 329390 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 329390 INFO (Thread-594) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 329390 INFO (Thread-594) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 329491 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.ZkTestServer start zk server on port:59245
[junit4] 2> 329492 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 329494 WARN (zkCallback-348-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4e90eb name: ZooKeeperConnection Watcher:127.0.0.1:59245 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 329494 WARN (zkCallback-348-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 329507 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 329508 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ae0dc8{/solr,null,AVAILABLE}
[junit4] 2> 329510 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.ServerConnector Started ServerConnector@c49b0f{HTTP/1.1,[http/1.1]}{127.0.0.1:59249}
[junit4] 2> 329510 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.Server Started @334955ms
[junit4] 2> 329510 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59249}
[junit4] 2> 329511 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 329511 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 329511 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 329511 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-09T16:35:39.634Z
[junit4] 2> 329514 WARN (jetty-launcher-347-thread-1-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 329514 WARN (zkCallback-349-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1368a20 name: ZooKeeperConnection Watcher:127.0.0.1:59245/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 329514 WARN (zkCallback-349-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 329517 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 329550 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59245/solr
[junit4] 2> 329551 WARN (jetty-launcher-347-thread-1-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 329551 WARN (zkCallback-352-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@325c58 name: ZooKeeperConnection Watcher:127.0.0.1:59245 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 329552 WARN (zkCallback-352-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 329556 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157aa4cc17f0002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 329558 WARN (jetty-launcher-347-thread-1-SendThread(127.0.0.1:59245)) [n:127.0.0.1:59249_solr ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 329559 WARN (zkCallback-353-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d69b05 name: ZooKeeperConnection Watcher:127.0.0.1:59245/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 329559 WARN (zkCallback-353-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 329616 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59249_solr
[junit4] 2> 329620 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.Overseer Overseer (id=96733163652317187-127.0.0.1:59249_solr-n_0000000000) starting
[junit4] 2> 329639 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59249_solr
[junit4] 2> 329643 INFO (OverseerStateUpdate-96733163652317187-127.0.0.1:59249_solr-n_0000000000) [n:127.0.0.1:59249_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 329667 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.security.KerberosPlugin
[junit4] 2> 329675 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder Setting up SPNego auth with config: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf
[junit4] 2> 329736 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.s.KerberosPlugin Params: {token.valid=30, kerberos.principal=HTTP/127.0.0.1, kerberos.keytab=C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\keytabs, cookie.domain=127.0.0.1, token.validity=36000, type=kerberos, delegation-token.token-kind=solr-dt, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, kerberos.name.rules=RULE:[1:$1@$0](.*EXAMPLE.COM)s/@.*//
[junit4] 2> RULE:[2:$2@$0](.*EXAMPLE.COM)s/@.*//
[junit4] 2> DEFAULT, signer.secret.provider=zookeeper}
[junit4] 2> 330110 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 330127 WARN (jetty-launcher-347-thread-1-SendThread(127.0.0.1:59245)) [n:127.0.0.1:59249_solr ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 330152 ERROR (jetty-launcher-347-thread-1-EventThread) [n:127.0.0.1:59249_solr ] o.a.c.ConnectionState Authentication failed
[junit4] 2> 330174 INFO (jetty-launcher-347-thread-1-EventThread) [n:127.0.0.1:59249_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 330548 ERROR (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.h.u.Shell Failed to locate the winutils binary in the hadoop binary path
[junit4] 2> java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
[junit4] 2> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
[junit4] 2> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
[junit4] 2> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
[junit4] 2> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
[junit4] 2> at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:115)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:148)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:118)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:238)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:209)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.initializeAuthHandler(DelegationTokenKerberosFilter.java:137)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:227)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:175)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.init(DelegationTokenKerberosFilter.java:66)
[junit4] 2> at org.apache.solr.security.KerberosPlugin.init(KerberosPlugin.java:111)
[junit4] 2> at org.apache.solr.core.CoreContainer.initializeAuthenticationPlugin(CoreContainer.java:320)
[junit4] 2> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:468)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:203)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:152)
[junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1542)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1576)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1268)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1113)
[junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:447)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:258)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:342)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:315)
[junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:353)
[junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:220)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 330808 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.s.i.Krb5HttpClientBuilder Setting up SPNego auth with config: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf
[junit4] 2> 330811 INFO (jetty-launcher-347-thread-1) [n:127.0.0.1:59249_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-002\node1\.
[junit4] 2> 330814 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 330815 WARN (zkCallback-355-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c31cda name: ZooKeeperConnection Watcher:127.0.0.1:59245 got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 330815 WARN (zkCallback-355-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 330848 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 330849 WARN (zkCallback-357-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b40dd6 name: ZooKeeperConnection Watcher:127.0.0.1:59245/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 330849 WARN (zkCallback-357-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 331859 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157aa4cc17f0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 331860 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 331860 WARN (zkCallback-359-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1de45e0 name: ZooKeeperConnection Watcher:127.0.0.1:59245/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 331861 WARN (zkCallback-359-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 331865 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 1> Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\keytabs refreshKrb5Config is true principal is solr tryFirstPass is false useFirstPass is false storePass is false clearPass is false
[junit4] 1> Refreshing Kerberos configuration
[junit4] 1> principal is solr@EXAMPLE.COM
[junit4] 1> Will use keytab
[junit4] 1> Commit Succeeded
[junit4] 1>
[junit4] 2> 333424 INFO (qtp3438455-2238) [n:127.0.0.1:59249_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&property.solr.tests.ramBufferSizeMB=100&property.solr.tests.useMergePolicyFactory=false&property.solr.tests.useMergePolicy=true&version=2&property.solr.tests.mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler&property.config=solrconfig-tlog.xml&property.solr.tests.maxBufferedDocs=100000&name=testkerberoscollection&action=CREATE&numShards=1&property.solr.tests.mergePolicy=org.apache.lucene.index.TieredMergePolicy&property.solr.directoryFactory=solr.RAMDirectoryFactory&wt=javabin and sendToOCPQueue=true
[junit4] 2> 333430 INFO (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.CreateCollectionCmd Create collection testkerberoscollection
[junit4] 2> 333431 INFO (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.CreateCollectionCmd Only one config set found in zk - using it:solrCloudCollectionConfig
[junit4] 2> 333547 ERROR (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.OverseerCollectionMessageHandler Error from shard: http://127.0.0.1:59249/solr
[junit4] 2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:59249/solr: Expected mime type application/octet-stream but got text/html. <html>
[junit4] 2> <head>
[junit4] 2> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
[junit4] 2> <title>Error 401 </title>
[junit4] 2> </head>
[junit4] 2> <body>
[junit4] 2> <h2>HTTP ERROR: 401</h2>
[junit4] 2> <p>Problem accessing /solr/admin/cores. Reason:
[junit4] 2> <pre> Authentication required</pre></p>
[junit4] 2> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
[junit4] 2> </body>
[junit4] 2> </html>
[junit4] 2>
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:578)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] 2> at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:195)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 333549 ERROR (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [testkerberoscollection].
[junit4] 2> 333550 INFO (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 334255 INFO (OverseerThreadFactory-1058-thread-1-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [testkerberoscollection]
[junit4] 2> 334261 INFO (qtp3438455-2238) [n:127.0.0.1:59249_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 335434 INFO (OverseerCollectionConfigSetProcessor-96733163652317187-127.0.0.1:59249_solr-n_0000000000) [n:127.0.0.1:59249_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 364279 ERROR (qtp3438455-2238) [n:127.0.0.1:59249_solr ] o.a.s.h.a.CollectionsHandler Timed out waiting for new collection's replicas to become ACTIVE with timeout=30
[junit4] 2> 364279 INFO (qtp3438455-2238) [n:127.0.0.1:59249_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&property.solr.tests.ramBufferSizeMB=100&property.solr.tests.useMergePolicyFactory=false&property.solr.tests.useMergePolicy=true&version=2&property.solr.tests.mergeScheduler=org.apache.lucene.index.ConcurrentMergeScheduler&property.config=solrconfig-tlog.xml&property.solr.tests.maxBufferedDocs=100000&name=testkerberoscollection&action=CREATE&numShards=1&property.solr.tests.mergePolicy=org.apache.lucene.index.TieredMergePolicy&property.solr.directoryFactory=solr.RAMDirectoryFactory&wt=javabin} status=0 QTime=30855
[junit4] 2> 364304 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=solr&p=solr@EXAMPLE.COM&t=kerberos-dt&e=1476066943305&s=FhbUMbJ8zHqPhan5Gtrbzrq+aII="; Path=/; Domain=127.0.0.1; Expires=Mo, 10-Okt-2016 02:35:43 GMT; HttpOnly". Invalid 'expires' attribute: Mo, 10-Okt-2016 02:35:43 GMT
[junit4] 2> 364307 WARN (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]-SendThread(127.0.0.1:59245)) [ ] o.a.z.ClientCnxn SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: 'C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001\tempDir-001\minikdc\jaas-client.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
[junit4] 2> 364308 WARN (zkCallback-360-thread-1) [ ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14f492a name: ZooKeeperConnection Watcher:127.0.0.1:59245/solr got event WatchedEvent state:AuthFailed type:None path:null path: null type: None
[junit4] 2> 364308 WARN (zkCallback-360-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient received AuthFailed
[junit4] 2> 364312 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 364314 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: testkerberoscollection failOnTimeout:true timeout (sec):330
[junit4] 1> -
[junit4] 2> 364316 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157aa4cc17f0008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 364319 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157aa4cc17f0007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 364321 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.ServerConnector Stopped ServerConnector@c49b0f{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 364322 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=9644637
[junit4] 2> 364327 INFO (jetty-launcher-347-thread-1) [ ] o.a.s.c.Overseer Overseer (id=96733163652317187-127.0.0.1:59249_solr-n_0000000000) closing
[junit4] 2> 364328 INFO (OverseerStateUpdate-96733163652317187-127.0.0.1:59249_solr-n_0000000000) [n:127.0.0.1:59249_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59249_solr
[junit4] 2> 364333 WARN (zkCallback-353-thread-3-processing-n:127.0.0.1:59249_solr) [n:127.0.0.1:59249_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 364334 ERROR (Thread[Thread-603,5,TGRP-TestSolrCloudWithKerberosAlt]) [n:127.0.0.1:59249_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 364380 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157aa4cc17f0004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 364380 INFO (jetty-launcher-347-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ae0dc8{/solr,null,UNAVAILABLE}
[junit4] 2> 364387 INFO (TEST-TestSolrCloudWithKerberosAlt.testBasics-seed#[9FF77C9C6E00EE4E]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59245 59245
[junit4] 2> 364388 INFO (Thread-594) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59245 59245
[junit4] 2> 364390 WARN (Thread-594) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 33 /solr/overseer/collection-queue-work
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithKerberosAlt -Dtests.method=testBasics -Dtests.seed=9FF77C9C6E00EE4E -Dtests.slow=true -Dtests.locale=de-GR -Dtests.timezone=BET -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 54.5s J1 | TestSolrCloudWithKerberosAlt.testBasics <<<
[junit4] > Throwable #1: java.lang.AssertionError: Could not find collection:testkerberoscollection
[junit4] > at __randomizedtesting.SeedInfo.seed([9FF77C9C6E00EE4E:A22FD2B056EEB03E]:0)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:153)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testCollectionCreateSearchDelete(TestSolrCloudWithKerberosAlt.java:206)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithKerberosAlt.testBasics(TestSolrCloudWithKerberosAlt.java:152)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestSolrCloudWithKerberosAlt_9FF77C9C6E00EE4E-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1176, maxMBSortInHeap=6.43667035656776, sim=RandomSimilarity(queryNorm=false): {}, locale=de-GR, timezone=BET
[junit4] 2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_102 (32-bit)/cpus=3,threads=1,free=102924720,total=284164096
[junit4] 2> NOTE: All tests run in this JVM: [BitVectorTest, StatsComponentTest, StatsReloadRaceTest, TermVectorComponentDistributedTest, TestDistribIDF, TestComplexPhraseQParserPlugin, TestAnalyzeInfixSuggestions, TestNoOpRegenerator, TestNonDefinedSimilarityFactory, ResponseHeaderTest, TestSolr4Spatial, BasicDistributedZk2Test, TestQueryTypes, CollectionTooManyReplicasTest, OverseerStatusTest, CachingDirectoryFactoryTest, TestClassicSimilarityFactory, TestFoldingMultitermQuery, ShowFileRequestHandlerTest, JsonLoaderTest, TestClusterStateMutator, ReturnFieldsTest, TestConfigSetImmutable, TestAnalyzedSuggestions, TestComponentsName, TestUseDocValuesAsStored2, DistribJoinFromCollectionTest, TestDistributedMissingSort, TestRuleBasedAuthorizationPlugin, TestSolrCloudWithKerberosAlt]
[junit4] Completed [68/635 (1!)] on J1 in 54.52s, 1 test, 1 failure <<< FAILURES!
[...truncated 54500 lines...]
[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build #
6168 - Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6168/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.lucene.mockfile.TestHandleLimitFS
Error Message:
Could not remove the following files (in the order of attempts): C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002 C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001
Stack Trace:
java.io.IOException: Could not remove the following files (in the order of attempts):
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001
at __randomizedtesting.SeedInfo.seed([26BD3F668E4E4360]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 1897 lines...]
[junit4] Suite: org.apache.lucene.mockfile.TestHandleLimitFS
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=false): {}, locale=fr-CA, timezone=America/Porto_Acre
[junit4] 2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=117024408,total=146079744
[junit4] 2> NOTE: All tests run in this JVM: [TestMockCharFilter, TestForceMergePolicy, TestHandleTrackingFS, TestAssertingNormsFormat, Nested, Nested1, ThrowInUncaught, TestMockDirectoryWrapper, TestGroupFiltering, TestShuffleFS, TestCompressingTermVectorsFormat, TestGraphTokenizers, TestLookaheadTokenFilter, TestCompressingStoredFieldsFormat, TestMockAnalyzer, Nested, Nested2, TestMockSynonymFilter, TestAssertingPointsFormat, TestAssertingLeafReader, TestDisableFsyncFS, TestExtrasFS, TestHandleLimitFS]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestHandleLimitFS -Dtests.seed=26BD3F668E4E4360 -Dtests.slow=true -Dtests.locale=fr-CA -Dtests.timezone=America/Porto_Acre -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | TestHandleLimitFS (suite) <<<
[junit4] > Throwable #1: java.io.IOException: Could not remove the following files (in the order of attempts):
[junit4] > C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002: java.nio.file.AccessDeniedException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001\tempDir-002
[junit4] > C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\test-framework\test\J1\temp\lucene.mockfile.TestHandleLimitFS_26BD3F668E4E4360-001
[junit4] > at __randomizedtesting.SeedInfo.seed([26BD3F668E4E4360]:0)
[junit4] > at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] Completed [34/42 (1!)] on J1 in 0.35s, 7 tests, 1 error <<< FAILURES!
[...truncated 71083 lines...]
[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_102) - Build #
6167 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6167/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard1NonLeaderClient
Error Message:
GC overhead limit exceeded
Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at __randomizedtesting.SeedInfo.seed([A1779DCC8FE98CA2:53B5634571832994]:0)
at java.util.HashMap.newNode(HashMap.java:1742)
at java.util.HashMap.putVal(HashMap.java:630)
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at sun.security.util.AlgorithmDecomposer.decompose(AlgorithmDecomposer.java:76)
at sun.security.ssl.SSLAlgorithmDecomposer.decompose(SSLAlgorithmDecomposer.java:248)
at sun.security.util.AbstractAlgorithmConstraints.checkAlgorithm(AbstractAlgorithmConstraints.java:104)
at sun.security.util.DisabledAlgorithmConstraints.permits(DisabledAlgorithmConstraints.java:91)
at sun.security.ssl.SSLAlgorithmConstraints.permits(SSLAlgorithmConstraints.java:144)
at sun.security.ssl.SignatureAndHashAlgorithm.getSupportedAlgorithms(SignatureAndHashAlgorithm.java:158)
at sun.security.ssl.Handshaker.getLocalSupportedSignAlgs(Handshaker.java:442)
at sun.security.ssl.ClientHandshaker.getKickstartMessage(ClientHandshaker.java:1392)
at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1014)
at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1487)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1351)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
Build Log:
[...truncated 10819 lines...]
[junit4] Suite: org.apache.solr.cloud.TestTolerantUpdateProcessorCloud
[junit4] 2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\init-core-data-001
[junit4] 2> 343800 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 343803 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 5 servers in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001
[junit4] 2> 343803 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 343804 INFO (Thread-786) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 343804 INFO (Thread-786) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 343905 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:61825
[junit4] 2> 343917 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a59850910000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 343921 INFO (jetty-launcher-880-thread-1) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 343921 INFO (jetty-launcher-880-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ff4f3a{/solr,null,AVAILABLE}
[junit4] 2> 343924 INFO (jetty-launcher-880-thread-5) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 343925 INFO (jetty-launcher-880-thread-5) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2e0cc3{/solr,null,AVAILABLE}
[junit4] 2> 343926 INFO (jetty-launcher-880-thread-2) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 343927 INFO (jetty-launcher-880-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d43230{/solr,null,AVAILABLE}
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.e.j.s.ServerConnector Started ServerConnector@153c3ee{SSL,[ssl, http/1.1]}{127.0.0.1:61830}
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.e.j.s.Server Started @348714ms
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61830}
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 343928 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T18:40:05.384Z
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.e.j.s.ServerConnector Started ServerConnector@4abf96{SSL,[ssl, http/1.1]}{127.0.0.1:61833}
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.e.j.s.Server Started @348715ms
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61833}
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 343929 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T18:40:05.385Z
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.e.j.s.ServerConnector Started ServerConnector@c60d2c{SSL,[ssl, http/1.1]}{127.0.0.1:61829}
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.e.j.s.Server Started @348718ms
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61829}
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 343931 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T18:40:05.387Z
[junit4] 2> 343936 INFO (jetty-launcher-880-thread-4) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 343936 INFO (jetty-launcher-880-thread-3) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 343938 INFO (jetty-launcher-880-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58265d{/solr,null,AVAILABLE}
[junit4] 2> 343938 INFO (jetty-launcher-880-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@67f866{/solr,null,AVAILABLE}
[junit4] 2> 343941 INFO (jetty-launcher-880-thread-4) [ ] o.e.j.s.ServerConnector Started ServerConnector@a24a7c{SSL,[ssl, http/1.1]}{127.0.0.1:61844}
[junit4] 2> 343941 INFO (jetty-launcher-880-thread-4) [ ] o.e.j.s.Server Started @348728ms
[junit4] 2> 343941 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61844}
[junit4] 2> 343941 INFO (jetty-launcher-880-thread-3) [ ] o.e.j.s.ServerConnector Started ServerConnector@12b8c6c{SSL,[ssl, http/1.1]}{127.0.0.1:61845}
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.e.j.s.Server Started @348728ms
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61845}
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T18:40:05.398Z
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 343942 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T18:40:05.398Z
[junit4] 2> 343954 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 343955 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 343960 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 343961 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 343974 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 343975 INFO (jetty-launcher-880-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61825/solr
[junit4] 2> 343976 INFO (jetty-launcher-880-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61825/solr
[junit4] 2> 343978 INFO (jetty-launcher-880-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61825/solr
[junit4] 2> 343984 INFO (jetty-launcher-880-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61825/solr
[junit4] 2> 344007 INFO (jetty-launcher-880-thread-5) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61825/solr
[junit4] 2> 344016 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a59850910006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 344023 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a5985091000a, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 344082 INFO (jetty-launcher-880-thread-4) [n:127.0.0.1:61844_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61844_solr
[junit4] 2> 344085 INFO (jetty-launcher-880-thread-4) [n:127.0.0.1:61844_solr ] o.a.s.c.Overseer Overseer (id=96727990617047053-127.0.0.1:61844_solr-n_0000000000) starting
[junit4] 2> 344085 INFO (jetty-launcher-880-thread-3) [n:127.0.0.1:61845_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61845_solr
[junit4] 2> 344086 INFO (jetty-launcher-880-thread-1) [n:127.0.0.1:61829_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61829_solr
[junit4] 2> 344086 INFO (jetty-launcher-880-thread-5) [n:127.0.0.1:61830_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61830_solr
[junit4] 2> 344086 INFO (jetty-launcher-880-thread-2) [n:127.0.0.1:61833_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61833_solr
[junit4] 2> 344090 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 344106 INFO (zkCallback-905-thread-1-processing-n:127.0.0.1:61845_solr) [n:127.0.0.1:61845_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 344108 INFO (jetty-launcher-880-thread-5) [n:127.0.0.1:61830_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node3\.
[junit4] 2> 344113 INFO (zkCallback-904-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 344114 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 344114 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 344127 INFO (jetty-launcher-880-thread-2) [n:127.0.0.1:61833_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node2\.
[junit4] 2> 344129 INFO (jetty-launcher-880-thread-3) [n:127.0.0.1:61845_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node5\.
[junit4] 2> 344129 INFO (jetty-launcher-880-thread-1) [n:127.0.0.1:61829_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node1\.
[junit4] 2> 344137 INFO (jetty-launcher-880-thread-4) [n:127.0.0.1:61844_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61844_solr
[junit4] 2> 344140 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 344140 INFO (zkCallback-904-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 344140 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 344142 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 344142 INFO (zkCallback-905-thread-1-processing-n:127.0.0.1:61845_solr) [n:127.0.0.1:61845_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 344155 INFO (jetty-launcher-880-thread-4) [n:127.0.0.1:61844_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node4\.
[junit4] 2> 344160 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a59850910010, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 344164 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
[junit4] 2> 345024 INFO (qtp555947-4031) [n:127.0.0.1:61844_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&collection.configName=solrCloudCollectionConfig&name=test_col&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 345032 INFO (OverseerThreadFactory-1909-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.CreateCollectionCmd Create collection test_col
[junit4] 2> 345166 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard1_replica1&action=CREATE&numShards=2&collection=test_col&shard=shard1&wt=javabin&version=2
[junit4] 2> 345264 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard1_replica2&action=CREATE&numShards=2&collection=test_col&shard=shard1&wt=javabin&version=2
[junit4] 2> 345274 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard2_replica1&action=CREATE&numShards=2&collection=test_col&shard=shard2&wt=javabin&version=2
[junit4] 2> 345330 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr ] o.a.s.h.a.CoreAdminOperation core create command property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard2_replica2&action=CREATE&numShards=2&collection=test_col&shard=shard2&wt=javabin&version=2
[junit4] 2> 345440 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 345440 INFO (zkCallback-904-thread-3-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 345442 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 345442 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346309 WARN (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 346312 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 346314 WARN (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 346314 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 346318 WARN (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 346319 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 346320 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
[junit4] 2> 346321 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
[junit4] 2> 346328 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.IndexSchema [test_col_shard1_replica1] Schema name=test
[junit4] 2> 346329 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.IndexSchema [test_col_shard1_replica2] Schema name=test
[junit4] 2> 346376 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
[junit4] 2> 346395 WARN (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.IndexSchema [test_col_shard1_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 346395 WARN (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.IndexSchema [test_col_shard1_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 346402 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.IndexSchema [test_col_shard2_replica1] Schema name=test
[junit4] 2> 346415 WARN (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
[junit4] 2> 346415 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 346429 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema15.xml instead
[junit4] 2> 346436 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.IndexSchema [test_col_shard2_replica2] Schema name=test
[junit4] 2> 346463 WARN (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.IndexSchema [test_col_shard2_replica1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 346464 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 346465 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 346499 WARN (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.IndexSchema [test_col_shard2_replica2] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
[junit4] 2> 346502 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 346507 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 346507 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
[junit4] 2> 346516 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml to /configs/solrCloudCollectionConfig/schema15.xml.bak
[junit4] 2> 346516 WARN (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.s.ManagedIndexSchemaFactory Error persisting managed schema resource managed-schema
[junit4] 2> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /configs/solrCloudCollectionConfig/schema15.xml
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:245)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:242)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:242)
[junit4] 2> at org.apache.solr.schema.ManagedIndexSchemaFactory.zkUgradeToManagedSchema(ManagedIndexSchemaFactory.java:349)
[junit4] 2> at org.apache.solr.schema.ManagedIndexSchemaFactory.upgradeToManagedSchema(ManagedIndexSchemaFactory.java:268)
[junit4] 2> at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:186)
[junit4] 2> at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
[junit4] 2> at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
[junit4] 2> at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:107)
[junit4] 2> at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:78)
[junit4] 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:862)
[junit4] 2> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:801)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:85)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:374)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:365)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:156)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:659)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:302)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:253)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:518)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 346520 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 x:test_col_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica2' using configuration from collection test_col
[junit4] 2> 346520 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 x:test_col_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard1_replica1' using configuration from collection test_col
[junit4] 2> 346521 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.SolrCore [[test_col_shard1_replica1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node4\test_col_shard1_replica1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node4\.\test_col_shard1_replica1\data\]
[junit4] 2> 346521 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.SolrCore [[test_col_shard1_replica2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node2\test_col_shard1_replica2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node2\.\test_col_shard1_replica2\data\]
[junit4] 2> 346521 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be45a5
[junit4] 2> 346521 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be45a5
[junit4] 2> 346524 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=158345057, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 346524 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=158345057, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 346530 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 346536 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
[junit4] 2> 346536 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
[junit4] 2> 346537 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 x:test_col_shard2_replica1] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica1' using configuration from collection test_col
[junit4] 2> 346537 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.SolrCore [[test_col_shard2_replica1] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node3\test_col_shard2_replica1], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node3\.\test_col_shard2_replica1\data\]
[junit4] 2> 346537 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be45a5
[junit4] 2> 346539 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=158345057, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 346551 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.ManagedIndexSchema Managed schema znode at /configs/solrCloudCollectionConfig/managed-schema already exists - no need to create it
[junit4] 2> 346551 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, the non-managed schema /configs/solrCloudCollectionConfig/schema15.xml no longer exists.
[junit4] 2> 346552 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 x:test_col_shard2_replica2] o.a.s.c.CoreContainer Creating SolrCore 'test_col_shard2_replica2' using configuration from collection test_col
[junit4] 2> 346552 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.SolrCore [[test_col_shard2_replica2] ] Opening new SolrCore at [C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node1\test_col_shard2_replica2], dataDir=[C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001\tempDir-001\node1\.\test_col_shard2_replica2\data\]
[junit4] 2> 346552 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be45a5
[junit4] 2> 346556 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=158345057, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 346563 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 346563 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 346565 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 346565 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 346566 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.21484375, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=19.007950752293773, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 346566 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 346566 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 346568 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 346568 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 346568 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.21484375, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=19.007950752293773, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 346579 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 346579 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 346581 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 346581 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 346583 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 346583 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 346583 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.21484375, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=19.007950752293773, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 346585 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 346585 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 346586 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=49, maxMergedSegmentMB=69.21484375, floorSegmentMB=1.158203125, forceMergeDeletesPctAllowed=19.007950752293773, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 346588 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@101256c[test_col_shard1_replica1] main]
[junit4] 2> 346588 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@ad3444[test_col_shard1_replica2] main]
[junit4] 2> 346589 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 346589 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 346590 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 346590 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 346590 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 346590 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 346591 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@16ea1b7[test_col_shard2_replica2] main]
[junit4] 2> 346592 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@533549[test_col_shard2_replica1] main]
[junit4] 2> 346593 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 346593 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/solrCloudCollectionConfig
[junit4] 2> 346593 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 346594 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 346594 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/solrCloudCollectionConfig
[junit4] 2> 346594 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 346595 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547647852794085376
[junit4] 2> 346595 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547647852794085376
[junit4] 2> 346596 INFO (searcherExecutor-1912-thread-1-processing-n:127.0.0.1:61833_solr x:test_col_shard1_replica2 s:shard1 c:test_col r:core_node2) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.SolrCore [test_col_shard1_replica2] Registered new searcher Searcher@ad3444[test_col_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 346596 INFO (searcherExecutor-1913-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.SolrCore [test_col_shard1_replica1] Registered new searcher Searcher@101256c[test_col_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 346599 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547647852798279680
[junit4] 2> 346599 INFO (searcherExecutor-1915-thread-1-processing-n:127.0.0.1:61829_solr x:test_col_shard2_replica2 s:shard2 c:test_col r:core_node4) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.SolrCore [test_col_shard2_replica2] Registered new searcher Searcher@16ea1b7[test_col_shard2_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 346610 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 346610 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 346610 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 346610 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:61833/solr/test_col_shard1_replica2/
[junit4] 2> 346610 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=test_col_shard1_replica2 url=https://127.0.0.1:61833/solr START replicas=[https://127.0.0.1:61844/solr/test_col_shard1_replica1/] nUpdates=100
[junit4] 2> 346610 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=test_col_shard1_replica2 url=https://127.0.0.1:61833/solr DONE. We have no versions. sync failed.
[junit4] 2> 346611 INFO (searcherExecutor-1914-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.SolrCore [test_col_shard2_replica1] Registered new searcher Searcher@533549[test_col_shard2_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 346611 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1547647852810862592
[junit4] 2> 346630 INFO (qtp555947-4036) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.S.Request [test_col_shard1_replica1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 346631 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 346631 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 346642 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:61833/solr/test_col_shard1_replica2/ shard1
[junit4] 2> 346747 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346747 INFO (zkCallback-904-thread-3-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346747 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346747 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346794 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 346799 INFO (qtp26371412-4016) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard1_replica2&action=CREATE&numShards=2&collection=test_col&shard=shard1&wt=javabin&version=2} status=0 QTime=1534
[junit4] 2> 346899 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346899 INFO (zkCallback-904-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346899 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 346899 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347111 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 347111 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 347111 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:61829/solr/test_col_shard2_replica2/
[junit4] 2> 347111 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.PeerSync PeerSync: core=test_col_shard2_replica2 url=https://127.0.0.1:61829/solr START replicas=[https://127.0.0.1:61830/solr/test_col_shard2_replica1/] nUpdates=100
[junit4] 2> 347112 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.PeerSync PeerSync: core=test_col_shard2_replica2 url=https://127.0.0.1:61829/solr DONE. We have no versions. sync failed.
[junit4] 2> 347123 INFO (qtp8849263-4060) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.S.Request [test_col_shard2_replica1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 347123 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 347123 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 347126 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:61829/solr/test_col_shard2_replica2/ shard2
[junit4] 2> 347233 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347233 INFO (zkCallback-904-thread-3-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347233 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347233 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347279 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 347284 INFO (qtp379183-4001) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard2_replica2&action=CREATE&numShards=2&collection=test_col&shard=shard2&wt=javabin&version=2} status=0 QTime=1954
[junit4] 2> 347385 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347386 INFO (zkCallback-904-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347386 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347386 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347610 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.ZkController Core needs to recover:test_col_shard1_replica1
[junit4] 2> 347611 INFO (updateExecutor-896-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 347611 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 347611 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 347611 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[test_col_shard1_replica1]
[junit4] 2> 347611 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 347611 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Publishing state of core [test_col_shard1_replica1] as recovering, leader is [https://127.0.0.1:61833/solr/test_col_shard1_replica2/] and I am [https://127.0.0.1:61844/solr/test_col_shard1_replica1/]
[junit4] 2> 347614 INFO (qtp555947-4050) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard1_replica1&action=CREATE&numShards=2&collection=test_col&shard=shard1&wt=javabin&version=2} status=0 QTime=2448
[junit4] 2> 347619 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:61833/solr]; [WaitForState: action=PREPRECOVERY&core=test_col_shard1_replica2&nodeName=127.0.0.1:61844_solr&coreNodeName=core_node1&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 347628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node1, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 347628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see test_col_shard1_replica2 (shard1 of test_col) have state: recovering
[junit4] 2> 347628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=test_col, shard=shard1, thisCore=test_col_shard1_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:61844_solr, coreNodeName=core_node1, onlyIfActiveCheckResult=false, nodeProps: core_node1:{"core":"test_col_shard1_replica1","base_url":"https://127.0.0.1:61844/solr","node_name":"127.0.0.1:61844_solr","state":"down"}
[junit4] 2> 347639 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.ZkController Core needs to recover:test_col_shard2_replica1
[junit4] 2> 347640 INFO (updateExecutor-898-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 347641 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 347641 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 347641 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[test_col_shard2_replica1]
[junit4] 2> 347641 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 347641 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Publishing state of core [test_col_shard2_replica1] as recovering, leader is [https://127.0.0.1:61829/solr/test_col_shard2_replica2/] and I am [https://127.0.0.1:61830/solr/test_col_shard2_replica1/]
[junit4] 2> 347644 INFO (qtp8849263-4007) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&qt=/admin/cores&collection.configName=solrCloudCollectionConfig&name=test_col_shard2_replica1&action=CREATE&numShards=2&collection=test_col&shard=shard2&wt=javabin&version=2} status=0 QTime=2370
[junit4] 2> 347644 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Sending prep recovery command to [https://127.0.0.1:61829/solr]; [WaitForState: action=PREPRECOVERY&core=test_col_shard2_replica2&nodeName=127.0.0.1:61830_solr&coreNodeName=core_node3&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 347656 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
[junit4] 2> 347657 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.h.a.PrepRecoveryOp Will wait a max of 183 seconds to see test_col_shard2_replica2 (shard2 of test_col) have state: recovering
[junit4] 2> 347657 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=test_col, shard=shard2, thisCore=test_col_shard2_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:61830_solr, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"test_col_shard2_replica1","base_url":"https://127.0.0.1:61830/solr","node_name":"127.0.0.1:61830_solr","state":"down"}
[junit4] 2> 347660 INFO (qtp555947-4031) [n:127.0.0.1:61844_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 347755 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347755 INFO (zkCallback-904-thread-1-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347755 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 347755 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 348628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=test_col, shard=shard1, thisCore=test_col_shard1_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:61844_solr, coreNodeName=core_node1, onlyIfActiveCheckResult=false, nodeProps: core_node1:{"core":"test_col_shard1_replica1","base_url":"https://127.0.0.1:61844/solr","node_name":"127.0.0.1:61844_solr","state":"recovering"}
[junit4] 2> 348628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 348628 INFO (qtp26371412-4056) [n:127.0.0.1:61833_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:61844_solr&onlyIfLeaderActive=true&core=test_col_shard1_replica2&coreNodeName=core_node1&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1001
[junit4] 2> 348658 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=test_col, shard=shard2, thisCore=test_col_shard2_replica2, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:61830_solr, coreNodeName=core_node3, onlyIfActiveCheckResult=false, nodeProps: core_node3:{"core":"test_col_shard2_replica1","base_url":"https://127.0.0.1:61830/solr","node_name":"127.0.0.1:61830_solr","state":"recovering"}
[junit4] 2> 348658 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 348658 INFO (qtp379183-3999) [n:127.0.0.1:61829_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:61830_solr&onlyIfLeaderActive=true&core=test_col_shard2_replica2&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1002
[junit4] 2> 349038 INFO (OverseerCollectionConfigSetProcessor-96727990617047053-127.0.0.1:61844_solr-n_0000000000) [n:127.0.0.1:61844_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 355632 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:61833/solr/test_col_shard1_replica2/] - recoveringAfterStartup=[true]
[junit4] 2> 355632 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.PeerSync PeerSync: core=test_col_shard1_replica1 url=https://127.0.0.1:61844/solr START replicas=[https://127.0.0.1:61833/solr/test_col_shard1_replica2/] nUpdates=100
[junit4] 2> 355641 INFO (qtp26371412-4017) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 355641 INFO (qtp26371412-4017) [n:127.0.0.1:61833_solr c:test_col s:shard1 r:core_node2 x:test_col_shard1_replica2] o.a.s.c.S.Request [test_col_shard1_replica2] webapp=/solr path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=2
[junit4] 2> 355643 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 355643 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 355643 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 355643 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 355644 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 355644 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 355644 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 355644 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 355644 INFO (recoveryExecutor-897-thread-1-processing-n:127.0.0.1:61844_solr x:test_col_shard1_replica1 s:shard1 c:test_col r:core_node1) [n:127.0.0.1:61844_solr c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 355647 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355647 INFO (zkCallback-904-thread-2-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355647 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355647 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355659 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [https://127.0.0.1:61829/solr/test_col_shard2_replica2/] - recoveringAfterStartup=[true]
[junit4] 2> 355659 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.PeerSync PeerSync: core=test_col_shard2_replica1 url=https://127.0.0.1:61830/solr START replicas=[https://127.0.0.1:61829/solr/test_col_shard2_replica2/] nUpdates=100
[junit4] 2> 355666 INFO (qtp379183-4047) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 355666 INFO (qtp379183-4047) [n:127.0.0.1:61829_solr c:test_col s:shard2 r:core_node4 x:test_col_shard2_replica2] o.a.s.c.S.Request [test_col_shard2_replica2] webapp=/solr path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 355668 INFO (recoveryExecutor-899-thread-1-processing-n:127.0.0.1:61830_solr x:test_col_shard2_replica1 s:shard2 c:test_col r:core_node3) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 355774 INFO (zkCallback-902-thread-1-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355775 INFO (zkCallback-904-thread-2-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355775 INFO (zkCallback-903-thread-1-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 355775 INFO (zkCallback-906-thread-1-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 356666 INFO (qtp555947-4031) [n:127.0.0.1:61844_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&property.schema=schema15.xml&property.config=solrconfig-distrib-update-processor-chains.xml&collection.configName=solrCloudCollectionConfig&name=test_col&action=CREATE&
[...truncated too long message...]
c:test_col s:shard1 r:core_node1 x:test_col_shard1_replica1] o.a.s.c.SolrCore [test_col_shard1_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@10701d3
[junit4] 2> 443032 INFO (coreCloseExecutor-1928-thread-1) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.c.SolrCore [test_col_shard2_replica1] CLOSING SolrCore org.apache.solr.core.SolrCore@1290717
[junit4] 2> 443036 INFO (coreCloseExecutor-1928-thread-1) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 443036 INFO (coreCloseExecutor-1928-thread-1) [n:127.0.0.1:61830_solr c:test_col s:shard2 r:core_node3 x:test_col_shard2_replica1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1ce567c
[junit4] 2> 443040 INFO (zkCallback-904-thread-4-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
[junit4] 2> 443410 WARN (zkCallback-905-thread-2-processing-n:127.0.0.1:61845_solr) [n:127.0.0.1:61845_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 443410 INFO (zkCallback-902-thread-2-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 443411 INFO (zkCallback-904-thread-6-processing-n:127.0.0.1:61844_solr) [n:127.0.0.1:61844_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [4])
[junit4] 2> 443411 INFO (zkCallback-906-thread-2-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 443411 INFO (zkCallback-903-thread-2-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test_col/state.json] for collection [test_col] has occurred - updating... (live nodes size: [5])
[junit4] 2> 443413 INFO (jetty-launcher-880-thread-8) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@67f866{/solr,null,UNAVAILABLE}
[junit4] 2> 443422 INFO (zkCallback-903-thread-3-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
[junit4] 2> 443422 INFO (zkCallback-906-thread-3-processing-n:127.0.0.1:61830_solr) [n:127.0.0.1:61830_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
[junit4] 2> 443422 INFO (zkCallback-902-thread-4-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
[junit4] 2> 443468 INFO (jetty-launcher-880-thread-10) [ ] o.a.s.c.Overseer Overseer (id=96727990617047053-127.0.0.1:61844_solr-n_0000000000) closing
[junit4] 2> 443481 INFO (OverseerStateUpdate-96727990617047053-127.0.0.1:61844_solr-n_0000000000) [n:127.0.0.1:61844_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:61844_solr
[junit4] 2> 443577 INFO (jetty-launcher-880-thread-10) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@58265d{/solr,null,UNAVAILABLE}
[junit4] 2> 443579 INFO (zkCallback-903-thread-2-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1)
[junit4] 2> 443582 INFO (jetty-launcher-880-thread-6) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2e0cc3{/solr,null,UNAVAILABLE}
[junit4] 2> 443584 INFO (zkCallback-903-thread-3-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61833_solr
[junit4] 2> 443593 WARN (zkCallback-903-thread-2-processing-n:127.0.0.1:61833_solr) [n:127.0.0.1:61833_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 445075 WARN (zkCallback-902-thread-4-processing-n:127.0.0.1:61829_solr) [n:127.0.0.1:61829_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 445075 INFO (jetty-launcher-880-thread-9) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@ff4f3a{/solr,null,UNAVAILABLE}
[junit4] 2> 445094 INFO (jetty-launcher-880-thread-7) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d43230{/solr,null,UNAVAILABLE}
[junit4] 2> 445096 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61825 61825
[junit4] 2> 445099 INFO (Thread-786) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:61825 61825
[junit4] 2> 445101 WARN (Thread-786) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 5 /solr/security.json
[junit4] 2> 4 /solr/configs/solrCloudCollectionConfig/managed-schema
[junit4] 2> 4 /solr/configs/solrCloudCollectionConfig
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 4 /solr/collections/test_col/state.json
[junit4] 2> 2 /solr/overseer_elect/election/96727990617047053-127.0.0.1:61844_solr-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 87 /solr/overseer/collection-queue-work
[junit4] 2> 21 /solr/overseer/queue
[junit4] 2> 6 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2> 3 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 445101 INFO (SUITE-TestTolerantUpdateProcessorCloud-seed#[A1779DCC8FE98CA2]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.cloud.TestTolerantUpdateProcessorCloud_A1779DCC8FE98CA2-001
[junit4] 2> Oct 08, 2016 6:41:46 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=cs, timezone=SystemV/MST7
[junit4] 2> NOTE: Windows 10 10.0 x86/Oracle Corporation 1.8.0_102 (32-bit)/cpus=3,threads=1,free=220805280,total=477364224
[junit4] 2> NOTE: All tests run in this JVM: [TestDistributedGrouping, TestNumericTerms64, BJQParserTest, TlogReplayBufferedWhileIndexingTest, SuggestComponentContextFilterQueryTest, EchoParamsTest, TestFastWriter, SharedFSAutoReplicaFailoverTest, SaslZkACLProviderTest, SolrCoreCheckLockOnStartupTest, RollingRestartTest, TestMaxScoreQueryParser, BasicDistributedZk2Test, SignatureUpdateProcessorFactoryTest, LeaderInitiatedRecoveryOnShardRestartTest, TestSubQueryTransformerCrossCore, SpellCheckComponentTest, CheckHdfsIndexTest, TestRestManager, ReplicationFactorTest, TestReloadAndDeleteDocs, UUIDFieldTest, ConnectionReuseTest, TestDFRSimilarityFactory, TestQueryWrapperFilter, TestSolrConfigHandlerCloud, TestBulkSchemaConcurrent, TestBadConfig, ConvertedLegacyTest, TestQueryUtils, TestAnalyzedSuggestions, TestFieldSortValues, TestDocSet, CircularListTest, CSVRequestHandlerTest, TestSolrIndexConfig, NotRequiredUniqueKeyTest, TestRandomFlRTGCloud, UpdateRequestProcessorFactoryTest, TestExclusionRuleCollectionAccess, TestDocBasedVersionConstraints, TestRestoreCore, TestDistributedStatsComponentCardinality, AnalyticsQueryTest, TestRequestForwarding, TestTolerantUpdateProcessorCloud]
[junit4] Completed [72/635 (1!)] on J1 in 101.83s, 19 tests, 1 error <<< FAILURES!
[...truncated 1826 lines...]
[junit4] JVM J1: stdout was not empty, see: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\temp\junit4-J1-20161008_183416_542.sysout
[junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
[junit4] Dumping heap to C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\heapdumps\java_pid6836.hprof ...
[junit4] Heap dump file created [438613486 bytes in 2.109 secs]
[junit4] <<< JVM J1: EOF ----
[...truncated 11027 lines...]
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:765: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:717: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid6836.hprof
Total time: 92 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_102) - Build #
6166 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6166/
Java: 32bit/jdk1.8.0_102 -client -XX:+UseG1GC
All tests passed
Build Log:
[...truncated 53365 lines...]
-ecj-javadoc-lint-src:
[mkdir] Created dir: C:\Users\jenkins\AppData\Local\Temp\ecj526087130
[ecj-lint] Compiling 127 source files to C:\Users\jenkins\AppData\Local\Temp\ecj526087130
[ecj-lint] ----------
[ecj-lint] 1. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 21)
[ecj-lint] import java.util.Collection;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.Collection is never used
[ecj-lint] ----------
[ecj-lint] 2. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 22)
[ecj-lint] import java.util.HashSet;
[ecj-lint] ^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.HashSet is never used
[ecj-lint] ----------
[ecj-lint] 3. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 24)
[ecj-lint] import java.util.List;
[ecj-lint] ^^^^^^^^^^^^^^
[ecj-lint] The import java.util.List is never used
[ecj-lint] ----------
[ecj-lint] 4. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 25)
[ecj-lint] import java.util.Set;
[ecj-lint] ^^^^^^^^^^^^^
[ecj-lint] The import java.util.Set is never used
[ecj-lint] ----------
[ecj-lint] 5. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 27)
[ecj-lint] import org.apache.lucene.analysis.Analyzer;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.analysis.Analyzer is never used
[ecj-lint] ----------
[ecj-lint] 6. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 33)
[ecj-lint] import org.apache.lucene.index.IndexableField;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.index.IndexableField is never used
[ecj-lint] ----------
[ecj-lint] 6 problems (6 errors)
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:765: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:101: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build.xml:204: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2177: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1992: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2031: Compile failed; see the compiler error output for details.
Total time: 90 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build #
6165 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6165/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail
Error Message:
expected:<200> but was:<404>
Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at __randomizedtesting.SeedInfo.seed([DD0F25CE4E18E341:B5B010E49E82F1AD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 12213 lines...]
[junit4] Suite: org.apache.solr.cloud.TestSolrCloudWithDelegationTokens
[junit4] 2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithDelegationTokens_DD0F25CE4E18E341-001\init-core-data-001
[junit4] 2> 2489043 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 2489049 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithDelegationTokens_DD0F25CE4E18E341-001\tempDir-001
[junit4] 2> 2489049 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2489050 INFO (Thread-4898) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2489050 INFO (Thread-4898) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2489151 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:58388
[junit4] 2> 2489160 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 2489160 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.Server jetty-9.3.8.v20160314
[junit4] 2> 2489162 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28265efb{/solr,null,AVAILABLE}
[junit4] 2> 2489162 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5f0d2ffb{/solr,null,AVAILABLE}
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.ServerConnector Started ServerConnector@67dedf6d{HTTP/1.1,[http/1.1]}{127.0.0.1:58393}
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.ServerConnector Started ServerConnector@6ca39ede{HTTP/1.1,[http/1.1]}{127.0.0.1:58392}
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.Server Started @2494268ms
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.Server Started @2494268ms
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58393}
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58392}
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.0.0
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2489163 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T01:12:23.403Z
[junit4] 2> 2489164 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2489164 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2016-10-08T01:12:23.404Z
[junit4] 2> 2489167 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2489167 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2489187 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2489196 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58388/solr
[junit4] 2> 2489196 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58388/solr
[junit4] 2> 2489227 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58392_solr
[junit4] 2> 2489228 INFO (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58393_solr
[junit4] 2> 2489228 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.c.Overseer Overseer (id=96723870894915589-127.0.0.1:58392_solr-n_0000000000) starting
[junit4] 2> 2489229 INFO (zkCallback-8973-thread-1-processing-n:127.0.0.1:58392_solr) [n:127.0.0.1:58392_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2489230 INFO (zkCallback-8974-thread-1-processing-n:127.0.0.1:58393_solr) [n:127.0.0.1:58393_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2489238 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58392_solr
[junit4] 2> 2489240 INFO (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 2489240 INFO (zkCallback-8974-thread-1-processing-n:127.0.0.1:58393_solr) [n:127.0.0.1:58393_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2489240 INFO (zkCallback-8973-thread-1-processing-n:127.0.0.1:58392_solr) [n:127.0.0.1:58392_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2489263 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.c.CoreContainer Authentication plugin class obtained from system property 'authenticationPlugin': org.apache.solr.security.HttpParamDelegationTokenPlugin
[junit4] 2> 2489303 INFO (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.s.s.KerberosPlugin Params: {token.valid=30, delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, token.validity=36000, signer.secret.provider.zookeeper.path=/token, type=kerberos, zk-dt-secret-manager.enable=true, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper}
[junit4] 2> 2489303 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.s.KerberosPlugin Params: {token.valid=30, delegation-token.token-kind=solr-dt, cookie.domain=127.0.0.1, cookie.path=/, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, token.validity=36000, signer.secret.provider.zookeeper.path=/token, type=kerberos, zk-dt-secret-manager.enable=true, kerberos.name.rules=DEFAULT, signer.secret.provider=zookeeper}
[junit4] 2> 2489624 INFO (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2489624 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2489685 INFO (jetty-launcher-8963-thread-2-EventThread) [n:127.0.0.1:58393_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2489685 INFO (jetty-launcher-8963-thread-1-EventThread) [n:127.0.0.1:58392_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2489891 ERROR (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.h.u.Shell Failed to locate the winutils binary in the hadoop binary path
[junit4] 2> java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
[junit4] 2> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
[junit4] 2> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
[junit4] 2> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
[junit4] 2> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
[junit4] 2> at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1437)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:115)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:148)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:118)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin$HttpParamDelegationTokenAuthenticationHandler.init(HttpParamDelegationTokenPlugin.java:178)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:238)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:209)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.initializeAuthHandler(DelegationTokenKerberosFilter.java:137)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:227)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:175)
[junit4] 2> at org.apache.solr.security.DelegationTokenKerberosFilter.init(DelegationTokenKerberosFilter.java:66)
[junit4] 2> at org.apache.solr.security.HttpParamDelegationTokenPlugin.init(HttpParamDelegationTokenPlugin.java:134)
[junit4] 2> at org.apache.solr.core.CoreContainer.initializeAuthenticationPlugin(CoreContainer.java:320)
[junit4] 2> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:468)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:203)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:152)
[junit4] 2> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1542)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1576)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1268)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1113)
[junit4] 2> at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:447)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:258)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
[junit4] 2> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:342)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:315)
[junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:353)
[junit4] 2> at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:220)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2490112 INFO (jetty-launcher-8963-thread-1) [n:127.0.0.1:58392_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithDelegationTokens_DD0F25CE4E18E341-001\tempDir-001\node1\.
[junit4] 2> 2490112 INFO (jetty-launcher-8963-thread-2) [n:127.0.0.1:58393_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithDelegationTokens_DD0F25CE4E18E341-001\tempDir-001\node2\.
[junit4] 2> 2490142 INFO (TEST-TestSolrCloudWithDelegationTokens.testZNodePaths-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testZNodePaths
[junit4] 2> 2491262 INFO (TEST-TestSolrCloudWithDelegationTokens.testZNodePaths-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testZNodePaths
[junit4] 2> 2491284 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenewFail
[junit4] 2> 2491302 WARN (qtp645236469-40724) [ ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo tries to renew a token with renewer bar
[junit4] 2> 2491303 WARN (qtp645236469-40724) [ ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.kerberosplugin.requestcontinues
[junit4] 2> 2491322 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491326 WARN (qtp1461279889-40720) [ ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: foo tries to renew a token with renewer bar
[junit4] 2> 2491326 WARN (qtp1461279889-40720) [ ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.kerberosplugin.requestcontinues
[junit4] 2> 2491327 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491332 WARN (qtp645236469-40724) [ ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
[junit4] 2> 2491332 WARN (qtp645236469-40724) [ ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.kerberosplugin.requestcontinues
[junit4] 2> 2491332 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491333 WARN (qtp1461279889-40722) [ ] o.a.h.s.a.s.AuthenticationFilter Authentication exception: org.apache.hadoop.security.AccessControlException: notRenewUser tries to renew a token with renewer renewUser
[junit4] 2> 2491333 WARN (qtp1461279889-40722) [ ] o.a.s.s.KerberosPlugin Could not find org.apache.solr.security.kerberosplugin.requestcontinues
[junit4] 2> 2491334 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491334 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenewFail-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenewFail
[junit4] 2> 2491365 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenVerify
[junit4] 2> 2491402 INFO (qtp645236469-40715) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2ScFigFXxeWrBQQEFKGpISVCv0Bocg4rEjk8LvvLYiuWB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491402 INFO (qtp645236469-40715) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2ScFigFXxeWrBQQEFKGpISVCv0Bocg4rEjk8LvvLYiuWB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491408 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491411 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491435 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe000b, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491437 INFO (qtp1461279889-40726) [n:127.0.0.1:58393_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2ScFigFXxeWrBQQEFKGpISVCv0Bocg4rEjk8LvvLYiuWB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491437 INFO (qtp1461279889-40726) [n:127.0.0.1:58393_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2ScFigFXxeWrBQQEFKGpISVCv0Bocg4rEjk8LvvLYiuWB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491440 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491440 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenVerify-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenVerify
[junit4] 2> 2491461 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationOtherOp
[junit4] 2> 2491463 INFO (qtp645236469-40728) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params op=someSolrOperation&action=LIST&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491463 INFO (qtp645236469-40728) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={op=someSolrOperation&action=LIST&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491464 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth="u=bar&p=bar&t=test&e=1475925145703&s=8bFQx/osf5eTkfwqMmVulvfp8VQ="; Path=/; Domain=127.0.0.1; Expires= , 08- -2016 11:12:25 GMT; HttpOnly". Invalid 'expires' attribute: , 08- -2016 11:12:25 GMT
[junit4] 2> 2491464 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationOtherOp-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationOtherOp
[junit4] 2> 2491484 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenRenew
[junit4] 2> 2491510 INFO (qtp645236469-40725) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2Sd8igFXxeWrfAUEFHwxis9H9JKCGZtRYAyUuukM42WRB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491510 INFO (qtp645236469-40725) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2Sd8igFXxeWrfAUEFHwxis9H9JKCGZtRYAyUuukM42WRB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491514 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491516 INFO (qtp1461279889-40720) [n:127.0.0.1:58393_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2Sd8igFXxeWrfAUEFHwxis9H9JKCGZtRYAyUuukM42WRB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491516 INFO (qtp1461279889-40720) [n:127.0.0.1:58393_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2Sd8igFXxeWrfAUEFHwxis9H9JKCGZtRYAyUuukM42WRB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491524 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491526 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491526 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe000c, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491543 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SepigFXxeWrqQYEFGm4wLthRAH8zkRCad1hzR44KkNkB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491543 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SepigFXxeWrqQYEFGm4wLthRAH8zkRCad1hzR44KkNkB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491548 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491551 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491552 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe000d, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491555 INFO (qtp1461279889-40718) [n:127.0.0.1:58393_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SepigFXxeWrqQYEFGm4wLthRAH8zkRCad1hzR44KkNkB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491555 INFO (qtp1461279889-40718) [n:127.0.0.1:58393_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SepigFXxeWrqQYEFGm4wLthRAH8zkRCad1hzR44KkNkB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491557 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491570 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfGigFXxeWrxgcEFCpqkeRCfZH4PCsekvXsdgNOcrMuB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491570 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfGigFXxeWrxgcEFCpqkeRCfZH4PCsekvXsdgNOcrMuB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491572 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491577 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491580 INFO (qtp645236469-40727) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfGigFXxeWrxgcEFCpqkeRCfZH4PCsekvXsdgNOcrMuB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491580 INFO (qtp645236469-40727) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfGigFXxeWrxgcEFCpqkeRCfZH4PCsekvXsdgNOcrMuB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491581 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe000e, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491586 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491589 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491589 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe000f, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491603 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491606 INFO (qtp645236469-40724) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfmigFXxeWr5ggEFLNQbe9PpemyDBVqnzXj-rMxbaMYB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491606 INFO (qtp645236469-40724) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfmigFXxeWr5ggEFLNQbe9PpemyDBVqnzXj-rMxbaMYB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491607 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0010, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491612 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491615 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491620 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491623 INFO (qtp645236469-40727) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfmigFXxeWr5ggEFLNQbe9PpemyDBVqnzXj-rMxbaMYB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491623 INFO (qtp645236469-40727) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SfmigFXxeWr5ggEFLNQbe9PpemyDBVqnzXj-rMxbaMYB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491624 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0012, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491626 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491627 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenRenew-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenRenew
[junit4] 2> 2491647 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenSolrClient
[junit4] 2> 2491652 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491655 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SgfigFXxeWsHwkEFCoos7ZAmrb2ccPxJ1JToDIVp9wtB3NvbHItZHQA&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2491655 INFO (qtp645236469-40721) [n:127.0.0.1:58392_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=LIST&delegation=GgADYmFyA2JhcgCKAVeh2SgfigFXxeWsHwkEFCoos7ZAmrb2ccPxJ1JToDIVp9wtB3NvbHItZHQA&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2491656 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClient-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenSolrClient
[junit4] 2> 2491684 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancel
[junit4] 2> 2491710 ERROR (qtp645236469-40719) [ ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_10]
[junit4] 2> 2491739 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491742 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491743 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0013, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491745 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491757 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491760 ERROR (qtp645236469-40719) [ ] o.a.h.s.t.d.ZKDelegationTokenSecretManager No node in path [/ZKDTSMTokensRoot/DT_11]
[junit4] 2> 2491765 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491769 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2491772 WARN (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.h.c.p.ResponseProcessCookies Invalid cookie header: "Set-Cookie: hadoop.auth=; Path=/; Domain=127.0.0.1; Expires= , 01- -1970 00:00:00 GMT; HttpOnly". Invalid 'expires' attribute: , 01- -1970 00:00:00 GMT
[junit4] 2> 2491772 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0015, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491773 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancel-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancel
[junit4] 2> 2491793 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testDelegationTokenCancelFail
[junit4] 2> 2491804 INFO (TEST-TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail-seed#[DD0F25CE4E18E341]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testDelegationTokenCancelFail
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestSolrCloudWithDelegationTokens -Dtests.method=testDelegationTokenCancelFail -Dtests.seed=DD0F25CE4E18E341 -Dtests.slow=true -Dtests.locale=ar-EG -Dtests.timezone=Pacific/Rarotonga -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 0.03s J0 | TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<200> but was:<404>
[junit4] > at __randomizedtesting.SeedInfo.seed([DD0F25CE4E18E341:B5B010E49E82F1AD]:0)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
[junit4] > at org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491807 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.ServerConnector Stopped ServerConnector@6ca39ede{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2491808 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1649924468
[junit4] 2> 2491808 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.ServerConnector Stopped ServerConnector@67dedf6d{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 2491808 INFO (jetty-launcher-8963-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1899865842
[junit4] 2> 2491812 INFO (jetty-launcher-8963-thread-1) [ ] o.a.s.c.Overseer Overseer (id=96723870894915589-127.0.0.1:58392_solr-n_0000000000) closing
[junit4] 2> 2491812 INFO (OverseerStateUpdate-96723870894915589-127.0.0.1:58392_solr-n_0000000000) [n:127.0.0.1:58392_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:58392_solr
[junit4] 2> 2491814 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x157a1d91dfe0006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 2491814 WARN (zkCallback-8974-thread-1-processing-n:127.0.0.1:58393_solr) [n:127.0.0.1:58393_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2491814 ERROR (Thread[Thread-4916,5,TGRP-TestSolrCloudWithDelegationTokens]) [n:127.0.0.1:58393_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2491841 INFO (jetty-launcher-8963-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5f0d2ffb{/solr,null,UNAVAILABLE}
[junit4] 2> 2493315 WARN (zkCallback-8973-thread-1-processing-n:127.0.0.1:58392_solr) [n:127.0.0.1:58392_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2493315 ERROR (Thread[Thread-4915,5,TGRP-TestSolrCloudWithDelegationTokens]) [n:127.0.0.1:58392_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2493318 INFO (jetty-launcher-8963-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@28265efb{/solr,null,UNAVAILABLE}
[junit4] 2> 2493319 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58388 58388
[junit4] 2> 2493320 INFO (Thread-4898) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58388 58388
[junit4] 2> 2493320 WARN (Thread-4898) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 13 /solr/aliases.json
[junit4] 2> 13 /solr/clusterprops.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 13 /solr/clusterstate.json
[junit4] 2> 4 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_7
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_6
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_9
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_8
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_5
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_4
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_12
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_11
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot/DT_10
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 13 /solr/live_nodes
[junit4] 2> 13 /solr/collections
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
[junit4] 2>
[junit4] 2> 2493322 INFO (SUITE-TestSolrCloudWithDelegationTokens-seed#[DD0F25CE4E18E341]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.cloud.TestSolrCloudWithDelegationTokens_DD0F25CE4E18E341-001
[junit4] 2> Oct 08, 2016 1:12:27 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=734, maxMBSortInHeap=6.978534138773725, sim=ClassicSimilarity, locale=ar-EG, timezone=Pacific/Rarotonga
[junit4] 2> NOTE: Windows 10 10.0 amd64/Oracle Corporation 1.8.0_102 (64-bit)/cpus=3,threads=1,free=207799840,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestStressReorder, TestRandomCollapseQParserPlugin, TriLevelCompositeIdRoutingTest, DocValuesTest, TestRTGBase, DOMUtilTest, TestFreeTextSuggestions, DeleteLastCustomShardedReplicaTest, NumericFieldsTest, QueryResultKeyTest, TestDFISimilarityFactory, TestDistributedSearch, MigrateRouteKeyTest, SpatialRPTFieldTypeTest, TestRandomDVFaceting, SpatialFilterTest, TestSolrQueryParserResource, CollectionsAPIAsyncDistributedZkTest, DistributedFacetExistsSmallTest, BaseCdcrDistributedZkTest, SparseHLLTest, TestLeaderElectionZkExpiry, ConjunctionSolrSpellCheckerTest, TestCustomDocTransformer, SpatialHeatmapFacetsTest, TestCopyFieldCollectionResource, HdfsLockFactoryTest, CurrencyFieldXmlFileTest, TestFaceting, SchemaVersionSpecificBehaviorTest, DateFieldTest, DistributedFacetPivotLongTailTest, TestCryptoKeys, DirectSolrConnectionTest, OutOfBoxZkACLAndCredentialsProvidersTest, StandardRequestHandlerTest, TestSimpleQParserPlugin, TestLegacyNumericRangeQueryBuilder, TestCursorMarkWithoutUniqueKey, TestTrieFacet, TestBlobHandler, TestCoreContainer, PeerSyncTest, TestFieldCollectionResource, BlockCacheTest, OpenExchangeRatesOrgProviderTest, CollectionStateFormat2Test, PreAnalyzedUpdateProcessorTest, CloneFieldUpdateProcessorFactoryTest, TestHdfsCloudBackupRestore, TestLRUStatsCache, SolrIndexConfigTest, IndexSchemaRuntimeFieldTest, TestHashPartitioner, TestLockTree, TestSurroundQueryParser, TestSort, MBeansHandlerTest, TestLazyCores, TestGraphMLResponseWriter, TestStressVersions, TestReloadDeadlock, ZkStateReaderTest, TestUpdate, TestDownShardTolerantSearch, OutputWriterTest, TestQuerySenderNoQuery, SolrPluginUtilsTest, TestStressLiveNodes, TestLFUCache, SimpleFacetsTest, TestTolerantSearch, SimplePostToolTest, TestSchemaSimilarityResource, TestConfigOverlay, TestPostingsSolrHighlighter, HighlighterMaxOffsetTest, HdfsRestartWhileUpdatingTest, IndexBasedSpellCheckerTest, BufferStoreTest, SynonymTokenizerTest, SampleTest, BadCopyFieldTest, TestReRankQParserPlugin, TestCollationField, HdfsCollectionsAPIDistributedZkTest, TestSubQueryTransformerDistrib, PKIAuthenticationIntegrationTest, TestPartialUpdateDeduplication, TermsComponentTest, TestLRUCache, HdfsRecoverLeaseTest, SmileWriterTest, TestCSVLoader, RequiredFieldsTest, TestSha256AuthenticationProvider, ShardSplitTest, DataDrivenBlockJoinTest, TestXIncludeConfig, DistribDocExpirationUpdateProcessorTest, TestOverriddenPrefixQueryForCustomFieldType, TestFieldTypeCollectionResource, TestDefaultStatsCache, DistributedQueueTest, MultiTermTest, ConcurrentDeleteAndCreateCollectionTest, TestShortCircuitedRequests, TestElisionMultitermQuery, TestInfoStreamLogging, WrapperMergePolicyFactoryTest, TestExactSharedStatsCache, TestExpandComponent, BinaryUpdateRequestHandlerTest, TestHighlightDedupGrouping, TestSolrDynamicMBean, TestRandomRequestDistribution, TestNRTOpen, DocumentBuilderTest, TestTolerantUpdateProcessorRandomCloud, AnalysisErrorHandlingTest, TestWordDelimiterFilterFactory, TestUninvertingReader, DirectUpdateHandlerTest, SegmentsInfoRequestHandlerTest, SimpleMLTQParserTest, SystemInfoHandlerTest, ClusterStateUpdateTest, CustomCollectionTest, CopyFieldTest, FileUtilsTest, RecoveryZkTest, HdfsBasicDistributedZk2Test, TestConfigSetProperties, TestDocumentBuilder, DistributedMLTComponentTest, SyncSliceTest, CoreAdminRequestStatusTest, OverseerTest, TestCloudSchemaless, TestCharFilters, LeaderElectionTest, TestBlendedInfixSuggestions, TestJsonRequest, TestReqParamsAPI, TestCloudDeleteByQuery, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SolrRequestParserTest, TestXmlQParserPlugin, TestExtendedDismaxParser, HdfsChaosMonkeyNothingIsSafeTest, TestExceedMaxTermLength, TestManagedStopFilterFactory, BasicDistributedZkTest, UnloadDistributedZkTest, TestReplicationHandler, TermVectorComponentDistributedTest, HardAutoCommitTest, SolrCoreTest, StatsComponentTest, QueryElevationComponentTest, TestFunctionQuery, TestBadConfig, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, AnalysisAfterCoreReloadTest, SuggesterFSTTest, CoreAdminHandlerTest, TestStressLucene, PolyFieldTest, TestPseudoReturnFields, QueryEqualityTest, SortByFunctionTest, TestRemoteStreaming, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, DebugComponentTest, LukeRequestHandlerTest, TestReversedWildcardFilterFactory, TestQueryTypes, TestOmitPositions, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FastVectorHighlighterTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, SearchHandlerTest, TestMergePolicyConfig, TestSearchPerf, TestSolrCoreProperties, TestCodecSupport, TestPerFieldSimilarity, TestIBSimilarityFactory, TestFastWriter, TestFastOutputStream, PluginInfoTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, RAMDirectoryFactoryTest, TestSolrJ, TestUtils, ZkNodePropsTest, UUIDFieldTest, CircularListTest, CursorPagingTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestDistributedMissingSort, TestEmbeddedSolrServerConstructors, ActionThrottleTest, AliasIntegrationTest, AssignTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, ConfigSetsAPITest, DeleteNodeTest, DeleteShardTest, DeleteStatusTest, DistributedVersionInfoTest, HttpPartitionTest, LeaderInitiatedRecoveryOnCommitTest, OverseerStatusTest, OverseerTaskQueueTest, RecoveryAfterSoftCommitTest, ReplaceNodeTest, ReplicationFactorTest, RestartWhileUpdatingTest, SSLMigrationTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestCollectionAPI, TestLeaderElectionWithEmptyReplica, TestMiniSolrCloudCluster, TestRandomFlRTGCloud, TestRequestStatusCollectionAPI, TestSolrCloudWithDelegationTokens]
[junit4] Completed [500/635 (1!)] on J0 in 4.80s, 8 tests, 1 failure <<< FAILURES!
[...truncated 41141 lines...]
-ecj-javadoc-lint-src:
[mkdir] Created dir: C:\Users\jenkins\AppData\Local\Temp\ecj16339416
[ecj-lint] Compiling 127 source files to C:\Users\jenkins\AppData\Local\Temp\ecj16339416
[ecj-lint] ----------
[ecj-lint] 1. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 21)
[ecj-lint] import java.util.Collection;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.Collection is never used
[ecj-lint] ----------
[ecj-lint] 2. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 22)
[ecj-lint] import java.util.HashSet;
[ecj-lint] ^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.HashSet is never used
[ecj-lint] ----------
[ecj-lint] 3. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 24)
[ecj-lint] import java.util.List;
[ecj-lint] ^^^^^^^^^^^^^^
[ecj-lint] The import java.util.List is never used
[ecj-lint] ----------
[ecj-lint] 4. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 25)
[ecj-lint] import java.util.Set;
[ecj-lint] ^^^^^^^^^^^^^
[ecj-lint] The import java.util.Set is never used
[ecj-lint] ----------
[ecj-lint] 5. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 27)
[ecj-lint] import org.apache.lucene.analysis.Analyzer;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.analysis.Analyzer is never used
[ecj-lint] ----------
[ecj-lint] 6. ERROR in C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\benchmark\src\java\org\apache\lucene\benchmark\byTask\tasks\ReadTask.java (at line 33)
[ecj-lint] import org.apache.lucene.index.IndexableField;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import org.apache.lucene.index.IndexableField is never used
[ecj-lint] ----------
[ecj-lint] 6 problems (6 errors)
BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:765: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:101: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build.xml:204: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2177: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1992: The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:2031: Compile failed; see the compiler error output for details.
Total time: 80 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any