You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/02/28 13:28:51 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b22) - Build # 11897 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11897/
Java: 32bit/jdk1.8.0_40-ea-b22 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields

Error Message:
SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))

Stack Trace:
org.apache.lucene.index.CorruptIndexException: SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
	at __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
	at org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
	at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
	at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
	at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
	at org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
	at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
	at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
	Suppressed: java.lang.AssertionError
		at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
		at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
		... 48 more




Build Log:
[...truncated 5611 lines...]
   [junit4] Suite: org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimpleTextDocValuesFormat -Dtests.method=testSortedFixedLengthVsStoredFields -Dtests.seed=4879A5F99AD2035B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1.68s J1 | TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields <<<
   [junit4]    > Throwable #1: org.apache.lucene.index.CorruptIndexException: SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
   [junit4]    > 	at org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
   [junit4]    > 	at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
   [junit4]    > 	at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
   [junit4]    > 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
   [junit4]    > 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
   [junit4]    > 	at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
   [junit4]    > 	at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	Suppressed: java.lang.AssertionError
   [junit4]    > 		at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
   [junit4]    > 		at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
   [junit4]    > 		... 48 more
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/test/J1/temp/lucene.codecs.simpletext.TestSimpleTextDocValuesFormat 4879A5F99AD2035B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=et, timezone=Africa/Addis_Ababa
   [junit4]   2> NOTE: Linux 3.13.0-45-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=27185056,total=64880640
   [junit4]   2> NOTE: All tests run in this JVM: [TestSimpleTextFieldInfoFormat, TestVarGapDocFreqIntervalPostingsFormat, TestSimpleTextNormsFormat, TestBloomPostingsFormat, TestSimpleTextPostingsFormat, TestSimpleTextDocValuesFormat]
   [junit4] Completed on J1 in 57.94s, 84 tests, 1 error <<< FAILURES!

[...truncated 50 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:456: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2081: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:58: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958: There were test failures: 19 suites, 501 tests, 1 error, 19 ignored (19 assumptions)

Total time: 27 minutes 47 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b22 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_31) - Build # 11898 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11898/
Java: 64bit/jdk1.8.0_31 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:60811, https://127.0.0.1:50092, https://127.0.0.1:46672]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:60811, https://127.0.0.1:50092, https://127.0.0.1:46672]
	at __randomizedtesting.SeedInfo.seed([A9A51ACA5447870:82CE6E760BB81588]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:349)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1043)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:809)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1125)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:202)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:60811: ADDREPLICA failed to create replica
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:370)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	... 47 more




Build Log:
[...truncated 8871 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/init-core-data-001
   [junit4]   2> 1230 T14 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1277 T14 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1320 T14 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1349 T15 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1447 T14 oasc.ZkTestServer.run start zk server on port:53803
   [junit4]   2> 1473 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1517 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1596 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25d61b7b name:ZooKeeperConnection Watcher:127.0.0.1:53803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1596 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1599 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1601 T14 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1628 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1629 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1632 T25 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31b9261e name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1632 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1633 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1639 T14 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1644 T14 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1647 T14 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1650 T14 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1654 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1666 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1671 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1672 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1676 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1677 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1681 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1682 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1685 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1686 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1692 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1693 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1698 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1698 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1714 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1714 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1722 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1723 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1731 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1732 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1735 T14 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1736 T14 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1742 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1743 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1745 T28 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ba5ceac name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1745 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1746 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2268 T14 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1
   [junit4]   2> 2316 T14 oejul.Log.initialized Logging initialized @3837ms
   [junit4]   2> 2527 T14 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 2617 T14 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@273c2b68{/,null,AVAILABLE}
   [junit4]   2> 3358 T14 oejs.AbstractConnector.doStart Started ServerConnector@13317c92{SSL-http/1.1}{127.0.0.1:60811}
   [junit4]   2> 3359 T14 oejs.Server.doStart Started @4884ms
   [junit4]   2> 3570 T14 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 3680 T14 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3681 T14 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001
   [junit4]   2> 3682 T14 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/'
   [junit4]   2> 3926 T14 oasc.SolrXmlConfig.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/solr.xml
   [junit4]   2> 4211 T14 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores
   [junit4]   2> 4217 T14 oasc.CoreContainer.<init> New CoreContainer 1008613381
   [junit4]   2> 4218 T14 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/]
   [junit4]   2> 4219 T14 oasc.CoreContainer.load loading shared library: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/lib
   [junit4]   2> 4219 T14 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/lib).
   [junit4]   2> 4290 T14 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 4290 T14 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 4298 T14 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 4299 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4299 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 4300 T14 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4300 T14 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4301 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4302 T14 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4302 T14 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4303 T14 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 4349 T14 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4356 T14 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4358 T14 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4361 T14 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 4362 T14 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53803/solr
   [junit4]   2> 4380 T14 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 4381 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 4382 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4386 T43 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57290e4a name:ZooKeeperConnection Watcher:127.0.0.1:53803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4387 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4388 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 4411 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 4420 T46 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73c4abf2 name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4421 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 4430 T14 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 4442 T14 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 4450 T14 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 4460 T14 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 4464 T14 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 4483 T14 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 4486 T14 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60811_
   [junit4]   2> 4498 T14 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60811_
   [junit4]   2> 4504 T14 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 4507 T14 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 4509 T14 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 4519 T14 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:60811_
   [junit4]   2> 4520 T14 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 4523 T14 oasc.Overseer.start Overseer (id=93398250006577156-127.0.0.1:60811_-n_0000000000) starting
   [junit4]   2> 4529 T14 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 4584 T14 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 4683 T48 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 4685 T14 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 4689 T14 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 4694 T14 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 4702 T47 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 4768 T14 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores
   [junit4]   2> 4791 T14 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/
   [junit4]   2> 4792 T14 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 4796 T50 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 4797 T50 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 4802 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 4802 T50 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 4805 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60811",
   [junit4]   2> 	  "node_name":"127.0.0.1:60811_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 4815 T47 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60811",
   [junit4]   2> 	  "node_name":"127.0.0.1:60811_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 4817 T47 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 4825 T47 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 4831 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 5804 T50 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 5804 T50 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 5805 T50 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 5806 T50 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 5807 T50 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 5808 T50 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/'
   [junit4]   2> 5832 T50 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 5857 T50 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 5913 T50 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 5940 T50 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 6007 T50 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 6016 T50 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 6121 T50 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 7225 T50 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 7252 T50 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 7254 T50 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 7445 T50 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 7452 T50 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 7457 T50 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 7472 T50 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 7472 T50 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 7473 T50 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 7475 T50 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 7475 T50 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 7479 T50 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 7515 T50 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 7520 T50 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/data/
   [junit4]   2> 7524 T50 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d2a83
   [junit4]   2> 7586 T50 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/data
   [junit4]   2> 7588 T50 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/data/index/
   [junit4]   2> 7588 T50 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 7602 T50 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/control-001/cores/collection1/data/index
   [junit4]   2> 7603 T50 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=96.6611328125, floorSegmentMB=1.4775390625, forceMergeDeletesPctAllowed=28.852509199252363, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20338264693827954
   [junit4]   2> 7674 T50 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@323ec63b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@59388c3d),segFN=segments_1,generation=1}
   [junit4]   2> 7676 T50 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 7934 T50 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 7962 T50 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 7968 T50 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 7969 T50 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 7969 T50 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 7970 T50 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 8001 T50 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 8001 T50 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 8002 T50 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 8042 T50 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8044 T50 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8049 T50 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8056 T50 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8058 T50 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 8061 T50 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 8082 T50 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 8086 T50 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 8089 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 8091 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 8092 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 8094 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 8116 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 8117 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 8118 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 8126 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 8128 T50 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 8136 T50 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 8234 T50 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8240 T50 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8241 T50 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8243 T50 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8245 T50 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 8268 T50 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 8280 T50 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 8280 T50 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 8282 T50 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=46.650390625, floorSegmentMB=1.572265625, forceMergeDeletesPctAllowed=21.43306147359764, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31335281901899237
   [junit4]   2> 8285 T50 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@323ec63b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@59388c3d),segFN=segments_1,generation=1}
   [junit4]   2> 8285 T50 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 8302 T50 oass.SolrIndexSearcher.<init> Opening Searcher@3e32af7c[collection1] main
   [junit4]   2> 8311 T50 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 8327 T50 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 8327 T50 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 8330 T50 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 8331 T50 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 8352 T50 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 8353 T50 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 8354 T50 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 8354 T50 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 8378 T50 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 8380 T51 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e32af7c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 8383 T50 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 8388 T54 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60811 collection:control_collection shard:shard1
   [junit4]   2> 8389 T14 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 8389 T14 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 8410 T54 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 8426 T54 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 8444 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 8446 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 8450 T54 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 8452 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 8463 T54 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 8463 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@3f663b03 url=https://127.0.0.1:60811/collection1 node=127.0.0.1:60811_ C0_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:60811, node_name=127.0.0.1:60811_, state=down}
   [junit4]   2> 8468 T54 C0 P60811 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60811/collection1/
   [junit4]   2> 8472 T54 C0 P60811 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 8472 T54 C0 P60811 oasc.SyncStrategy.syncToMe https://127.0.0.1:60811/collection1/ has no replicas
   [junit4]   2> 8482 T54 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60811/collection1/ shard1
   [junit4]   2> 8481 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 8490 T57 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21107ead name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 8490 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 8491 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 8497 T14 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 8498 T54 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 8508 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 8510 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60811",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 2
   [junit4]   2> 8511 T14 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 8618 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 8618 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 8662 T54 oasc.ZkController.register We are https://127.0.0.1:60811/collection1/ and leader is https://127.0.0.1:60811/collection1/
   [junit4]   2> 8664 T54 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60811
   [junit4]   2> 8664 T54 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 8665 T54 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 8665 T54 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 8668 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 8670 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60811",
   [junit4]   2> 	  "node_name":"127.0.0.1:60811_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 8671 T47 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:60811",
   [junit4]   2> 	  "node_name":"127.0.0.1:60811_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 8785 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 8785 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 9000 T14 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1
   [junit4]   2> 9002 T14 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 9006 T14 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 9018 T14 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4866275e{/,null,AVAILABLE}
   [junit4]   2> 9041 T14 oejs.AbstractConnector.doStart Started ServerConnector@7c4ed904{SSL-http/1.1}{127.0.0.1:44469}
   [junit4]   2> 9042 T14 oejs.Server.doStart Started @10568ms
   [junit4]   2> 9047 T14 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 9048 T14 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 9049 T14 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001
   [junit4]   2> 9049 T14 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/'
   [junit4]   2> 9075 T14 oasc.SolrXmlConfig.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/solr.xml
   [junit4]   2> 9099 T14 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores
   [junit4]   2> 9099 T14 oasc.CoreContainer.<init> New CoreContainer 1326700639
   [junit4]   2> 9100 T14 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/]
   [junit4]   2> 9100 T14 oasc.CoreContainer.load loading shared library: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/lib
   [junit4]   2> 9101 T14 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/lib).
   [junit4]   2> 9118 T14 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 9119 T14 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 9119 T14 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 9120 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 9120 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 9120 T14 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 9121 T14 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 9121 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 9122 T14 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 9122 T14 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 9123 T14 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 9131 T14 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 9133 T14 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 9134 T14 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 9134 T14 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 9135 T14 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53803/solr
   [junit4]   2> 9135 T14 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 9136 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 9147 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 9161 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6aceca75 name:ZooKeeperConnection Watcher:127.0.0.1:53803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 9162 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 9163 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 9183 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 9186 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a1e214f name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 9187 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 9195 T14 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 10199 T14 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44469_
   [junit4]   2> 10201 T14 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44469_
   [junit4]   2> 10204 T14 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 10208 T14 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores
   [junit4]   2> 10209 T14 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/
   [junit4]   2> 10209 T14 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 10212 T75 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 10212 T75 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 10213 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 10213 T75 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 10214 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44469",
   [junit4]   2> 	  "node_name":"127.0.0.1:44469_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 10215 T47 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44469",
   [junit4]   2> 	  "node_name":"127.0.0.1:44469_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 10216 T47 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 10216 T47 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 10320 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 10320 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 10320 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11214 T75 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 11214 T75 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 11215 T75 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 11215 T75 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 11216 T75 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 11217 T75 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/'
   [junit4]   2> 11231 T75 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 11233 T75 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 11251 T75 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 11260 T75 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 11280 T75 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 11282 T75 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 11328 T75 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 11601 T75 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 11610 T75 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 11612 T75 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 11650 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 11658 T75 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 11671 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 11673 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 11673 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 11673 T75 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 11674 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 11675 T75 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 11675 T75 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 11676 T75 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 11676 T75 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/data/
   [junit4]   2> 11676 T75 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d2a83
   [junit4]   2> 11679 T75 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/data
   [junit4]   2> 11679 T75 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/data/index/
   [junit4]   2> 11680 T75 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 11680 T75 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-1-001/cores/collection1/data/index
   [junit4]   2> 11681 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=96.6611328125, floorSegmentMB=1.4775390625, forceMergeDeletesPctAllowed=28.852509199252363, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20338264693827954
   [junit4]   2> 11682 T75 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6af5c8ee lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6091f895),segFN=segments_1,generation=1}
   [junit4]   2> 11682 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 11692 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 11693 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 11693 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 11693 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 11694 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 11694 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 11695 T75 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 11695 T75 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 11696 T75 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 11697 T75 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 11699 T75 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 11700 T75 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 11701 T75 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 11702 T75 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 11704 T75 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 11709 T75 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 11711 T75 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 11712 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 11714 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 11715 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 11716 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 11717 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 11721 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 11723 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 11724 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 11730 T75 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 11731 T75 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 11737 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 11744 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 11747 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 11752 T75 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 11754 T75 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 11761 T75 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 11762 T75 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 11775 T75 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 11788 T75 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=46.650390625, floorSegmentMB=1.572265625, forceMergeDeletesPctAllowed=21.43306147359764, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.31335281901899237
   [junit4]   2> 11789 T75 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@6af5c8ee lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6091f895),segFN=segments_1,generation=1}
   [junit4]   2> 11790 T75 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 11795 T75 oass.SolrIndexSearcher.<init> Opening Searcher@333b418c[collection1] main
   [junit4]   2> 11796 T75 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 11797 T75 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 11798 T75 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 11799 T75 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 11799 T75 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 11799 T75 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 11800 T75 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 11801 T75 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 11801 T75 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 11801 T75 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 11804 T76 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@333b418c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 11805 T75 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 11816 T14 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 11816 T14 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 11817 T79 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:44469 collection:collection1 shard:shard2
   [junit4]   2> 11820 T79 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 11845 T79 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 11848 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 11850 T79 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 11851 T79 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@60ce8f11 url=https://127.0.0.1:44469/collection1 node=127.0.0.1:44469_ C1_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:44469, node_name=127.0.0.1:44469_, state=down}
   [junit4]   2> 11851 T79 C1 P44469 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:44469/collection1/
   [junit4]   2> 11851 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 5
   [junit4]   2> 11852 T79 C1 P44469 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 11853 T79 C1 P44469 oasc.SyncStrategy.syncToMe https://127.0.0.1:44469/collection1/ has no replicas
   [junit4]   2> 11853 T79 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:44469/collection1/ shard2
   [junit4]   2> 11854 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11854 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11854 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11866 T79 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 11879 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 11881 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44469",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 6
   [junit4]   2> 11989 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11989 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 11989 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 12031 T79 oasc.ZkController.register We are https://127.0.0.1:44469/collection1/ and leader is https://127.0.0.1:44469/collection1/
   [junit4]   2> 12032 T79 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:44469
   [junit4]   2> 12032 T79 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 12032 T79 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 12032 T79 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 12038 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 12040 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44469",
   [junit4]   2> 	  "node_name":"127.0.0.1:44469_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 7
   [junit4]   2> 12042 T47 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44469",
   [junit4]   2> 	  "node_name":"127.0.0.1:44469_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 12167 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 12167 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 12168 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 12225 T14 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1
   [junit4]   2> 12226 T14 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 12231 T14 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 12245 T14 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7bf2ed{/,null,AVAILABLE}
   [junit4]   2> 12272 T14 oejs.AbstractConnector.doStart Started ServerConnector@57c62df9{SSL-http/1.1}{127.0.0.1:43827}
   [junit4]   2> 12274 T14 oejs.Server.doStart Started @13799ms
   [junit4]   2> 12278 T14 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@c387f44
   [junit4]   2> 12278 T14 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 12279 T14 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001
   [junit4]   2> 12279 T14 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/'
   [junit4]   2> 12313 T14 oasc.SolrXmlConfig.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/solr.xml
   [junit4]   2> 12323 T14 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores
   [junit4]   2> 12324 T14 oasc.CoreContainer.<init> New CoreContainer 1572400079
   [junit4]   2> 12325 T14 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/]
   [junit4]   2> 12325 T14 oasc.CoreContainer.load loading shared library: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/lib
   [junit4]   2> 12326 T14 oasc.SolrResourceLoader.addToClassLoader WARN Can't find (or read) directory to add to classloader: lib (resolved as: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/lib).
   [junit4]   2> 12339 T14 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 12340 T14 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 12340 T14 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 12341 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 12341 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 12342 T14 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 12342 T14 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 12342 T14 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 12343 T14 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 12343 T14 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 12344 T14 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 12346 T14 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 12348 T14 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 12349 T14 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 12349 T14 oasc.CoreContainer.load Node Name: 127.0.0.1
   [junit4]   2> 12350 T14 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53803/solr
   [junit4]   2> 12350 T14 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 12350 T14 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 12359 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 12362 T93 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23efd8fd name:ZooKeeperConnection Watcher:127.0.0.1:53803 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 12363 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 12364 T14 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 12391 T14 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 12394 T96 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@437edd74 name:ZooKeeperConnection Watcher:127.0.0.1:53803/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 12394 T14 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 12403 T14 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 13408 T14 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43827_
   [junit4]   2> 13410 T14 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43827_
   [junit4]   2> 13413 T14 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 13416 T14 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores
   [junit4]   2> 13417 T14 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/
   [junit4]   2> 13418 T14 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 13419 T97 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 13420 T97 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 13420 T46 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 13421 T97 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 13422 T47 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43827",
   [junit4]   2> 	  "node_name":"127.0.0.1:43827_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 8
   [junit4]   2> 13423 T47 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:43827",
   [junit4]   2> 	  "node_name":"127.0.0.1:43827_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 13423 T47 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 13423 T47 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 13527 T74 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 13527 T46 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 13527 T57 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 13527 T96 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 14421 T97 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 14421 T97 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 14422 T97 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 14423 T97 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 14424 T97 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 14425 T97 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/'
   [junit4]   2> 14456 T97 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 14458 T97 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 14479 T97 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 14488 T97 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 14542 T97 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 14543 T97 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 14582 T97 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 14767 T97 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 14811 T97 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 14815 T97 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 14897 T97 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 14907 T97 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 14913 T97 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 14914 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 14915 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 14915 T97 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 14916 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 14916 T97 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 14917 T97 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 14918 T97 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 14920 T97 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/data/
   [junit4]   2> 14920 T97 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@737d2a83
   [junit4]   2> 14927 T97 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/data
   [junit4]   2> 14927 T97 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/data/index/
   [junit4]   2> 14927 T97 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 14932 T97 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-2-001/cores/collection1/data/index
   [junit4]   2> 14934 T97 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=96.6611328125, floorSegmentMB=1.4775390625, forceMergeDeletesPctAllowed=28.852509199252363, segmentsPerTier=34.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20338264693827954
   [junit4]   2> 14936 T97 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@56d8f89e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13b48d58),segFN=segments_1,generation=1}
   [junit4]   2> 14937 T97 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 14953 T97 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 14954 T97 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 14955 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 14955 T97 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 14955 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 14956 T97 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 14956 T97 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 14957 T97 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 14957 T97 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 14958 T97 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14959 T97 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14960 T97 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14961 T97 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14962 T97 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 14964 T97 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 14980 T97 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 14981 T97 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 14981 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 14982 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 14983 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 14984 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 14985 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 14985 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 14987 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 14988 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 14991 T97 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 14993 T97 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 14995 T97 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14997 T97 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14998 T97 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15007 T97 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15008 T97 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 15009 T97 oasu.UpdateHandler.<init> Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 15010 T97 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 15010 T97 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 15018 T97 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.inde

[...truncated too long message...]

 to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-4-001/cores/acollectionafterbaddelete_shard1_replica2/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-4-001/cores/acollectionafterbaddelete_shard1_replica2/data;done=false>>]
   [junit4]   2> 128643 T580 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001/shard-4-001/cores/acollectionafterbaddelete_shard1_replica2/data
   [junit4]   2> 128643 T580 oascc.SolrZkClient.makePath makePath: /collections/acollectionafterbaddelete/leaders/shard1
   [junit4]   2> 128644 T580 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:147)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:323)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:198)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:159)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:390)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:262)
   [junit4]   2> 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:292)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:289)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:289)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:484)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:453)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:440)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:139)
   [junit4]   2> 		at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:135)
   [junit4]   2> 		... 11 more
   [junit4]   2> 	
   [junit4]   2> 128648 T14 oejs.AbstractConnector.doStop Stopped ServerConnector@3acf69a3{SSL-http/1.1}{127.0.0.1:0}
   [junit4]   2> 128648 T14 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@75bd2be2{/,null,UNAVAILABLE}
   [junit4]   2> 128650 T14 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53803 53803
   [junit4]   2> 129228 T293 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53803 53803
   [junit4]   2> 129229 T293 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		2	/solr/collections/acollectionafterbaddelete/state.json
   [junit4]   2> 		2	/solr/collections/halfdeletedcollection/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent data watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		2	/solr/overseer_elect/election/93398250006577156-127.0.0.1:60811_-n_0000000000
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		5	/solr/live_nodes
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=test -Dtests.seed=A9A51ACA5447870 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_CO -Dtests.timezone=Etc/Greenwich -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    128s J1 | CollectionsAPIDistributedZkTest.test <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:60811, https://127.0.0.1:50092, https://127.0.0.1:46672]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A9A51ACA5447870:82CE6E760BB81588]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:349)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1043)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:809)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.addReplicaTest(CollectionsAPIDistributedZkTest.java:1125)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:202)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:945)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:920)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:60811: ADDREPLICA failed to create replica
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:370)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
   [junit4]    > 	... 47 more
   [junit4]   2> 129259 T14 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest A9A51ACA5447870-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {other_tl1=Lucene50(blocksize=128), _version_=Lucene50(blocksize=128), multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=Lucene50(blocksize=128), intDefault=Lucene50(blocksize=128), a_i1=Lucene50(blocksize=128), id=BlockTreeOrds(blocksize=128), text=PostingsFormat(name=Asserting), range_facet_l=Lucene50(blocksize=128), timestamp=Lucene50(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Lucene50)}, sim=DefaultSimilarity, locale=es_CO, timezone=Etc/Greenwich
   [junit4]   2> NOTE: Linux 3.13.0-45-generic amd64/Oracle Corporation 1.8.0_31 (64-bit)/cpus=8,threads=1,free=238081304,total=482344960
   [junit4]   2> NOTE: All tests run in this JVM: [CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 129.51s, 1 test, 1 error <<< FAILURES!

[...truncated 1438 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:191: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958: There were test failures: 468 suites, 1868 tests, 1 error, 48 ignored (21 assumptions)

Total time: 71 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_31 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b22) - Build # 11897 - Still Failing!

Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
Ha!  Randomized testing FTW!

~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley

On Sat, Feb 28, 2015 at 10:29 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> I committed a fix .. this was a fun one: SimpleText had a bug where if
> you indexed a SORTED doc value with the string value "END", its
> checkIntegrity got confused and falsely detected corruption.
>
> It just took our random tests this long to index the string "END" ...
>
> Soon we will be indexing the full works of Shakespeare...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Feb 28, 2015 at 7:28 AM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
> > Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11897/
> > Java: 32bit/jdk1.8.0_40-ea-b22 -server -XX:+UseConcMarkSweepGC
> >
> > 1 tests failed.
> > FAILED:
> org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields
> >
> > Error Message:
> > SimpleText failure: expected checksum line but got length 3
> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
> >
> > Stack Trace:
> > org.apache.lucene.index.CorruptIndexException: SimpleText failure:
> expected checksum line but got length 3
> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
> >         at
> __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
> >         at
> org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
> >         at
> org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
> >         at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
> >         at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
> >         at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
> >         at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
> >         at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
> >         at
> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
> >         at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
> >         at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
> >         at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
> >         at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
> >         at
> org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
> >         at
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
> >         at
> org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >         at java.lang.reflect.Method.invoke(Method.java:497)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> >         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> >         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> >         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> >         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> >         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> >         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> >         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> >         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> >         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> >         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> >         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> >         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> >         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
> >         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> >         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> >         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> >         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> >         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> >         at java.lang.Thread.run(Thread.java:745)
> >         Suppressed: java.lang.AssertionError
> >                 at
> org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
> >                 at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
> >                 ... 48 more
> >
> >
> >
> >
> > Build Log:
> > [...truncated 5611 lines...]
> >    [junit4] Suite:
> org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat
> >    [junit4]   2> NOTE: reproduce with: ant test
> -Dtestcase=TestSimpleTextDocValuesFormat
> -Dtests.method=testSortedFixedLengthVsStoredFields
> -Dtests.seed=4879A5F99AD2035B -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=et -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> >    [junit4] ERROR   1.68s J1 |
> TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields <<<
> >    [junit4]    > Throwable #1:
> org.apache.lucene.index.CorruptIndexException: SimpleText failure: expected
> checksum line but got length 3
> (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
> >    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
> >    [junit4]    >        at
> org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
> >    [junit4]    >        at
> org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
> >    [junit4]    >        at
> org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
> >    [junit4]    >        at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
> >    [junit4]    >        at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
> >    [junit4]    >        at
> org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
> >    [junit4]    >        at
> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
> >    [junit4]    >        at
> org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
> >    [junit4]    >        at
> org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
> >    [junit4]    >        at
> org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
> >    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
> >    [junit4]    >        Suppressed: java.lang.AssertionError
> >    [junit4]    >                at
> org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
> >    [junit4]    >                at
> org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
> >    [junit4]    >                ... 48 more
> >    [junit4]   2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/test/J1/temp/lucene.codecs.simpletext.TestSimpleTextDocValuesFormat
> 4879A5F99AD2035B-001
> >    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {},
> docValues:{}, sim=DefaultSimilarity, locale=et, timezone=Africa/Addis_Ababa
> >    [junit4]   2> NOTE: Linux 3.13.0-45-generic i386/Oracle Corporation
> 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=27185056,total=64880640
> >    [junit4]   2> NOTE: All tests run in this JVM:
> [TestSimpleTextFieldInfoFormat, TestVarGapDocFreqIntervalPostingsFormat,
> TestSimpleTextNormsFormat, TestBloomPostingsFormat,
> TestSimpleTextPostingsFormat, TestSimpleTextDocValuesFormat]
> >    [junit4] Completed on J1 in 57.94s, 84 tests, 1 error <<< FAILURES!
> >
> > [...truncated 50 lines...]
> > BUILD FAILED
> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The
> following error occurred while executing this line:
> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The
> following error occurred while executing this line:
> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The
> following error occurred while executing this line:
> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39:
> The following error occurred while executing this line:
> > /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:456:
> The following error occurred while executing this line:
> >
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2081:
> The following error occurred while executing this line:
> >
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:58:
> The following error occurred while executing this line:
> >
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351:
> The following error occurred while executing this line:
> >
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958:
> There were test failures: 19 suites, 501 tests, 1 error, 19 ignored (19
> assumptions)
> >
> > Total time: 27 minutes 47 seconds
> > Build step 'Invoke Ant' marked build as failure
> > [description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b22
> -server -XX:+UseConcMarkSweepGC
> > Archiving artifacts
> > Recording test results
> > Email was triggered for: Failure - Any
> > Sending email for trigger: Failure - Any
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b22) - Build # 11897 - Still Failing!

Posted by Michael McCandless <lu...@mikemccandless.com>.
I committed a fix .. this was a fun one: SimpleText had a bug where if
you indexed a SORTED doc value with the string value "END", its
checkIntegrity got confused and falsely detected corruption.

It just took our random tests this long to index the string "END" ...

Soon we will be indexing the full works of Shakespeare...

Mike McCandless

http://blog.mikemccandless.com


On Sat, Feb 28, 2015 at 7:28 AM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11897/
> Java: 32bit/jdk1.8.0_40-ea-b22 -server -XX:+UseConcMarkSweepGC
>
> 1 tests failed.
> FAILED:  org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields
>
> Error Message:
> SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
>
> Stack Trace:
> org.apache.lucene.index.CorruptIndexException: SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
>         at __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
>         at org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
>         at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
>         at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
>         at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
>         at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
>         at org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>         Suppressed: java.lang.AssertionError
>                 at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
>                 at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
>                 ... 48 more
>
>
>
>
> Build Log:
> [...truncated 5611 lines...]
>    [junit4] Suite: org.apache.lucene.codecs.simpletext.TestSimpleTextDocValuesFormat
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSimpleTextDocValuesFormat -Dtests.method=testSortedFixedLengthVsStoredFields -Dtests.seed=4879A5F99AD2035B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   1.68s J1 | TestSimpleTextDocValuesFormat.testSortedFixedLengthVsStoredFields <<<
>    [junit4]    > Throwable #1: org.apache.lucene.index.CorruptIndexException: SimpleText failure: expected checksum line but got length 3 (resource=BufferedChecksumIndexInput(MockIndexInputWrapper(_w.dat)))
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([4879A5F99AD2035B:A4FCD66955DBA1EC]:0)
>    [junit4]    >        at org.apache.lucene.codecs.simpletext.SimpleTextUtil.checkFooter(SimpleTextUtil.java:90)
>    [junit4]    >        at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesReader.checkIntegrity(SimpleTextDocValuesReader.java:527)
>    [junit4]    >        at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:135)
>    [junit4]    >        at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:143)
>    [junit4]    >        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:105)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3928)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3509)
>    [junit4]    >        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1798)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2733)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2838)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2805)
>    [junit4]    >        at org.apache.lucene.index.RandomIndexWriter.commit(RandomIndexWriter.java:252)
>    [junit4]    >        at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestSortedVsStoredFields(BaseDocValuesFormatTestCase.java:1448)
>    [junit4]    >        at org.apache.lucene.index.BaseDocValuesFormatTestCase.testSortedFixedLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1493)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]    >        Suppressed: java.lang.AssertionError
>    [junit4]    >                at org.apache.lucene.codecs.simpletext.SimpleTextDocValuesWriter.close(SimpleTextDocValuesWriter.java:441)
>    [junit4]    >                at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:144)
>    [junit4]    >                ... 48 more
>    [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/codecs/test/J1/temp/lucene.codecs.simpletext.TestSimpleTextDocValuesFormat 4879A5F99AD2035B-001
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=et, timezone=Africa/Addis_Ababa
>    [junit4]   2> NOTE: Linux 3.13.0-45-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=27185056,total=64880640
>    [junit4]   2> NOTE: All tests run in this JVM: [TestSimpleTextFieldInfoFormat, TestVarGapDocFreqIntervalPostingsFormat, TestSimpleTextNormsFormat, TestBloomPostingsFormat, TestSimpleTextPostingsFormat, TestSimpleTextDocValuesFormat]
>    [junit4] Completed on J1 in 57.94s, 84 tests, 1 error <<< FAILURES!
>
> [...truncated 50 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:456: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2081: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:58: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1351: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:958: There were test failures: 19 suites, 501 tests, 1 error, 19 ignored (19 assumptions)
>
> Total time: 27 minutes 47 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b22 -server -XX:+UseConcMarkSweepGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org