You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/07/19 11:39:58 UTC
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 325 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/325/
1 tests failed.
REGRESSION: org.apache.lucene.search.TestFieldCache.testIntFieldCache
Error Message:
expected:<0> but was:<-7>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<-7>
at __randomizedtesting.SeedInfo.seed([2172D358836432C1:1D23D5247DB7E1DE]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.lucene.search.TestFieldCache.testIntFieldCache(TestFieldCache.java:772)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 938 lines...]
[junit4] Suite: org.apache.lucene.search.TestFieldCache
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFieldCache -Dtests.method=testIntFieldCache -Dtests.seed=2172D358836432C1 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=da -Dtests.timezone=Asia/Dili -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 0.03s J1 | TestFieldCache.testIntFieldCache <<<
[junit4] > Throwable #1: java.lang.AssertionError: expected:<0> but was:<-7>
[junit4] > at __randomizedtesting.SeedInfo.seed([2172D358836432C1:1D23D5247DB7E1DE]:0)
[junit4] > at org.apache.lucene.search.TestFieldCache.testIntFieldCache(TestFieldCache.java:772)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=da, timezone=Asia/Dili
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=1,free=173617920,total=450428928
[junit4] 2> NOTE: All tests run in this JVM: [TestDocument, TestCollectionUtil, TestAtomicUpdate, TestMinimize, TestOmitPositions, TestAttributeSource, TestSpansAdvanced, TestNamedSPILoader, TestPayloadNearQuery, TestTermRangeFilter, TestCharFilter, Nested1, TestPerFieldDocValuesFormat, TestHighCompressionMode, TestSearchWithThreads, TestSpans, TestIndexWriterUnicode, TestDocumentWriter, TestIndexWriterDelete, TestTransactions, TestAssertions, TestDocValuesIndexing, TestNorms, TestMockDirectoryWrapper, TestMixedCodecs, TestMultiTermConstantScore, TestBoolean2, TestSnapshotDeletionPolicy, TestReusableStringReader, TestVersion, TestTermVectorsFormat, TestCodecHoldsOpenFiles, TestIndexFileDeleter, TestSortDocValues, TestDocValuesScoring, TestFieldValueFilter, TestField, TestGraphTokenizers, TestNumericUtils, TestLongPostings, TestSpanExplanations, TestPerSegmentDeletes, TestFileSwitchDirectory, TestCachingTokenFilter, TestIndexWriterMergePolicy, TestPositiveScoresOnlyCollector, TestPayloadExplanations, NestedSetupChain, NestedTeardownChain, TestSimpleSearchEquivalence, TestFlex, TestBinaryDocument, TestRegexpRandom2, TestRamUsageEstimatorOnWildAnimals, TestIdentityHashSet, TestCopyBytes, TestBasics, TestCharsRef, TestRecyclingIntBlockAllocator, TestUnicodeUtil, TestMathUtil, TestIndexWriterOnJRECrash, TestCheckIndex, TestDuelingCodecs, TestDoc, TestPerFieldPostingsFormat, TestByteArrayDataInput, TestNRTCachingDirectory, TestLiveFieldValues, TestIndexWriterWithThreads, TestTermdocPerf, TestRollingBuffer, TestCachingWrapperFilter, TestFieldMaskingSpanQuery, TestNewestSegment, TestBooleanScorer, TestSpanSearchEquivalence, TestInPlaceMergeSorter, TestForceMergeForever, TestVersionComparator, TestMultiThreadTermVectors, TestCompressingStoredFieldsFormat, TestSmallFloat, TestCodecs, TestBlockPostingsFormat2, TestPrefixQuery, TestTopDocsCollector, TestDateSort, TestWeakIdentityMap, Nested1, TestNumericTokenStream, TestDocValuesFormat, TestLucene40TermVectorsFormat, TestIndexWriterMerging, TestFSTs, TestNumericRangeQuery64, TestPackedInts, TestLucene42DocValuesFormat, TestBackwardsCompatibility, TestAddIndexes, TestIndexWriterExceptions, TestMultiMMap, TestDeletionPolicy, TestBooleanOr, TestFlushByRamOrCountsPolicy, TestDirectoryReaderReopen, TestTimeLimitingCollector, TestSearchAfter, TestConcurrentMergeScheduler, TestOpenBitSet, TestNeverDelete, TestStressIndexing, TestDocTermOrdsRewriteMethod, TestLucene40PostingsReader, TestNRTReaderWithThreads, TestTieredMergePolicy, TestIndexWriterForceMerge, TestArrayUtil, TestLevenshteinAutomata, TestBufferedIndexInput, TestTermsEnum, TestConsistentFieldNumbers, TestDocTermOrdsRangeFilter, TestSegmentMerger, TestOmitNorms, TestFixedBitSet, TestDoubleBarrelLRUCache, TestComplexExplanations, TestSegmentReader, TestFastDecompressionMode, TestParallelCompositeReader, TestIndexWriterNRTIsCurrent, TestFieldCacheRewriteMethod, TestPagedBytes, TestSegmentTermDocs, TestPerFieldPostingsFormat2, TestBlockPostingsFormat, TestBitVector, TestMultiDocValues, TestFieldCacheSanityChecker, TestToken, TestSimilarityBase, TestHugeRamFile, TestDeterminizeLexicon, Test2BDocs, TestIndexWriterOnDiskFull, TestFieldCache]
[junit4] Completed on J1 in 3.43s, 10 tests, 1 failure <<< FAILURES!
[...truncated 449 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1247: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:890: There were test failures: 363 suites, 2297 tests, 1 failure, 30 ignored (17 assumptions)
Total time: 33 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 331 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/331/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([6C00272C40A9A4C]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
at __randomizedtesting.SeedInfo.seed([6C00272C40A9A4C]:0)
Build Log:
[...truncated 9782 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 1368643 T2935 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pqj/
[junit4] 2> 1368647 T2935 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374822399322
[junit4] 2> 1368649 T2935 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1368649 T2936 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1368750 T2935 oasc.ZkTestServer.run start zk server on port:42819
[junit4] 2> 1368751 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1368755 T2942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78b09077 name:ZooKeeperConnection Watcher:127.0.0.1:42819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1368755 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1368756 T2935 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1368767 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1368769 T2944 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@103c587d name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1368769 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1368770 T2935 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1368780 T2935 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1368782 T2935 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1368785 T2935 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1368793 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1368793 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1368798 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1368798 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1368901 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1368902 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1368905 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1368905 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1368908 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1368908 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1368911 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1368911 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1368914 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1368914 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1368922 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1368923 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1368925 T2935 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1368926 T2935 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1369214 T2935 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1369219 T2935 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42822
[junit4] 2> 1369220 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1369221 T2935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1369221 T2935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609
[junit4] 2> 1369221 T2935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/'
[junit4] 2> 1369260 T2935 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/solr.xml
[junit4] 2> 1369348 T2935 oasc.CoreContainer.<init> New CoreContainer 1554702447
[junit4] 2> 1369349 T2935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/]
[junit4] 2> 1369350 T2935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1369351 T2935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1369351 T2935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1369352 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1369352 T2935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1369352 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1369353 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1369353 T2935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1369354 T2935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1369354 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1369372 T2935 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42819/solr
[junit4] 2> 1369372 T2935 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1369373 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1369376 T2955 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3100f3da name:ZooKeeperConnection Watcher:127.0.0.1:42819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1369376 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1369387 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1369399 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1369402 T2957 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c8e6e24 name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1369402 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1369409 T2935 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1369417 T2935 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1369419 T2935 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1369426 T2935 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42822_pqj
[junit4] 2> 1369428 T2935 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42822_pqj
[junit4] 2> 1369430 T2935 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1369443 T2935 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1369445 T2935 oasc.Overseer.start Overseer (id=90100360762294275-127.0.0.1:42822_pqj-n_0000000000) starting
[junit4] 2> 1369455 T2935 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1369465 T2959 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1369465 T2935 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1369467 T2935 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1369475 T2935 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1369477 T2958 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1369480 T2960 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1369480 T2960 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1369482 T2960 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1370980 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1370982 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42822_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42822/pqj"}
[junit4] 2> 1370982 T2958 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 1370982 T2958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1370998 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1371484 T2960 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1371484 T2960 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/collection1
[junit4] 2> 1371484 T2960 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1371485 T2960 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1371485 T2960 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1371486 T2960 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/collection1/'
[junit4] 2> 1371488 T2960 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/collection1/lib/classes/' to classloader
[junit4] 2> 1371488 T2960 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/collection1/lib/README' to classloader
[junit4] 2> 1371549 T2960 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1371608 T2960 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1371709 T2960 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1371719 T2960 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1372309 T2960 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1372321 T2960 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1372325 T2960 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1372338 T2960 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1372343 T2960 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1372347 T2960 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1372348 T2960 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1372349 T2960 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1372349 T2960 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1372350 T2960 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1372350 T2960 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1372351 T2960 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1372351 T2960 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374822399609/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/control/data/
[junit4] 2> 1372351 T2960 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@547e97d8
[junit4] 2> 1372352 T2960 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/control/data
[junit4] 2> 1372352 T2960 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/control/data/index/
[junit4] 2> 1372353 T2960 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1372353 T2960 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/control/data/index
[junit4] 2> 1372358 T2960 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74b719fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@216e7f30),segFN=segments_1,generation=1}
[junit4] 2> 1372358 T2960 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1372363 T2960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1372363 T2960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1372364 T2960 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1372365 T2960 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1372365 T2960 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1372365 T2960 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1372366 T2960 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1372366 T2960 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1372367 T2960 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1372384 T2960 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1372391 T2960 oass.SolrIndexSearcher.<init> Opening Searcher@652f24f5 main
[junit4] 2> 1372393 T2960 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1372393 T2960 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1372397 T2961 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@652f24f5 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1372399 T2960 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1372400 T2960 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42822/pqj collection:control_collection shard:shard1
[junit4] 2> 1372400 T2960 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1372419 T2960 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1372438 T2960 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1372450 T2960 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1372450 T2960 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1372450 T2960 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42822/pqj/collection1/
[junit4] 2> 1372450 T2960 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1372451 T2960 oasc.SyncStrategy.syncToMe http://127.0.0.1:42822/pqj/collection1/ has no replicas
[junit4] 2> 1372451 T2960 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42822/pqj/collection1/ shard1
[junit4] 2> 1372451 T2960 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1372502 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1372516 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1372567 T2960 oasc.ZkController.register We are http://127.0.0.1:42822/pqj/collection1/ and leader is http://127.0.0.1:42822/pqj/collection1/
[junit4] 2> 1372567 T2960 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42822/pqj
[junit4] 2> 1372567 T2960 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1372567 T2960 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1372567 T2960 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1372569 T2960 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1372570 T2935 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1372571 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1372572 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1372584 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1372586 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1372588 T2964 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bd8e5e name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1372588 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1372589 T2935 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1372591 T2935 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1372868 T2935 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1372871 T2935 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42826
[junit4] 2> 1372872 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1372872 T2935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1372873 T2935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265
[junit4] 2> 1372873 T2935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/'
[junit4] 2> 1372912 T2935 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/solr.xml
[junit4] 2> 1372999 T2935 oasc.CoreContainer.<init> New CoreContainer 996517898
[junit4] 2> 1373000 T2935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/]
[junit4] 2> 1373002 T2935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1373002 T2935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1373002 T2935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1373003 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1373003 T2935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1373003 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1373004 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1373004 T2935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1373005 T2935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1373005 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1373022 T2935 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42819/solr
[junit4] 2> 1373023 T2935 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1373024 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1373027 T2975 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3219a20d name:ZooKeeperConnection Watcher:127.0.0.1:42819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1373027 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1373041 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1373053 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1373055 T2977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@265f918 name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1373056 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1373066 T2935 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1374026 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1374027 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42822_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42822/pqj"}
[junit4] 2> 1374041 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1374041 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1374041 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1374068 T2935 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42826_pqj
[junit4] 2> 1374070 T2935 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42826_pqj
[junit4] 2> 1374072 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1374072 T2957 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1374072 T2977 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1374072 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1374072 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1374073 T2964 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1374087 T2978 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1374087 T2978 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1374089 T2978 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1375545 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1375546 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42826_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42826/pqj"}
[junit4] 2> 1375546 T2958 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 1375547 T2958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1375562 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1375562 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1375562 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1376091 T2978 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1376091 T2978 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/collection1
[junit4] 2> 1376091 T2978 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1376092 T2978 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1376092 T2978 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1376093 T2978 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/collection1/'
[junit4] 2> 1376095 T2978 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/collection1/lib/README' to classloader
[junit4] 2> 1376095 T2978 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/collection1/lib/classes/' to classloader
[junit4] 2> 1376149 T2978 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1376207 T2978 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1376308 T2978 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1376319 T2978 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1376764 T2978 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1376776 T2978 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1376780 T2978 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1376794 T2978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1376799 T2978 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1376804 T2978 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1376805 T2978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1376805 T2978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1376805 T2978 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1376806 T2978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1376806 T2978 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1376807 T2978 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1376807 T2978 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374822403265/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty1/
[junit4] 2> 1376807 T2978 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@547e97d8
[junit4] 2> 1376808 T2978 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty1
[junit4] 2> 1376809 T2978 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty1/index/
[junit4] 2> 1376809 T2978 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1376809 T2978 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty1/index
[junit4] 2> 1376813 T2978 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5fd5b7f8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e756029),segFN=segments_1,generation=1}
[junit4] 2> 1376813 T2978 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1376819 T2978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1376819 T2978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1376820 T2978 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1376821 T2978 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1376821 T2978 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1376822 T2978 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1376822 T2978 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1376823 T2978 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1376823 T2978 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1376840 T2978 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1376848 T2978 oass.SolrIndexSearcher.<init> Opening Searcher@358df8de main
[junit4] 2> 1376849 T2978 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1376849 T2978 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1376855 T2979 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@358df8de main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1376857 T2978 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1376858 T2978 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42826/pqj collection:collection1 shard:shard1
[junit4] 2> 1376858 T2978 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1376875 T2978 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1376906 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1376916 T2978 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1376916 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1376916 T2978 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42826/pqj/collection1/
[junit4] 2> 1376916 T2978 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1376917 T2978 oasc.SyncStrategy.syncToMe http://127.0.0.1:42826/pqj/collection1/ has no replicas
[junit4] 2> 1376917 T2978 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42826/pqj/collection1/ shard1
[junit4] 2> 1376917 T2978 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1377066 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1377081 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1377081 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1377081 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1377133 T2978 oasc.ZkController.register We are http://127.0.0.1:42826/pqj/collection1/ and leader is http://127.0.0.1:42826/pqj/collection1/
[junit4] 2> 1377133 T2978 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42826/pqj
[junit4] 2> 1377133 T2978 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1377133 T2978 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1377134 T2978 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1377135 T2978 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1377136 T2935 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1377137 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1377138 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1377406 T2935 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1377409 T2935 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42829
[junit4] 2> 1377409 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1377410 T2935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1377410 T2935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823
[junit4] 2> 1377411 T2935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/'
[junit4] 2> 1377446 T2935 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/solr.xml
[junit4] 2> 1377527 T2935 oasc.CoreContainer.<init> New CoreContainer 1227320939
[junit4] 2> 1377528 T2935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/]
[junit4] 2> 1377529 T2935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1377530 T2935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1377530 T2935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1377530 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1377531 T2935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1377531 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1377531 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1377532 T2935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1377532 T2935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1377533 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1377549 T2935 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42819/solr
[junit4] 2> 1377549 T2935 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1377550 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1377553 T2991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@466008ac name:ZooKeeperConnection Watcher:127.0.0.1:42819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1377554 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1377568 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1377580 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1377582 T2993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6efcc134 name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1377582 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1377591 T2935 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1378594 T2935 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42829_pqj
[junit4] 2> 1378595 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1378596 T2935 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42829_pqj
[junit4] 2> 1378596 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42826_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42826/pqj"}
[junit4] 2> 1378612 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1378613 T2977 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1378613 T2993 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1378613 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378614 T2964 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1378621 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378621 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378621 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378623 T2957 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1378624 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378624 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1378627 T2994 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1378627 T2994 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1378629 T2994 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1380125 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1380126 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42829_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42829/pqj"}
[junit4] 2> 1380126 T2958 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1380126 T2958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1380140 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1380141 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1380141 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1380140 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1380631 T2994 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1380631 T2994 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/collection1
[junit4] 2> 1380631 T2994 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1380632 T2994 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1380632 T2994 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1380633 T2994 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/collection1/'
[junit4] 2> 1380635 T2994 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/collection1/lib/classes/' to classloader
[junit4] 2> 1380635 T2994 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/collection1/lib/README' to classloader
[junit4] 2> 1380689 T2994 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1380747 T2994 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1380848 T2994 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1380859 T2994 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1381444 T2994 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1381456 T2994 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1381466 T2994 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1381479 T2994 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1381483 T2994 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1381488 T2994 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1381488 T2994 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1381489 T2994 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1381489 T2994 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1381490 T2994 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1381490 T2994 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1381490 T2994 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1381490 T2994 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374822407823/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty2/
[junit4] 2> 1381491 T2994 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@547e97d8
[junit4] 2> 1381492 T2994 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty2
[junit4] 2> 1381492 T2994 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty2/index/
[junit4] 2> 1381492 T2994 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1381493 T2994 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty2/index
[junit4] 2> 1381496 T2994 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6df0430e lockFactory=org.apache.lucene.store.NativeFSLockFactory@1fca331e),segFN=segments_1,generation=1}
[junit4] 2> 1381497 T2994 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1381502 T2994 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1381502 T2994 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1381503 T2994 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1381504 T2994 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1381504 T2994 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1381505 T2994 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1381505 T2994 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1381506 T2994 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1381506 T2994 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1381522 T2994 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1381530 T2994 oass.SolrIndexSearcher.<init> Opening Searcher@3f12384e main
[junit4] 2> 1381531 T2994 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1381531 T2994 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1381536 T2995 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f12384e main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1381538 T2994 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1381539 T2994 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42829/pqj collection:collection1 shard:shard2
[junit4] 2> 1381539 T2994 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1381556 T2994 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1381584 T2994 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1381594 T2994 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1381594 T2994 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1381594 T2994 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42829/pqj/collection1/
[junit4] 2> 1381595 T2994 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1381595 T2994 oasc.SyncStrategy.syncToMe http://127.0.0.1:42829/pqj/collection1/ has no replicas
[junit4] 2> 1381595 T2994 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42829/pqj/collection1/ shard2
[junit4] 2> 1381595 T2994 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1381645 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1381659 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1381659 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1381659 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1381659 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1381709 T2994 oasc.ZkController.register We are http://127.0.0.1:42829/pqj/collection1/ and leader is http://127.0.0.1:42829/pqj/collection1/
[junit4] 2> 1381709 T2994 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42829/pqj
[junit4] 2> 1381709 T2994 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1381709 T2994 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1381709 T2994 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1381711 T2994 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1381712 T2935 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1381713 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1381713 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1382002 T2935 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1382005 T2935 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42832
[junit4] 2> 1382006 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1382006 T2935 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1382006 T2935 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399
[junit4] 2> 1382007 T2935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/'
[junit4] 2> 1382043 T2935 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/solr.xml
[junit4] 2> 1382138 T2935 oasc.CoreContainer.<init> New CoreContainer 1101369824
[junit4] 2> 1382139 T2935 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/]
[junit4] 2> 1382141 T2935 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1382141 T2935 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1382141 T2935 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1382142 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1382142 T2935 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1382143 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1382143 T2935 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1382143 T2935 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1382144 T2935 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1382144 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1382165 T2935 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42819/solr
[junit4] 2> 1382165 T2935 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1382166 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1382194 T3007 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c1587a4 name:ZooKeeperConnection Watcher:127.0.0.1:42819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1382194 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1382196 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1382208 T2935 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1382210 T3009 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fba5f1c name:ZooKeeperConnection Watcher:127.0.0.1:42819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1382210 T2935 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1382214 T2935 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1383164 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1383165 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42829_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42829/pqj"}
[junit4] 2> 1383178 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383178 T3009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383178 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383178 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383178 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383216 T2935 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42832_pqj
[junit4] 2> 1383217 T2935 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42832_pqj
[junit4] 2> 1383219 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383219 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383219 T3009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1383219 T2977 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1383219 T2957 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1383220 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1383220 T2964 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1383220 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1383221 T2993 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1383222 T3009 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1383228 T3010 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1383228 T3010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1383230 T3010 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1384682 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1384683 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42832_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42832/pqj"}
[junit4] 2> 1384683 T2958 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1384684 T2958 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1384700 T2977 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1384700 T2993 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1384700 T3009 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1384700 T2964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1384700 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1385232 T3010 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1385232 T3010 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/collection1
[junit4] 2> 1385232 T3010 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1385233 T3010 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1385233 T3010 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1385234 T3010 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/collection1/'
[junit4] 2> 1385236 T3010 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/collection1/lib/README' to classloader
[junit4] 2> 1385236 T3010 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/collection1/lib/classes/' to classloader
[junit4] 2> 1385291 T3010 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1385349 T3010 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1385450 T3010 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1385461 T3010 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1386046 T3010 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1386057 T3010 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1386061 T3010 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1386073 T3010 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1386078 T3010 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1386081 T3010 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1386082 T3010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1386083 T3010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1386083 T3010 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1386084 T3010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1386084 T3010 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1386084 T3010 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1386084 T3010 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374822412399/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty3/
[junit4] 2> 1386085 T3010 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@547e97d8
[junit4] 2> 1386086 T3010 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty3
[junit4] 2> 1386086 T3010 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty3/index/
[junit4] 2> 1386086 T3010 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1386086 T3010 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374822399322/jetty3/index
[junit4] 2> 1386090 T3010 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48140b1e lockFactory=org.apache.lucene.store.NativeFSLockFactory@795e0ac1),segFN=segments_1,generation=1}
[junit4] 2> 1386090 T3010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1386095 T3010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1386096 T3010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1386096 T3010 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1386097 T3010 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1386098 T3010 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1386098 T3010 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1386098 T3010 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1386099 T3010 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1386099 T3010 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1386116 T3010 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1386123 T3010 oass.SolrIndexSearcher.<init> Opening Searcher@7d349410 main
[junit4] 2> 1386125 T3010 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1386125 T3010 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1386130 T3011 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7d349410 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1386132 T3010 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1386133 T3010 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42832/pqj collection:collection1 shard:shard1
[junit4] 2> 1386133 T3010 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1386152 T3010 oasc.ZkController.register We are http://127.0.0.1:42832/pqj/collection1/ and leader is http://127.0.0.1:42826/pqj/collection1/
[junit4] 2> 1386152 T3010 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42832/pqj
[junit4] 2> 1386152 T3010 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1386153 T3010 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C984 name=collection1 org.apache.solr.core.SolrCore@73e94795 url=http://127.0.0.1:42832/pqj/collection1 node=127.0.0.1:42832_pqj C984_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42832_pqj, base_url=http://127.0.0.1:42832/pqj}
[junit4] 2> 1386153 T3012 C984 P42832 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1386153 T3010 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1386155 T3012 C984 P42832 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1386155 T3012 C984 P42832 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1386155 T3012 C984 P42832 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1386156 T2935 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1386157 T2935 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1386157 T2935 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1386207 T3012 C984 P42832 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1386208 T2958 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1386209 T2958 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42832_pqj",
[junit4] 2> "base_url":"http://127.0.0.1:42832/pqj"}
[junit4] 2> 1386220 T2967 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1386297 T2957 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurre
[...truncated too long message...]
), multiDefault=PostingsFormat(name=Lucene41WithOrds), n_tf1=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), n_dt1=PostingsFormat(name=Lucene41WithOrds), genre_s=PostingsFormat(name=Asserting), author_t=PostingsFormat(name=Lucene41WithOrds), range_facet_l=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), n_ti1=PostingsFormat(name=MockSep), text=PostingsFormat(name=Asserting), _version_=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), val_i=PostingsFormat(name=MockSep), SubjectTerms_mfacet=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), series_t=PostingsFormat(name=Lucene41WithOrds), a_t=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), n_tdt1=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), regex_dup_A_s=PostingsFormat(name=Asserting), price=PostingsFormat(name=Asserting), other_tl1=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), n_l1=Pulsing41(freqCutoff=20 minBlockSize=99 maxBlockSize=225), a_si=PostingsFormat(name=Lucene41WithOrds), inStock=PostingsFormat(name=Asserting)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=ar_SY, timezone=America/Dawson_Creek
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=2,free=89224440,total=376700928
[junit4] 2> NOTE: All tests run in this JVM: [TestFastWriter, TestFaceting, UnloadDistributedZkTest, TestStressVersions, SearchHandlerTest, JsonLoaderTest, TestLFUCache, TestPHPSerializedResponseWriter, TestFastLRUCache, FieldMutatingUpdateProcessorTest, ReturnFieldsTest, TestGroupingSearch, TestSolrXmlPersistence, DistributedQueryElevationComponentTest, DocumentBuilderTest, AlternateDirectoryTest, TestFunctionQuery, SynonymTokenizerTest, ShowFileRequestHandlerTest, TestPropInjectDefaults, TestFieldTypeResource, ClusterStateUpdateTest, TestSolrQueryParser, SpatialFilterTest, TestFieldResource, ZkControllerTest, TestSolrCoreProperties, TestRandomDVFaceting, BadComponentTest, DirectUpdateHandlerOptimizeTest, TestSearchPerf, TestPhraseSuggestions, TestMaxScoreQueryParser, TestSchemaVersionResource, ResourceLoaderTest, PeerSyncTest, MBeansHandlerTest, RecoveryZkTest, TestDefaultSearchFieldResource, DocValuesTest, TestSolrQueryParserResource, TestBadConfig, PrimitiveFieldTypeTest, QueryElevationComponentTest, LoggingHandlerTest, IndexSchemaTest, AnalysisAfterCoreReloadTest, SolrIndexSplitterTest, TestShardHandlerFactory, SchemaVersionSpecificBehaviorTest, PrimUtilsTest, SolrTestCaseJ4Test, TestFuzzyAnalyzedSuggestions, SolrCoreTest, ScriptEngineTest, TestCopyFieldCollectionResource, TestXIncludeConfig, SuggesterTest, AutoCommitTest, TestLRUCache, SoftAutoCommitTest, TestReplicationHandler, DirectSolrConnectionTest, TestExtendedDismaxParser, TestSerializedLuceneMatchVersion, ExternalFileFieldSortTest, TestAddFieldRealTimeGet, BasicFunctionalityTest, LukeRequestHandlerTest, OverseerTest, QueryParsingTest, UpdateRequestProcessorFactoryTest, StatelessScriptUpdateProcessorFactoryTest, CircularListTest, TestPluginEnable, LegacyHTMLStripCharFilterTest, TestSolrIndexConfig, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=6C00272C40A9A4C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_SY -Dtests.timezone=America/Dawson_Creek -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4] > at __randomizedtesting.SeedInfo.seed([6C00272C40A9A4C]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3445, name=recoveryCmdExecutor-1703-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:724)
[junit4] > at __randomizedtesting.SeedInfo.seed([6C00272C40A9A4C]:0)
[junit4] Completed on J0 in 416.31s, 1 test, 2 errors <<< FAILURES!
[...truncated 483 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 654 ignored (4 assumptions)
Total time: 86 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 330 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/330/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([254F2F6795AF1A1F]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([254F2F6795AF1A1F]:0)
Build Log:
[...truncated 9869 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 1319446 T2490 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /yiz/
[junit4] 2> 1319451 T2490 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374738258793
[junit4] 2> 1319453 T2490 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1319454 T2491 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1319555 T2490 oasc.ZkTestServer.run start zk server on port:38445
[junit4] 2> 1319556 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1319560 T2497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56870785 name:ZooKeeperConnection Watcher:127.0.0.1:38445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1319560 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1319560 T2490 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1319581 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1319583 T2499 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bf646a1 name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1319583 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1319583 T2490 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1319594 T2490 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1319597 T2490 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1319600 T2490 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1319608 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1319609 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1319622 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1319623 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1319726 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1319727 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1319730 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1319731 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1319742 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1319742 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1319745 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1319746 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1319749 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1319750 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1319753 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1319754 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1319757 T2490 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1319758 T2490 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1320032 T2490 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1320036 T2490 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38448
[junit4] 2> 1320037 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1320037 T2490 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1320038 T2490 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110
[junit4] 2> 1320038 T2490 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/'
[junit4] 2> 1320075 T2490 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/solr.xml
[junit4] 2> 1320160 T2490 oasc.CoreContainer.<init> New CoreContainer 993750239
[junit4] 2> 1320161 T2490 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/]
[junit4] 2> 1320163 T2490 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1320163 T2490 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1320164 T2490 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1320164 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1320165 T2490 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1320165 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1320166 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1320166 T2490 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1320167 T2490 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1320167 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1320185 T2490 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38445/solr
[junit4] 2> 1320185 T2490 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1320186 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1320189 T2510 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42afada0 name:ZooKeeperConnection Watcher:127.0.0.1:38445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1320190 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1320202 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1320214 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1320216 T2512 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb91d29 name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1320216 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1320223 T2490 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1320228 T2490 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1320237 T2490 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1320239 T2490 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38448_yiz
[junit4] 2> 1320240 T2490 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38448_yiz
[junit4] 2> 1320248 T2490 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1320261 T2490 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1320268 T2490 oasc.Overseer.start Overseer (id=90094846528782339-127.0.0.1:38448_yiz-n_0000000000) starting
[junit4] 2> 1320273 T2490 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1320286 T2514 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1320287 T2490 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1320289 T2490 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1320296 T2490 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1320299 T2513 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1320305 T2515 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1320305 T2515 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1320306 T2515 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1321803 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1321804 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38448_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38448/yiz"}
[junit4] 2> 1321804 T2513 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 1321804 T2513 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1321820 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1322309 T2515 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1322309 T2515 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/collection1
[junit4] 2> 1322309 T2515 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1322310 T2515 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1322311 T2515 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1322312 T2515 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/collection1/'
[junit4] 2> 1322313 T2515 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/collection1/lib/README' to classloader
[junit4] 2> 1322314 T2515 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/collection1/lib/classes/' to classloader
[junit4] 2> 1322367 T2515 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1322426 T2515 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1322528 T2515 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1322538 T2515 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1323115 T2515 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1323127 T2515 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1323131 T2515 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1323145 T2515 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1323150 T2515 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1323155 T2515 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1323156 T2515 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1323156 T2515 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1323156 T2515 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1323157 T2515 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1323158 T2515 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1323158 T2515 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1323158 T2515 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374738259110/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/control/data/
[junit4] 2> 1323159 T2515 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@199ee8a8
[junit4] 2> 1323160 T2515 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/control/data
[junit4] 2> 1323160 T2515 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/control/data/index/
[junit4] 2> 1323160 T2515 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1323162 T2515 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/control/data/index
[junit4] 2> 1323194 T2515 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/index1011510138tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@f292bb7),segFN=segments_1,generation=1}
[junit4] 2> 1323195 T2515 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1323199 T2515 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1323200 T2515 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1323200 T2515 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1323201 T2515 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1323202 T2515 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1323202 T2515 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1323202 T2515 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1323203 T2515 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1323203 T2515 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1323220 T2515 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1323229 T2515 oass.SolrIndexSearcher.<init> Opening Searcher@116e5955 main
[junit4] 2> 1323230 T2515 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1323231 T2515 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1323234 T2516 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@116e5955 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1323235 T2515 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1323236 T2515 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38448/yiz collection:control_collection shard:shard1
[junit4] 2> 1323236 T2515 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1323254 T2515 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1323271 T2515 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1323282 T2515 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1323282 T2515 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1323283 T2515 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38448/yiz/collection1/
[junit4] 2> 1323283 T2515 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1323283 T2515 oasc.SyncStrategy.syncToMe http://127.0.0.1:38448/yiz/collection1/ has no replicas
[junit4] 2> 1323283 T2515 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38448/yiz/collection1/ shard1
[junit4] 2> 1323284 T2515 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1323325 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1323346 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1323402 T2515 oasc.ZkController.register We are http://127.0.0.1:38448/yiz/collection1/ and leader is http://127.0.0.1:38448/yiz/collection1/
[junit4] 2> 1323402 T2515 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38448/yiz
[junit4] 2> 1323402 T2515 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1323402 T2515 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1323403 T2515 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1323405 T2515 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1323406 T2490 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1323406 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1323407 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1323419 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1323420 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1323422 T2519 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17746af0 name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1323423 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1323424 T2490 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1323426 T2490 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1323688 T2490 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1323691 T2490 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38452
[junit4] 2> 1323692 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1323692 T2490 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1323693 T2490 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767
[junit4] 2> 1323693 T2490 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/'
[junit4] 2> 1323730 T2490 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/solr.xml
[junit4] 2> 1323814 T2490 oasc.CoreContainer.<init> New CoreContainer 421993604
[junit4] 2> 1323815 T2490 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/]
[junit4] 2> 1323817 T2490 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1323817 T2490 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1323818 T2490 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1323818 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1323819 T2490 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1323819 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1323820 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1323820 T2490 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1323821 T2490 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1323821 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1323838 T2490 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38445/solr
[junit4] 2> 1323839 T2490 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1323840 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1323843 T2530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ead4929 name:ZooKeeperConnection Watcher:127.0.0.1:38445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1323843 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1323845 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1323858 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1323860 T2532 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@427ab5a1 name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1323860 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1323872 T2490 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1324852 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1324854 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38448_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38448/yiz"}
[junit4] 2> 1324858 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1324858 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1324858 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1324875 T2490 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38452_yiz
[junit4] 2> 1324877 T2490 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38452_yiz
[junit4] 2> 1324879 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1324879 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1324880 T2519 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1324880 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1324881 T2532 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1324881 T2512 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1324888 T2533 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1324889 T2533 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1324890 T2533 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1326372 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1326374 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38452_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38452/yiz"}
[junit4] 2> 1326374 T2513 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 1326374 T2513 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1326388 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1326388 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1326388 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1326893 T2533 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1326893 T2533 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/collection1
[junit4] 2> 1326893 T2533 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1326894 T2533 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1326894 T2533 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1326895 T2533 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/collection1/'
[junit4] 2> 1326897 T2533 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/collection1/lib/README' to classloader
[junit4] 2> 1326897 T2533 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/collection1/lib/classes/' to classloader
[junit4] 2> 1326951 T2533 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1327010 T2533 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1327111 T2533 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1327121 T2533 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1327696 T2533 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1327708 T2533 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1327711 T2533 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1327725 T2533 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1327730 T2533 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1327734 T2533 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1327735 T2533 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1327736 T2533 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1327736 T2533 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1327737 T2533 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1327737 T2533 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1327738 T2533 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1327738 T2533 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374738262767/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty1/
[junit4] 2> 1327738 T2533 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@199ee8a8
[junit4] 2> 1327739 T2533 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty1
[junit4] 2> 1327739 T2533 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty1/index/
[junit4] 2> 1327740 T2533 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1327742 T2533 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty1/index
[junit4] 2> 1327749 T2533 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/index1011510139tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c4a66b9),segFN=segments_1,generation=1}
[junit4] 2> 1327750 T2533 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1327755 T2533 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1327755 T2533 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1327756 T2533 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1327756 T2533 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1327757 T2533 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1327757 T2533 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1327757 T2533 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1327758 T2533 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1327758 T2533 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1327775 T2533 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1327783 T2533 oass.SolrIndexSearcher.<init> Opening Searcher@6003e06d main
[junit4] 2> 1327784 T2533 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1327784 T2533 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1327789 T2534 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6003e06d main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1327791 T2533 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1327791 T2533 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38452/yiz collection:collection1 shard:shard1
[junit4] 2> 1327791 T2533 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1327808 T2533 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1327824 T2533 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1327833 T2533 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1327834 T2533 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1327834 T2533 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38452/yiz/collection1/
[junit4] 2> 1327834 T2533 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1327834 T2533 oasc.SyncStrategy.syncToMe http://127.0.0.1:38452/yiz/collection1/ has no replicas
[junit4] 2> 1327835 T2533 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38452/yiz/collection1/ shard1
[junit4] 2> 1327835 T2533 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1327894 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1327910 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1327910 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1327910 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1327953 T2533 oasc.ZkController.register We are http://127.0.0.1:38452/yiz/collection1/ and leader is http://127.0.0.1:38452/yiz/collection1/
[junit4] 2> 1327953 T2533 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38452/yiz
[junit4] 2> 1327953 T2533 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1327953 T2533 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1327954 T2533 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1327956 T2533 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1327957 T2490 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1327957 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1327958 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1328239 T2490 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1328243 T2490 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38455
[junit4] 2> 1328244 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1328244 T2490 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1328245 T2490 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309
[junit4] 2> 1328245 T2490 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/'
[junit4] 2> 1328282 T2490 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/solr.xml
[junit4] 2> 1328368 T2490 oasc.CoreContainer.<init> New CoreContainer 1441817408
[junit4] 2> 1328369 T2490 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/]
[junit4] 2> 1328371 T2490 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1328371 T2490 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1328372 T2490 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1328372 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1328373 T2490 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1328373 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1328374 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1328374 T2490 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1328374 T2490 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1328375 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1328391 T2490 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38445/solr
[junit4] 2> 1328391 T2490 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1328392 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328421 T2546 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f22bd6b name:ZooKeeperConnection Watcher:127.0.0.1:38445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328422 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328424 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1328434 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1328436 T2548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@89e92fc name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1328437 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1328448 T2490 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1329424 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1329426 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38452_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38452/yiz"}
[junit4] 2> 1329437 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329437 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329437 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329437 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329451 T2490 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38455_yiz
[junit4] 2> 1329453 T2490 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38455_yiz
[junit4] 2> 1329456 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329456 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329456 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1329456 T2519 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1329457 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1329458 T2548 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1329459 T2532 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1329459 T2512 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1329466 T2549 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1329466 T2549 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1329467 T2549 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1330943 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1330945 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38455_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38455/yiz"}
[junit4] 2> 1330945 T2513 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1330945 T2513 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1330951 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1330951 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1330951 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1330951 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1331471 T2549 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1331471 T2549 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/collection1
[junit4] 2> 1331471 T2549 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1331472 T2549 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1331473 T2549 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1331474 T2549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/collection1/'
[junit4] 2> 1331475 T2549 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/collection1/lib/README' to classloader
[junit4] 2> 1331476 T2549 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/collection1/lib/classes/' to classloader
[junit4] 2> 1331529 T2549 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1331591 T2549 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1331693 T2549 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1331704 T2549 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1332338 T2549 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1332350 T2549 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1332354 T2549 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1332368 T2549 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1332373 T2549 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1332378 T2549 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1332379 T2549 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1332379 T2549 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1332379 T2549 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1332380 T2549 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1332381 T2549 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1332381 T2549 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1332382 T2549 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374738267309/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty2/
[junit4] 2> 1332382 T2549 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@199ee8a8
[junit4] 2> 1332383 T2549 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty2
[junit4] 2> 1332383 T2549 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty2/index/
[junit4] 2> 1332383 T2549 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1332385 T2549 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty2/index
[junit4] 2> 1332392 T2549 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/index1011510140tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@53214248),segFN=segments_1,generation=1}
[junit4] 2> 1332392 T2549 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1332397 T2549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1332398 T2549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1332398 T2549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1332399 T2549 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1332400 T2549 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1332400 T2549 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1332400 T2549 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1332401 T2549 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1332401 T2549 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1332419 T2549 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1332428 T2549 oass.SolrIndexSearcher.<init> Opening Searcher@3ba1b77c main
[junit4] 2> 1332429 T2549 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1332430 T2549 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1332434 T2550 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ba1b77c main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1332436 T2549 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1332436 T2549 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38455/yiz collection:collection1 shard:shard2
[junit4] 2> 1332436 T2549 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1332454 T2549 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1332535 T2549 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1332597 T2549 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1332597 T2549 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1332597 T2549 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38455/yiz/collection1/
[junit4] 2> 1332598 T2549 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1332598 T2549 oasc.SyncStrategy.syncToMe http://127.0.0.1:38455/yiz/collection1/ has no replicas
[junit4] 2> 1332598 T2549 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38455/yiz/collection1/ shard2
[junit4] 2> 1332598 T2549 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1334036 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1334054 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1334055 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1334055 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1334054 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1334097 T2549 oasc.ZkController.register We are http://127.0.0.1:38455/yiz/collection1/ and leader is http://127.0.0.1:38455/yiz/collection1/
[junit4] 2> 1334097 T2549 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38455/yiz
[junit4] 2> 1334097 T2549 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1334098 T2549 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1334098 T2549 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1334101 T2549 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1334102 T2490 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1334103 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1334103 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1334377 T2490 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1334381 T2490 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:51578
[junit4] 2> 1334381 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1334382 T2490 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1334382 T2490 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456
[junit4] 2> 1334383 T2490 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/'
[junit4] 2> 1334422 T2490 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/solr.xml
[junit4] 2> 1334513 T2490 oasc.CoreContainer.<init> New CoreContainer 1510273587
[junit4] 2> 1334514 T2490 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/]
[junit4] 2> 1334516 T2490 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1334516 T2490 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1334517 T2490 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1334517 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1334518 T2490 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1334518 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1334519 T2490 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1334519 T2490 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1334520 T2490 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1334520 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1334538 T2490 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:38445/solr
[junit4] 2> 1334538 T2490 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1334539 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1334542 T2562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27657383 name:ZooKeeperConnection Watcher:127.0.0.1:38445 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1334543 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1334545 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1334557 T2490 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1334559 T2564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30302867 name:ZooKeeperConnection Watcher:127.0.0.1:38445/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1334559 T2490 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1334572 T2490 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1335561 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1335563 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38455_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:38455/yiz"}
[junit4] 2> 1335589 T2490 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51578_yiz
[junit4] 2> 1335605 T2490 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51578_yiz
[junit4] 2> 1335606 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335606 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335606 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335606 T2564 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335606 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335617 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335617 T2519 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1335618 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1335618 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335618 T2564 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1335618 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1335619 T2532 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1335620 T2548 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1335620 T2512 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1335626 T2565 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1335626 T2565 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1335628 T2565 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1337119 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1337121 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:51578_yiz",
[junit4] 2> "base_url":"http://127.0.0.1:51578/yiz"}
[junit4] 2> 1337121 T2513 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1337121 T2513 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1337138 T2548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1337138 T2532 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1337138 T2512 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1337138 T2564 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1337138 T2519 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1337632 T2565 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1337632 T2565 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/collection1
[junit4] 2> 1337632 T2565 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1337633 T2565 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1337634 T2565 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1337635 T2565 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/collection1/'
[junit4] 2> 1337636 T2565 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/collection1/lib/classes/' to classloader
[junit4] 2> 1337637 T2565 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/collection1/lib/README' to classloader
[junit4] 2> 1337680 T2565 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1337719 T2565 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1337820 T2565 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1337831 T2565 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1338437 T2565 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1338449 T2565 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1338452 T2565 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1338465 T2565 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1338470 T2565 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1338474 T2565 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1338475 T2565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1338475 T2565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1338476 T2565 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1338476 T2565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1338477 T2565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1338477 T2565 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1338478 T2565 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374738273456/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty3/
[junit4] 2> 1338478 T2565 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@199ee8a8
[junit4] 2> 1338479 T2565 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty3
[junit4] 2> 1338479 T2565 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty3/index/
[junit4] 2> 1338479 T2565 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1338481 T2565 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374738258793/jetty3/index
[junit4] 2> 1338488 T2565 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/index1011510141tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@76ce5486),segFN=segments_1,generation=1}
[junit4] 2> 1338488 T2565 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1338493 T2565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1338493 T2565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1338494 T2565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1338495 T2565 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1338495 T2565 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1338495 T2565 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1338496 T2565 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1338496 T2565 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1338497 T2565 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1338513 T2565 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1338521 T2565 oass.SolrIndexSearcher.<init> Opening Searcher@8ef2bf5 main
[junit4] 2> 1338522 T2565 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1338523 T2565 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1338526 T2566 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8ef2bf5 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1338528 T2565 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1338529 T2565 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51578/yiz collection:collection1 shard:shard1
[junit4] 2> 1338529 T2565 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1338548 T2565 oasc.ZkController.register We are http://127.0.0.1:51578/yiz/collection1/ and leader is http://127.0.0.1:38452/yiz/collection1/
[junit4] 2> 1338549 T2565 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51578/yiz
[junit4] 2> 1338549 T2565 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1338549 T2565 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C669 name=collection1 org.apache.solr.core.SolrCore@4efc8ba4 url=http://127.0.0.1:51578/yiz/collection1 node=127.0.0.1:51578_yiz C669_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:51578_yiz, base_url=http://127.0.0.1:51578/yiz}
[junit4] 2> 1338549 T2567 C669 P51578 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1338550 T2565 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1338550 T2567 C669 P51578 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1338551 T2567 C669 P51578 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1338551 T2567 C669 P51578 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1338551 T2490 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1338552 T2490 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1338552 T2490 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1338552 T2567 C669 P51578 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1338562 T2522 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1338643 T2513 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1338644 T2513 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"recovering"
[...truncated too long message...]
ze=128), sequence_i=Lucene41(blocksize=128), name=Pulsing41(freqCutoff=5 minBlockSize=74 maxBlockSize=170), foo_i=PostingsFormat(name=TestBloomFilteredLucene41Postings), regex_dup_B_s=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), n_tf1=PostingsFormat(name=Direct), n_dt1=PostingsFormat(name=TestBloomFilteredLucene41Postings), genre_s=Lucene41(blocksize=128), author_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_l=PostingsFormat(name=Direct), n_ti1=Pulsing41(freqCutoff=5 minBlockSize=74 maxBlockSize=170), text=Lucene41(blocksize=128), _version_=PostingsFormat(name=Direct), val_i=Pulsing41(freqCutoff=5 minBlockSize=74 maxBlockSize=170), SubjectTerms_mfacet=PostingsFormat(name=Direct), series_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=Direct), n_tdt1=PostingsFormat(name=Direct), regex_dup_A_s=Lucene41(blocksize=128), price=Lucene41(blocksize=128), other_tl1=PostingsFormat(name=Direct), n_l1=PostingsFormat(name=Direct), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), inStock=Lucene41(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=DefaultSimilarity, locale=et, timezone=Europe/Mariehamn
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=224758160,total=320339968
[junit4] 2> NOTE: All tests run in this JVM: [TestSurroundQueryParser, CacheHeaderTest, SuggesterFSTTest, FastVectorHighlighterTest, TestBadConfig, TestSolrXml, TestPerFieldSimilarity, TestIBSimilarityFactory, ScriptEngineTest, FileBasedSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestDocSet, RecoveryZkTest, SpatialFilterTest, PrimUtilsTest, TestShardHandlerFactory, TestStressRecovery, TestConfig, TestGroupingSearch, ChaosMonkeyNothingIsSafeTest, SearchHandlerTest, TestFieldResource, UpdateRequestProcessorFactoryTest, JsonLoaderTest, AnalysisAfterCoreReloadTest, SliceStateUpdateTest, TestXIncludeConfig, DistributedQueryElevationComponentTest, ParsingFieldUpdateProcessorsTest, TestPluginEnable, TestQuerySenderListener, ResourceLoaderTest, TestSerializedLuceneMatchVersion, ClusterStateUpdateTest, InfoHandlerTest, SimpleFacetsTest, TestLRUCache, PreAnalyzedFieldTest, TestFieldTypeCollectionResource, TestFunctionQuery, TestFieldTypeResource, TestSolrIndexConfig, TestSolrDeletionPolicy1, AddSchemaFieldsUpdateProcessorFactoryTest, SuggesterTest, TestPhraseSuggestions, DistributedSpellCheckComponentTest, TestSchemaVersionResource, TestSchemaNameResource, BadComponentTest, QueryEqualityTest, TestRandomDVFaceting, TestWriterPerf, TestCharFilters, ZkCLITest, SyncSliceTest, TermsComponentTest, TestSort, LukeRequestHandlerTest, CoreAdminHandlerTest, OverseerCollectionProcessorTest, UUIDFieldTest, QueryParsingTest, SolrTestCaseJ4Test, TestValueSourceCache, MinimalSchemaTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, ReturnFieldsTest, AliasIntegrationTest, StressHdfsTest, AutoCommitTest, SpellingQueryConverterTest, MoreLikeThisHandlerTest, TestFastWriter, TestCopyFieldCollectionResource, ShardRoutingTest, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=254F2F6795AF1A1F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=et -Dtests.timezone=Europe/Mariehamn -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([254F2F6795AF1A1F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3003, name=recoveryCmdExecutor-1116-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([254F2F6795AF1A1F]:0)
[junit4] Completed on J0 in 531.70s, 1 test, 2 errors <<< FAILURES!
[...truncated 481 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1230: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:873: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 567 ignored (3 assumptions)
Total time: 84 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 329 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/329/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([C4391FDD5D0E0708]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([C4391FDD5D0E0708]:0)
Build Log:
[...truncated 9600 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 586984 T1156 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /c_/uk
[junit4] 2> 586988 T1156 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374567225438
[junit4] 2> 586989 T1156 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 586989 T1157 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 587090 T1156 oasc.ZkTestServer.run start zk server on port:30042
[junit4] 2> 587091 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587098 T1163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a2048ea name:ZooKeeperConnection Watcher:127.0.0.1:30042 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587098 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587098 T1156 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 587118 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587120 T1165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dd7480d name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587121 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587121 T1156 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 587131 T1156 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 587134 T1156 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 587137 T1156 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 587145 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 587146 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 587159 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 587160 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 587263 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 587264 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 587267 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 587267 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 587278 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 587278 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 587281 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 587282 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 587285 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587285 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 587288 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 587289 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 587296 T1156 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 587297 T1156 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 587557 T1156 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 587560 T1156 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63741
[junit4] 2> 587561 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 587561 T1156 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 587562 T1156 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751
[junit4] 2> 587562 T1156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/'
[junit4] 2> 587597 T1156 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/solr.xml
[junit4] 2> 587682 T1156 oasc.CoreContainer.<init> New CoreContainer 1751868799
[junit4] 2> 587683 T1156 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/]
[junit4] 2> 587684 T1156 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 587685 T1156 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 587685 T1156 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 587686 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 587686 T1156 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 587687 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 587687 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 587687 T1156 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 587688 T1156 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 587688 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 587705 T1156 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 587705 T1156 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30042/solr
[junit4] 2> 587706 T1156 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 587707 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587713 T1176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27e9bf53 name:ZooKeeperConnection Watcher:127.0.0.1:30042 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587714 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587717 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 587727 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 587730 T1178 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@561c6ffd name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 587730 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 587732 T1156 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 587742 T1156 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 587746 T1156 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 587753 T1156 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63741_c_%2Fuk
[junit4] 2> 587755 T1156 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63741_c_%2Fuk
[junit4] 2> 587759 T1156 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 587778 T1156 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 587781 T1156 oasc.Overseer.start Overseer (id=90083637686763523-127.0.0.1:63741_c_%2Fuk-n_0000000000) starting
[junit4] 2> 587794 T1156 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 587805 T1180 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 587806 T1156 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 587809 T1156 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 587811 T1156 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 587814 T1179 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 587820 T1181 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 587820 T1181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 587822 T1181 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 589319 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 589320 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:63741_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:63741/c_/uk"}
[junit4] 2> 589321 T1179 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 589321 T1179 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 589338 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 589824 T1181 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 589824 T1181 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/collection1
[junit4] 2> 589824 T1181 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 589825 T1181 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 589826 T1181 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 589827 T1181 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/collection1/'
[junit4] 2> 589828 T1181 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/collection1/lib/README' to classloader
[junit4] 2> 589829 T1181 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/collection1/lib/classes/' to classloader
[junit4] 2> 589881 T1181 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 589944 T1181 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 590046 T1181 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 590056 T1181 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 590625 T1181 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 590637 T1181 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 590641 T1181 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 590657 T1181 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 590662 T1181 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 590666 T1181 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 590668 T1181 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 590668 T1181 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 590668 T1181 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 590670 T1181 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 590670 T1181 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 590670 T1181 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 590671 T1181 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374567225751/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/control/data/
[junit4] 2> 590671 T1181 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4494356f
[junit4] 2> 590673 T1181 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/control/data
[junit4] 2> 590674 T1181 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/control/data/index/
[junit4] 2> 590674 T1181 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 590675 T1181 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/control/data/index
[junit4] 2> 590679 T1181 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2a7c25fb lockFactory=org.apache.lucene.store.NativeFSLockFactory@5347e91c),segFN=segments_1,generation=1}
[junit4] 2> 590680 T1181 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 590683 T1181 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 590683 T1181 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 590684 T1181 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 590685 T1181 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 590685 T1181 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 590685 T1181 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 590686 T1181 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 590686 T1181 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 590686 T1181 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 590703 T1181 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 590710 T1181 oass.SolrIndexSearcher.<init> Opening Searcher@43a205cc main
[junit4] 2> 590712 T1181 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 590712 T1181 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 590715 T1182 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43a205cc main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 590716 T1181 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 590717 T1181 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63741/c_/uk collection:control_collection shard:shard1
[junit4] 2> 590717 T1181 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 590733 T1181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 590753 T1181 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 590756 T1181 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 590756 T1181 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 590756 T1181 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63741/c_/uk/collection1/
[junit4] 2> 590757 T1181 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 590757 T1181 oasc.SyncStrategy.syncToMe http://127.0.0.1:63741/c_/uk/collection1/ has no replicas
[junit4] 2> 590757 T1181 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63741/c_/uk/collection1/ shard1
[junit4] 2> 590758 T1181 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 590843 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 590859 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 590867 T1181 oasc.ZkController.register We are http://127.0.0.1:63741/c_/uk/collection1/ and leader is http://127.0.0.1:63741/c_/uk/collection1/
[junit4] 2> 590867 T1181 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63741/c_/uk
[junit4] 2> 590867 T1181 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 590867 T1181 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 590868 T1181 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 590869 T1181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 590870 T1156 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 590871 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 590871 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 590882 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 590886 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 590891 T1185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2108a072 name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 590891 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 590893 T1156 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 590896 T1156 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 591165 T1156 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 591168 T1156 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:26506
[junit4] 2> 591169 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 591169 T1156 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 591170 T1156 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346
[junit4] 2> 591170 T1156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/'
[junit4] 2> 591205 T1156 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/solr.xml
[junit4] 2> 591290 T1156 oasc.CoreContainer.<init> New CoreContainer 433139994
[junit4] 2> 591291 T1156 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/]
[junit4] 2> 591292 T1156 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 591293 T1156 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 591293 T1156 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 591294 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 591294 T1156 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 591295 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 591295 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 591295 T1156 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 591296 T1156 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 591296 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 591314 T1156 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 591315 T1156 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30042/solr
[junit4] 2> 591315 T1156 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 591316 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 591323 T1196 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c6807ce name:ZooKeeperConnection Watcher:127.0.0.1:30042 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 591324 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 591347 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 591360 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 591362 T1198 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2dca4c76 name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 591362 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 591375 T1156 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 592366 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 592367 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:63741_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:63741/c_/uk"}
[junit4] 2> 592383 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592383 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592383 T1156 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:26506_c_%2Fuk
[junit4] 2> 592383 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592386 T1156 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:26506_c_%2Fuk
[junit4] 2> 592389 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592389 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4] 2> 592390 T1198 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 592390 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 592391 T1178 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 592391 T1185 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 592403 T1199 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 592403 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 592406 T1199 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 593889 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 593890 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:26506_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:26506/c_/uk"}
[junit4] 2> 593890 T1179 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 593891 T1179 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 593905 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 593905 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 593905 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 594408 T1199 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 594408 T1199 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/collection1
[junit4] 2> 594408 T1199 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 594409 T1199 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 594409 T1199 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 594411 T1199 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/collection1/'
[junit4] 2> 594412 T1199 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/collection1/lib/README' to classloader
[junit4] 2> 594413 T1199 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/collection1/lib/classes/' to classloader
[junit4] 2> 594465 T1199 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 594528 T1199 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 594629 T1199 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 594639 T1199 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 595214 T1199 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 595226 T1199 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 595230 T1199 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 595247 T1199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 595252 T1199 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 595258 T1199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 595259 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 595260 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 595260 T1199 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 595261 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 595261 T1199 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 595262 T1199 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 595262 T1199 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374567229346/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty1/
[junit4] 2> 595262 T1199 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4494356f
[junit4] 2> 595265 T1199 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty1
[junit4] 2> 595266 T1199 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty1/index/
[junit4] 2> 595266 T1199 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 595266 T1199 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty1/index
[junit4] 2> 595270 T1199 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6ed0ccc2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2af9862d),segFN=segments_1,generation=1}
[junit4] 2> 595271 T1199 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 595275 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 595275 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 595276 T1199 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 595276 T1199 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 595277 T1199 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 595277 T1199 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 595278 T1199 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 595278 T1199 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 595279 T1199 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 595296 T1199 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 595304 T1199 oass.SolrIndexSearcher.<init> Opening Searcher@11eabd93 main
[junit4] 2> 595305 T1199 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 595306 T1199 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 595309 T1200 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@11eabd93 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 595311 T1199 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 595311 T1199 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:26506/c_/uk collection:collection1 shard:shard1
[junit4] 2> 595311 T1199 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 595331 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 595348 T1199 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 595359 T1199 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 595360 T1199 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 595360 T1199 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:26506/c_/uk/collection1/
[junit4] 2> 595360 T1199 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 595361 T1199 oasc.SyncStrategy.syncToMe http://127.0.0.1:26506/c_/uk/collection1/ has no replicas
[junit4] 2> 595361 T1199 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:26506/c_/uk/collection1/ shard1
[junit4] 2> 595361 T1199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 595410 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595426 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595426 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595426 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 595477 T1199 oasc.ZkController.register We are http://127.0.0.1:26506/c_/uk/collection1/ and leader is http://127.0.0.1:26506/c_/uk/collection1/
[junit4] 2> 595477 T1199 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:26506/c_/uk
[junit4] 2> 595477 T1199 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 595477 T1199 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 595478 T1199 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 595480 T1199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 595481 T1156 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 595482 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 595482 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 595770 T1156 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 595773 T1156 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:42351
[junit4] 2> 595773 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 595774 T1156 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 595774 T1156 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944
[junit4] 2> 595775 T1156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/'
[junit4] 2> 595810 T1156 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/solr.xml
[junit4] 2> 595897 T1156 oasc.CoreContainer.<init> New CoreContainer 2006674892
[junit4] 2> 595898 T1156 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/]
[junit4] 2> 595900 T1156 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 595900 T1156 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 595901 T1156 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 595901 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 595902 T1156 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 595902 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 595902 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 595903 T1156 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 595903 T1156 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 595904 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 595921 T1156 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 595922 T1156 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30042/solr
[junit4] 2> 595923 T1156 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 595924 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 595952 T1212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27b95600 name:ZooKeeperConnection Watcher:127.0.0.1:30042 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 595952 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 595994 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 596007 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 596013 T1214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6075debf name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 596014 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 596032 T1156 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 596941 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 596942 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:26506_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:26506/c_/uk"}
[junit4] 2> 596965 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 596966 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 596965 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 596965 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 597035 T1156 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42351_c_%2Fuk
[junit4] 2> 597037 T1156 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42351_c_%2Fuk
[junit4] 2> 597040 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 597040 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 597040 T1214 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 597040 T1198 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 597040 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 597041 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 597042 T1178 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 597042 T1185 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 597057 T1215 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 597058 T1215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 597059 T1215 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 598477 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 598478 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42351_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:42351/c_/uk"}
[junit4] 2> 598479 T1179 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 598479 T1179 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 598497 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 598497 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 598497 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 598497 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 599062 T1215 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 599062 T1215 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/collection1
[junit4] 2> 599062 T1215 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 599063 T1215 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 599064 T1215 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 599065 T1215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/collection1/'
[junit4] 2> 599066 T1215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/collection1/lib/README' to classloader
[junit4] 2> 599067 T1215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/collection1/lib/classes/' to classloader
[junit4] 2> 599125 T1215 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 599183 T1215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 599285 T1215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 599295 T1215 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 599854 T1215 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 599862 T1215 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 599864 T1215 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 599878 T1215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 599882 T1215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 599886 T1215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 599887 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 599887 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 599888 T1215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 599889 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 599889 T1215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 599889 T1215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 599889 T1215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374567233944/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty2/
[junit4] 2> 599889 T1215 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4494356f
[junit4] 2> 599891 T1215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty2
[junit4] 2> 599892 T1215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty2/index/
[junit4] 2> 599892 T1215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 599892 T1215 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty2/index
[junit4] 2> 599895 T1215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@36e6f68d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fb4bac6),segFN=segments_1,generation=1}
[junit4] 2> 599895 T1215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 599898 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 599898 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 599899 T1215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 599899 T1215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 599900 T1215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 599900 T1215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 599900 T1215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 599900 T1215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 599901 T1215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 599911 T1215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 599916 T1215 oass.SolrIndexSearcher.<init> Opening Searcher@713c9f08 main
[junit4] 2> 599917 T1215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 599918 T1215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 599920 T1216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@713c9f08 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 599921 T1215 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 599922 T1215 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42351/c_/uk collection:collection1 shard:shard2
[junit4] 2> 599922 T1215 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 599934 T1215 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 599953 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 599963 T1215 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 599963 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 599964 T1215 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42351/c_/uk/collection1/
[junit4] 2> 599964 T1215 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 599964 T1215 oasc.SyncStrategy.syncToMe http://127.0.0.1:42351/c_/uk/collection1/ has no replicas
[junit4] 2> 599964 T1215 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42351/c_/uk/collection1/ shard2
[junit4] 2> 599964 T1215 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 600002 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 600018 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 600018 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 600018 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 600018 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 600034 T1215 oasc.ZkController.register We are http://127.0.0.1:42351/c_/uk/collection1/ and leader is http://127.0.0.1:42351/c_/uk/collection1/
[junit4] 2> 600034 T1215 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42351/c_/uk
[junit4] 2> 600034 T1215 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 600034 T1215 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 600035 T1215 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 600036 T1215 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 600037 T1156 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 600038 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 600038 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 600227 T1156 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 600229 T1156 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38974
[junit4] 2> 600229 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 600230 T1156 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 600230 T1156 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496
[junit4] 2> 600230 T1156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/'
[junit4] 2> 600253 T1156 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/solr.xml
[junit4] 2> 600309 T1156 oasc.CoreContainer.<init> New CoreContainer 849026707
[junit4] 2> 600310 T1156 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/]
[junit4] 2> 600311 T1156 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 600311 T1156 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 600311 T1156 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 600312 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 600312 T1156 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 600312 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 600313 T1156 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 600313 T1156 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 600313 T1156 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 600313 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 600325 T1156 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 600326 T1156 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:30042/solr
[junit4] 2> 600326 T1156 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 600327 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 600330 T1228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f59a3a5 name:ZooKeeperConnection Watcher:127.0.0.1:30042 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 600330 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 600347 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 600360 T1156 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 600362 T1230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6789e638 name:ZooKeeperConnection Watcher:127.0.0.1:30042/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 600363 T1156 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 600375 T1156 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 601379 T1156 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38974_c_%2Fuk
[junit4] 2> 601384 T1156 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38974_c_%2Fuk
[junit4] 2> 601387 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 601387 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 601388 T1214 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 601388 T1230 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 601388 T1198 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 601388 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601389 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601390 T1185 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 601390 T1178 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 601407 T1231 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 601407 T1231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 601408 T1231 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 601525 T1179 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 601526 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:42351_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:42351/c_/uk"}
[junit4] 2> 601531 T1179 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:38974_c_%2Fuk",
[junit4] 2> "base_url":"http://127.0.0.1:38974/c_/uk"}
[junit4] 2> 601532 T1179 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 601532 T1179 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 601548 T1214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601548 T1230 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601548 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601548 T1198 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 601548 T1178 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 602410 T1231 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 602410 T1231 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/collection1
[junit4] 2> 602410 T1231 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 602411 T1231 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 602411 T1231 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 602413 T1231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/collection1/'
[junit4] 2> 602414 T1231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/collection1/lib/classes/' to classloader
[junit4] 2> 602415 T1231 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/collection1/lib/README' to classloader
[junit4] 2> 602467 T1231 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 602525 T1231 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 602626 T1231 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 602636 T1231 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 603209 T1231 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 603227 T1231 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 603231 T1231 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 603247 T1231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 603252 T1231 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 603257 T1231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 603259 T1231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 603259 T1231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 603259 T1231 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 603261 T1231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 603261 T1231 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 603261 T1231 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 603262 T1231 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374567238496/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty3/
[junit4] 2> 603262 T1231 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4494356f
[junit4] 2> 603264 T1231 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty3
[junit4] 2> 603265 T1231 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty3/index/
[junit4] 2> 603265 T1231 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 603266 T1231 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374567225438/jetty3/index
[junit4] 2> 603269 T1231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4cb4ec73 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74647510),segFN=segments_1,generation=1}
[junit4] 2> 603270 T1231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 603274 T1231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 603274 T1231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 603275 T1231 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 603275 T1231 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 603276 T1231 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 603276 T1231 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 603276 T1231 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 603277 T1231 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 603277 T1231 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 603294 T1231 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 603301 T1231 oass.SolrIndexSearcher.<init> Opening Searcher@7b4f42b main
[junit4] 2> 603302 T1231 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 603303 T1231 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 603306 T1232 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b4f42b main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 603308 T1231 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 603308 T1231 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38974/c_/uk collection:collection1 shard:shard1
[junit4] 2> 603308 T1231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 603330 T1231 oasc.ZkController.register We are http://127.0.0.1:38974/c_/uk/collection1/ and leader is http://127.0.0.1:26506/c_/uk/collection1/
[junit4] 2> 603330 T1231 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38974/c_/uk
[junit4] 2> 603330 T1231 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 603330 T1231 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C661 name=collection1 org.apache.solr.core.SolrCore@2b4568e5 url=http://127.0.0.1:38974/c_/uk/collection1 node=127.0.0.1:38974_c_%2Fuk C661_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:38974_c_%2Fuk, base_url=http://127.0.0.1:38974/c_/uk}
[junit4] 2> 603331 T1233 C661 P38974 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 603331 T1233 C661 P38974 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 603331 T1231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 603332 T1233 C661 P38974 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 603332 T1233 C661 P38974 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 603333 T1156 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 603333 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 603334 T1233 C661 P38974 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 603334 T1156 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 603351 T1188 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 603617 T1156 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 603620 T1156 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:52668
[junit4] 2> 603620 T1156 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 603621 T1156 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 603621 T1156 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374567241800
[junit4] 2> 603622 T1156 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374567241800/'
[junit4] 2> 603657 T1156 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374567241800/solr.xml
[junit4] 2> 603742 T1156 oasc.CoreContainer.<init> New CoreContainer 449200358
[junit4] 2> 603743 T1156 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374567241800/]
[junit4] 2> 603745 T1156 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 603745 T1156 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 603745 T1156 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 603746 T1156 oashc.HttpShardHandlerFactory.getParameter Se
[...truncated too long message...]
rg.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_MX, timezone=Africa/Blantyre
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=159114760,total=358547456
[junit4] 2> NOTE: All tests run in this JVM: [AutoCommitTest, IndexSchemaTest, AnalysisAfterCoreReloadTest, FileBasedSpellCheckerTest, FastVectorHighlighterTest, TestSolrCoreProperties, BlockDirectoryTest, TestFastLRUCache, TestSolrQueryParser, TestSolrDeletionPolicy2, TestSolrQueryParserDefaultOperatorResource, TestCopyFieldCollectionResource, QueryEqualityTest, ScriptEngineTest, RegexBoostProcessorTest, CacheHeaderTest, TestJoin, TestSolrQueryParserResource, CoreAdminHandlerTest, UUIDFieldTest, LukeRequestHandlerTest, TestConfig, TestStressVersions, TestPseudoReturnFields, TestSolrIndexConfig, PrimitiveFieldTypeTest, TestShardHandlerFactory, BinaryUpdateRequestHandlerTest, TestLRUCache, TestSchemaVersionResource, ZkControllerTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestFaceting, TestQuerySenderListener, TestDocSet, SuggesterTest, AliasIntegrationTest, OverseerCollectionProcessorTest, TestPHPSerializedResponseWriter, TestAddFieldRealTimeGet, SynonymTokenizerTest, SuggesterFSTTest, SearchHandlerTest, PreAnalyzedFieldTest, SpatialFilterTest, SolrTestCaseJ4Test, LoggingHandlerTest, TestValueSourceCache, SolrCoreTest, ShowFileRequestHandlerTest, BadComponentTest, SchemaVersionSpecificBehaviorTest, TestSurroundQueryParser, PrimUtilsTest, DocValuesTest, MBeansHandlerTest, StressHdfsTest, RAMDirectoryFactoryTest, TestSerializedLuceneMatchVersion, SystemInfoHandlerTest, DocumentBuilderTest, TestCoreContainer, SliceStateTest, ResourceLoaderTest, BadCopyFieldTest, TestStressRecovery, StatelessScriptUpdateProcessorFactoryTest, PluginInfoTest, SpellingQueryConverterTest, AssignTest, TestFieldTypeCollectionResource, TestDynamicFieldCollectionResource, TestGroupingSearch, TestSearchPerf, ResponseLogComponentTest, ClusterStateUpdateTest, RecoveryZkTest, TermsComponentTest, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=C4391FDD5D0E0708 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_MX -Dtests.timezone=Africa/Blantyre -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([C4391FDD5D0E0708]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=1665, name=recoveryCmdExecutor-792-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([C4391FDD5D0E0708]:0)
[junit4] Completed on J0 in 416.32s, 1 test, 2 errors <<< FAILURES!
[...truncated 549 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1247: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:890: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 671 ignored (4 assumptions)
Total time: 90 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 328 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/328/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([467975AF4A11C5A7]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([467975AF4A11C5A7]:0)
Build Log:
[...truncated 9683 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 1407854 T2790 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1407860 T2790 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZkTest-1374506353654
[junit4] 2> 1407861 T2790 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1407862 T2791 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1407963 T2790 oasc.ZkTestServer.run start zk server on port:13631
[junit4] 2> 1407964 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1407968 T2797 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7270548b name:ZooKeeperConnection Watcher:127.0.0.1:13631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1407968 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1407969 T2790 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1407992 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1407994 T2799 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cb0c245 name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1407994 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1407994 T2790 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1408014 T2790 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1408117 T2790 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1408130 T2790 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1408133 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1408133 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1408215 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1408216 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1408325 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1408326 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1408330 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1408330 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1408349 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1408350 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1408354 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1408354 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1408360 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1408361 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1408365 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1408365 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1408373 T2790 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1408373 T2790 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1408676 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1408681 T2790 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:49018
[junit4] 2> 1408681 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1408682 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1408682 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182
[junit4] 2> 1408683 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/'
[junit4] 2> 1408723 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/solr.xml
[junit4] 2> 1408821 T2790 oasc.CoreContainer.<init> New CoreContainer 1494253927
[junit4] 2> 1408821 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/]
[junit4] 2> 1408823 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1408824 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1408824 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1408825 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1408825 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1408826 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1408826 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1408826 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1408827 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1408827 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1408846 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1408847 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13631/solr
[junit4] 2> 1408848 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1408849 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1408852 T2810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79dc133a name:ZooKeeperConnection Watcher:127.0.0.1:13631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1408852 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1408871 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1408884 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1408886 T2812 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3097333 name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1408886 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1408889 T2790 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1408899 T2790 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1408903 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1408905 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49018_
[junit4] 2> 1408912 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49018_
[junit4] 2> 1408916 T2790 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1408929 T2790 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1408932 T2790 oasc.Overseer.start Overseer (id=90079648393723907-127.0.0.1:49018_-n_0000000000) starting
[junit4] 2> 1408946 T2790 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1408958 T2814 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1408959 T2790 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1408962 T2790 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1408964 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1408967 T2813 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1408972 T2815 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1408972 T2815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1408974 T2815 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1410471 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1410472 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:49018_",
[junit4] 2> "base_url":"http://127.0.0.1:49018"}
[junit4] 2> 1410472 T2813 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 1410472 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1410494 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1410976 T2815 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1410976 T2815 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/collection1
[junit4] 2> 1410976 T2815 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1410977 T2815 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1410977 T2815 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1410978 T2815 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/collection1/'
[junit4] 2> 1410979 T2815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/collection1/lib/README' to classloader
[junit4] 2> 1410980 T2815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/collection1/lib/classes/' to classloader
[junit4] 2> 1411018 T2815 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1411057 T2815 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1411159 T2815 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1411167 T2815 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1411775 T2815 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1411784 T2815 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1411787 T2815 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1411800 T2815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1411804 T2815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1411807 T2815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1411808 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1411808 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1411808 T2815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1411809 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1411810 T2815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1411810 T2815 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1411810 T2815 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374506354182/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/control/data/
[junit4] 2> 1411810 T2815 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a4fce76
[junit4] 2> 1411811 T2815 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/control/data
[junit4] 2> 1411811 T2815 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/control/data/index/
[junit4] 2> 1411811 T2815 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1411812 T2815 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/control/data/index
[junit4] 2> 1411815 T2815 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7f8fa11f lockFactory=org.apache.lucene.store.NativeFSLockFactory@4e8c2bb8),segFN=segments_1,generation=1}
[junit4] 2> 1411815 T2815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1411818 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1411818 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1411819 T2815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1411819 T2815 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1411820 T2815 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1411820 T2815 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1411820 T2815 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1411820 T2815 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1411821 T2815 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1411831 T2815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1411836 T2815 oass.SolrIndexSearcher.<init> Opening Searcher@2fcb7ba8 main
[junit4] 2> 1411837 T2815 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1411837 T2815 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1411840 T2816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fcb7ba8 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1411841 T2815 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1411841 T2815 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49018 collection:control_collection shard:shard1
[junit4] 2> 1411841 T2815 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1411853 T2815 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1411875 T2815 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1411877 T2815 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1411878 T2815 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1411878 T2815 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49018/collection1/
[junit4] 2> 1411878 T2815 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1411878 T2815 oasc.SyncStrategy.syncToMe http://127.0.0.1:49018/collection1/ has no replicas
[junit4] 2> 1411878 T2815 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49018/collection1/ shard1
[junit4] 2> 1411878 T2815 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1412000 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1412017 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1412038 T2815 oasc.ZkController.register We are http://127.0.0.1:49018/collection1/ and leader is http://127.0.0.1:49018/collection1/
[junit4] 2> 1412038 T2815 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49018
[junit4] 2> 1412038 T2815 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1412038 T2815 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1412039 T2815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1412040 T2815 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1412041 T2790 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 1412042 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1412043 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1412055 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1412057 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1412059 T2819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a54b779 name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1412059 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1412061 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1412063 T2790 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1412252 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1412255 T2790 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:48251
[junit4] 2> 1412255 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1412256 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1412256 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857
[junit4] 2> 1412257 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/'
[junit4] 2> 1412284 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/solr.xml
[junit4] 2> 1412354 T2790 oasc.CoreContainer.<init> New CoreContainer 1838947562
[junit4] 2> 1412355 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/]
[junit4] 2> 1412356 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1412357 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1412357 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1412357 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1412358 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1412358 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1412358 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1412359 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1412359 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1412359 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1412371 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1412371 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13631/solr
[junit4] 2> 1412372 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1412373 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1412376 T2830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ada193f name:ZooKeeperConnection Watcher:127.0.0.1:13631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1412376 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1412392 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1412405 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1412406 T2832 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ff3bd65 name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1412407 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1412412 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1413415 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48251_
[junit4] 2> 1413418 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48251_
[junit4] 2> 1413450 T2819 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1413450 T2832 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1413450 T2812 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1413451 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1413459 T2833 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1413459 T2833 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1413461 T2833 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1413531 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1413532 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:49018_",
[junit4] 2> "base_url":"http://127.0.0.1:49018"}
[junit4] 2> 1413543 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:48251_",
[junit4] 2> "base_url":"http://127.0.0.1:48251"}
[junit4] 2> 1413544 T2813 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 1413544 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1413561 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1413561 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1413561 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1414463 T2833 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1414464 T2833 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/collection1
[junit4] 2> 1414464 T2833 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1414465 T2833 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1414465 T2833 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1414466 T2833 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/collection1/'
[junit4] 2> 1414467 T2833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/collection1/lib/classes/' to classloader
[junit4] 2> 1414468 T2833 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/collection1/lib/README' to classloader
[junit4] 2> 1414504 T2833 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1414544 T2833 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1414645 T2833 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1414652 T2833 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1415073 T2833 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1415082 T2833 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1415085 T2833 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1415096 T2833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1415100 T2833 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1415104 T2833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1415105 T2833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1415105 T2833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1415105 T2833 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1415106 T2833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1415106 T2833 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1415107 T2833 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1415107 T2833 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374506357857/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty1/
[junit4] 2> 1415107 T2833 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a4fce76
[junit4] 2> 1415108 T2833 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty1
[junit4] 2> 1415108 T2833 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty1/index/
[junit4] 2> 1415108 T2833 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1415109 T2833 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty1/index
[junit4] 2> 1415112 T2833 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@491c2280 lockFactory=org.apache.lucene.store.NativeFSLockFactory@40c54153),segFN=segments_1,generation=1}
[junit4] 2> 1415112 T2833 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1415116 T2833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1415116 T2833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1415116 T2833 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1415117 T2833 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1415117 T2833 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1415117 T2833 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1415118 T2833 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1415118 T2833 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1415118 T2833 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1415131 T2833 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1415137 T2833 oass.SolrIndexSearcher.<init> Opening Searcher@2af4d940 main
[junit4] 2> 1415138 T2833 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1415138 T2833 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1415141 T2834 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2af4d940 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1415143 T2833 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1415143 T2833 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48251 collection:collection1 shard:shard1
[junit4] 2> 1415143 T2833 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1415156 T2833 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1415190 T2833 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1415200 T2833 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1415200 T2833 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1415200 T2833 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48251/collection1/
[junit4] 2> 1415200 T2833 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1415201 T2833 oasc.SyncStrategy.syncToMe http://127.0.0.1:48251/collection1/ has no replicas
[junit4] 2> 1415201 T2833 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48251/collection1/ shard1
[junit4] 2> 1415201 T2833 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1416570 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1416588 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1416588 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1416588 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1416593 T2833 oasc.ZkController.register We are http://127.0.0.1:48251/collection1/ and leader is http://127.0.0.1:48251/collection1/
[junit4] 2> 1416593 T2833 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48251
[junit4] 2> 1416593 T2833 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1416593 T2833 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1416594 T2833 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1416599 T2833 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1416600 T2790 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 1416601 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1416601 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1416903 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1416907 T2790 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28891
[junit4] 2> 1416907 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1416908 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1416908 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406
[junit4] 2> 1416909 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/'
[junit4] 2> 1416947 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/solr.xml
[junit4] 2> 1417040 T2790 oasc.CoreContainer.<init> New CoreContainer 1898463364
[junit4] 2> 1417041 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/]
[junit4] 2> 1417043 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1417043 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1417044 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1417044 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1417045 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1417045 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1417046 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1417046 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1417047 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1417047 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1417065 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1417066 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13631/solr
[junit4] 2> 1417067 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1417068 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1417071 T2846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cdb1090 name:ZooKeeperConnection Watcher:127.0.0.1:13631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1417072 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1417086 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1417098 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1417100 T2848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1239bbf6 name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1417100 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1417110 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1418101 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1418102 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:48251_",
[junit4] 2> "base_url":"http://127.0.0.1:48251"}
[junit4] 2> 1418109 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418109 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418109 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418109 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418154 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28891_
[junit4] 2> 1418219 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28891_
[junit4] 2> 1418222 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418222 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418222 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1418223 T2812 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1418223 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1418224 T2819 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1418225 T2848 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1418255 T2832 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1418262 T2849 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1418262 T2849 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1418264 T2849 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1419657 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1419658 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:28891_",
[junit4] 2> "base_url":"http://127.0.0.1:28891"}
[junit4] 2> 1419659 T2813 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1419659 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1419679 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1419679 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1419679 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1419679 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1420266 T2849 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1420266 T2849 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/collection1
[junit4] 2> 1420266 T2849 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1420267 T2849 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1420268 T2849 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1420269 T2849 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/collection1/'
[junit4] 2> 1420270 T2849 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/collection1/lib/README' to classloader
[junit4] 2> 1420271 T2849 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/collection1/lib/classes/' to classloader
[junit4] 2> 1420326 T2849 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1420390 T2849 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1420492 T2849 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1420502 T2849 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1420998 T2849 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1421007 T2849 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1421010 T2849 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1421023 T2849 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1421027 T2849 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1421031 T2849 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1421032 T2849 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1421032 T2849 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1421032 T2849 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1421033 T2849 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1421033 T2849 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1421033 T2849 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1421034 T2849 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374506362406/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty2/
[junit4] 2> 1421034 T2849 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a4fce76
[junit4] 2> 1421035 T2849 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty2
[junit4] 2> 1421035 T2849 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty2/index/
[junit4] 2> 1421035 T2849 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1421035 T2849 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty2/index
[junit4] 2> 1421038 T2849 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@32600778 lockFactory=org.apache.lucene.store.NativeFSLockFactory@354343a3),segFN=segments_1,generation=1}
[junit4] 2> 1421039 T2849 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1421042 T2849 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1421042 T2849 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1421043 T2849 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1421043 T2849 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1421044 T2849 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1421044 T2849 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1421044 T2849 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1421045 T2849 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1421045 T2849 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1421058 T2849 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1421065 T2849 oass.SolrIndexSearcher.<init> Opening Searcher@5191ab3b main
[junit4] 2> 1421066 T2849 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1421066 T2849 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1421069 T2850 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5191ab3b main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1421071 T2849 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1421071 T2849 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28891 collection:collection1 shard:shard2
[junit4] 2> 1421071 T2849 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1421085 T2849 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1421104 T2849 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1421113 T2849 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1421114 T2849 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1421114 T2849 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28891/collection1/
[junit4] 2> 1421114 T2849 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1421114 T2849 oasc.SyncStrategy.syncToMe http://127.0.0.1:28891/collection1/ has no replicas
[junit4] 2> 1421114 T2849 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28891/collection1/ shard2
[junit4] 2> 1421114 T2849 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1421184 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1421201 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1421201 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1421201 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1421201 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1421229 T2849 oasc.ZkController.register We are http://127.0.0.1:28891/collection1/ and leader is http://127.0.0.1:28891/collection1/
[junit4] 2> 1421229 T2849 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28891
[junit4] 2> 1421229 T2849 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1421229 T2849 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1421229 T2849 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1421231 T2849 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1421232 T2790 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 1421233 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1421233 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1421442 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1421445 T2790 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35176
[junit4] 2> 1421445 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1421446 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1421446 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034
[junit4] 2> 1421447 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/'
[junit4] 2> 1421475 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/solr.xml
[junit4] 2> 1421541 T2790 oasc.CoreContainer.<init> New CoreContainer 606939796
[junit4] 2> 1421542 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/]
[junit4] 2> 1421543 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1421543 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1421544 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1421544 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1421544 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1421544 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1421545 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1421545 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1421545 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1421546 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1421557 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1421558 T2790 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:13631/solr
[junit4] 2> 1421558 T2790 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1421559 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1421562 T2862 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e0650d0 name:ZooKeeperConnection Watcher:127.0.0.1:13631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1421563 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1421565 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1421574 T2790 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1421576 T2864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@245a3f4b name:ZooKeeperConnection Watcher:127.0.0.1:13631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1421576 T2790 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1421588 T2790 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1422592 T2790 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35176_
[junit4] 2> 1422595 T2790 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35176_
[junit4] 2> 1422598 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1422598 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1422598 T2864 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1422598 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1422599 T2812 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1422599 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1422600 T2819 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1422608 T2848 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1422608 T2832 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1422622 T2865 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1422622 T2865 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1422624 T2865 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1422715 T2813 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1422716 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:28891_",
[junit4] 2> "base_url":"http://127.0.0.1:28891"}
[junit4] 2> 1422721 T2813 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:35176_",
[junit4] 2> "base_url":"http://127.0.0.1:35176"}
[junit4] 2> 1422721 T2813 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1422721 T2813 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1422732 T2864 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1422732 T2812 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1422732 T2832 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1422733 T2848 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1422732 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1423625 T2865 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1423625 T2865 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/collection1
[junit4] 2> 1423625 T2865 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1423626 T2865 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1423627 T2865 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1423628 T2865 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/collection1/'
[junit4] 2> 1423629 T2865 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/collection1/lib/README' to classloader
[junit4] 2> 1423630 T2865 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/collection1/lib/classes/' to classloader
[junit4] 2> 1423683 T2865 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1423740 T2865 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1423841 T2865 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1423851 T2865 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1424377 T2865 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1424385 T2865 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1424388 T2865 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1424401 T2865 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1424405 T2865 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1424408 T2865 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1424409 T2865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1424409 T2865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1424409 T2865 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1424410 T2865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1424411 T2865 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1424411 T2865 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1424411 T2865 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374506367034/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty3/
[junit4] 2> 1424411 T2865 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a4fce76
[junit4] 2> 1424412 T2865 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty3
[junit4] 2> 1424412 T2865 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty3/index/
[junit4] 2> 1424412 T2865 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1424413 T2865 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374506353653/jetty3/index
[junit4] 2> 1424415 T2865 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7f8a1af5 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6baf0d91),segFN=segments_1,generation=1}
[junit4] 2> 1424416 T2865 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1424419 T2865 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1424420 T2865 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1424420 T2865 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1424421 T2865 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1424421 T2865 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1424421 T2865 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1424422 T2865 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1424422 T2865 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1424422 T2865 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1424434 T2865 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1424439 T2865 oass.SolrIndexSearcher.<init> Opening Searcher@5f6d1d59 main
[junit4] 2> 1424440 T2865 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1424440 T2865 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1424444 T2866 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f6d1d59 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1424445 T2865 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1424446 T2865 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35176 collection:collection1 shard:shard1
[junit4] 2> 1424446 T2865 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1424461 T2865 oasc.ZkController.register We are http://127.0.0.1:35176/collection1/ and leader is http://127.0.0.1:48251/collection1/
[junit4] 2> 1424461 T2865 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35176
[junit4] 2> 1424461 T2865 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1424461 T2865 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C18794 name=collection1 org.apache.solr.core.SolrCore@1898da29 url=http://127.0.0.1:35176/collection1 node=127.0.0.1:35176_ C18794_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:35176_, base_url=http://127.0.0.1:35176}
[junit4] 2> 1424467 T2867 C18794 P35176 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1424468 T2865 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1424469 T2867 C18794 P35176 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1424469 T2867 C18794 P35176 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1424469 T2867 C18794 P35176 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1424470 T2790 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
[junit4] 2> 1424470 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1424471 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1424471 T2867 C18794 P35176 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1424495 T2822 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1424788 T2790 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1424791 T2790 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:21913
[junit4] 2> 1424792 T2790 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1424793 T2790 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1424793 T2790 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374506370277
[junit4] 2> 1424793 T2790 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374506370277/'
[junit4] 2> 1424831 T2790 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374506370277/solr.xml
[junit4] 2> 1424921 T2790 oasc.CoreContainer.<init> New CoreContainer 1280682196
[junit4] 2> 1424922 T2790 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374506370277/]
[junit4] 2> 1424924 T2790 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1424924 T2790 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1424925 T2790 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1424925 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1424926 T2790 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1424926 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1424926 T2790 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1424927 T2790 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1424927 T2790 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1424928 T2790 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1424946 T2790 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1424947 T2790 oasc.ZkC
[...truncated too long message...]
kSize=49), sequence_i=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49), name=PostingsFormat(name=Asserting), foo_i=PostingsFormat(name=SimpleText), regex_dup_B_s=MockFixedIntBlock(blockSize=1320), multiDefault=PostingsFormat(name=SimpleText), n_tf1=MockFixedIntBlock(blockSize=1320), n_dt1=PostingsFormat(name=SimpleText), genre_s=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49), author_t=PostingsFormat(name=SimpleText), range_facet_l=MockFixedIntBlock(blockSize=1320), n_ti1=PostingsFormat(name=Asserting), text=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49), _version_=MockFixedIntBlock(blockSize=1320), val_i=PostingsFormat(name=Asserting), SubjectTerms_mfacet=MockFixedIntBlock(blockSize=1320), series_t=PostingsFormat(name=SimpleText), a_t=MockFixedIntBlock(blockSize=1320), n_tdt1=MockFixedIntBlock(blockSize=1320), regex_dup_A_s=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49), price=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49), other_tl1=MockFixedIntBlock(blockSize=1320), n_l1=MockFixedIntBlock(blockSize=1320), a_si=PostingsFormat(name=SimpleText), inStock=Pulsing41(freqCutoff=9 minBlockSize=21 maxBlockSize=49)}, docValues:{timestamp=DocValuesFormat(name=Disk)}, sim=DefaultSimilarity, locale=es_US, timezone=Africa/Mogadishu
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=219201328,total=529727488
[junit4] 2> NOTE: All tests run in this JVM: [BasicZkTest, TestJmxIntegration, TestQueryTypes, DocValuesMultiTest, TestReload, TestNumberUtils, HdfsBasicDistributedZk2Test, TestRecovery, TestManagedSchema, TestCSVResponseWriter, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, ZkNodePropsTest, SpellCheckCollatorTest, TestManagedSchemaFieldResource, HighlighterTest, TestLMJelinekMercerSimilarityFactory, TestRangeQuery, HdfsRecoveryZkTest, SolrCmdDistributorTest, TestDFRSimilarityFactory, QueryResultKeyTest, ConvertedLegacyTest, TestSolrJ, SimplePostToolTest, CurrencyFieldXmlFileTest, BlockCacheTest, NumericFieldsTest, CurrencyFieldOpenExchangeTest, TimeZoneUtilsTest, BadIndexSchemaTest, OutputWriterTest, CoreContainerCoreInitFailuresTest, TestAnalyzedSuggestions, TestStressLucene, DirectUpdateHandlerTest, TestCollationField, HdfsChaosMonkeySafeLeaderTest, DebugComponentTest, XmlUpdateRequestHandlerTest, TestCoreDiscovery, HardAutoCommitTest, SampleTest, TestSolrXMLSerializer, TestUpdate, TestRTGBase, TestBinaryField, TestClassNameShortening, RequestHandlersTest, OpenCloseCoreStressTest, TestElisionMultitermQuery, TestCSVLoader, LeaderElectionIntegrationTest, TestHashPartitioner, DisMaxRequestHandlerTest, HighlighterConfigTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentDistributedTest, CopyFieldTest, HdfsSyncSliceTest, UniqFieldsUpdateProcessorFactoryTest, SpellCheckComponentTest, NotRequiredUniqueKeyTest, PingRequestHandlerTest, TestLuceneMatchVersion, TestMultiCoreConfBootstrap, ShardRoutingCustomTest, TestIndexingPerformance, TestFiltering, HdfsLockFactoryTest, OpenExchangeRatesOrgProviderTest, TestUtils, FieldAnalysisRequestHandlerTest, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=467975AF4A11C5A7 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_US -Dtests.timezone=Africa/Mogadishu -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([467975AF4A11C5A7]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3296, name=recoveryCmdExecutor-1414-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([467975AF4A11C5A7]:0)
[junit4] Completed on J1 in 419.88s, 1 test, 2 errors <<< FAILURES!
[...truncated 479 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1247: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:890: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 669 ignored (2 assumptions)
Total time: 86 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 327 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/327/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([8C9846E0944CD96C]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([8C9846E0944CD96C]:0)
Build Log:
[...truncated 9589 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 1479119 T2367 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1479126 T2367 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374412259653
[junit4] 2> 1479128 T2367 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1479128 T2368 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1479229 T2367 oasc.ZkTestServer.run start zk server on port:19230
[junit4] 2> 1479230 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1479236 T2374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c2863d5 name:ZooKeeperConnection Watcher:127.0.0.1:19230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1479237 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1479237 T2367 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1479255 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1479257 T2376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@760abc1b name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1479257 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1479258 T2367 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1479268 T2367 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1479272 T2367 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1479275 T2367 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1479279 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1479280 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1479292 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1479293 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1479398 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1479399 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1479402 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1479403 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1479407 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1479408 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1479412 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1479412 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1479416 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1479417 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1479420 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1479421 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1479424 T2367 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1479425 T2367 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1479722 T2367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1479728 T2367 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61020
[junit4] 2> 1479729 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1479730 T2367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1479730 T2367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961
[junit4] 2> 1479730 T2367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/'
[junit4] 2> 1479773 T2367 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/solr.xml
[junit4] 2> 1479867 T2367 oasc.CoreContainer.<init> New CoreContainer 109563919
[junit4] 2> 1479867 T2367 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/]
[junit4] 2> 1479869 T2367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1479870 T2367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1479870 T2367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1479871 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1479871 T2367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1479872 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1479872 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1479872 T2367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1479873 T2367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1479873 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1479893 T2367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1479894 T2367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19230/solr
[junit4] 2> 1479894 T2367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1479895 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1479898 T2387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2891abbf name:ZooKeeperConnection Watcher:127.0.0.1:19230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1479899 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1479910 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1479923 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1479925 T2389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75704e8d name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1479926 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1479928 T2367 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1479937 T2367 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1479940 T2367 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1479947 T2367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61020_
[junit4] 2> 1479949 T2367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61020_
[junit4] 2> 1479952 T2367 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1479971 T2367 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1479973 T2367 oasc.Overseer.start Overseer (id=90073481849143299-127.0.0.1:61020_-n_0000000000) starting
[junit4] 2> 1479984 T2367 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1479998 T2391 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1479998 T2367 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1480000 T2367 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1480002 T2367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1480005 T2390 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1480011 T2392 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1480012 T2392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1480013 T2392 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1481508 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1481510 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:61020_",
[junit4] 2> "base_url":"http://127.0.0.1:61020"}
[junit4] 2> 1481510 T2390 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 1481510 T2390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1481537 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1482015 T2392 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1482015 T2392 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/collection1
[junit4] 2> 1482015 T2392 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1482016 T2392 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1482016 T2392 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1482017 T2392 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/collection1/'
[junit4] 2> 1482018 T2392 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/collection1/lib/README' to classloader
[junit4] 2> 1482019 T2392 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/collection1/lib/classes/' to classloader
[junit4] 2> 1482065 T2392 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1482120 T2392 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1482221 T2392 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1482233 T2392 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1483063 T2392 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1483090 T2392 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1483094 T2392 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1483110 T2392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1483116 T2392 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1483123 T2392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1483124 T2392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1483125 T2392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1483126 T2392 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1483126 T2392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1483127 T2392 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1483127 T2392 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1483127 T2392 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374412259961/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/control/data/
[junit4] 2> 1483128 T2392 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65368aae
[junit4] 2> 1483128 T2392 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/control/data
[junit4] 2> 1483129 T2392 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/control/data/index/
[junit4] 2> 1483129 T2392 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1483129 T2392 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/control/data/index
[junit4] 2> 1483135 T2392 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@35347069 lockFactory=org.apache.lucene.store.NativeFSLockFactory@383c8221),segFN=segments_1,generation=1}
[junit4] 2> 1483135 T2392 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1483139 T2392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1483139 T2392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1483140 T2392 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1483140 T2392 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1483140 T2392 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1483141 T2392 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1483141 T2392 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1483141 T2392 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1483142 T2392 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1483161 T2392 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1483167 T2392 oass.SolrIndexSearcher.<init> Opening Searcher@f0f3f22 main
[junit4] 2> 1483169 T2392 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1483169 T2392 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1483172 T2393 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f0f3f22 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1483172 T2392 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1483173 T2392 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:61020 collection:control_collection shard:shard1
[junit4] 2> 1483173 T2392 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1483190 T2392 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1483217 T2392 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1483220 T2392 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1483221 T2392 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1483221 T2392 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:61020/collection1/
[junit4] 2> 1483221 T2392 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1483221 T2392 oasc.SyncStrategy.syncToMe http://127.0.0.1:61020/collection1/ has no replicas
[junit4] 2> 1483222 T2392 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:61020/collection1/ shard1
[junit4] 2> 1483222 T2392 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1484548 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1484599 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1484658 T2392 oasc.ZkController.register We are http://127.0.0.1:61020/collection1/ and leader is http://127.0.0.1:61020/collection1/
[junit4] 2> 1484658 T2392 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61020
[junit4] 2> 1484659 T2392 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1484659 T2392 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1484659 T2392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1484661 T2392 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1484663 T2367 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1484663 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1484664 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1484678 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1484680 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1484682 T2396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a4d685d name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1484682 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1484684 T2367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1484686 T2367 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1484967 T2367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1484971 T2367 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27902
[junit4] 2> 1484972 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1484972 T2367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1484973 T2367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213
[junit4] 2> 1484974 T2367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/'
[junit4] 2> 1485021 T2367 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/solr.xml
[junit4] 2> 1485131 T2367 oasc.CoreContainer.<init> New CoreContainer 35375164
[junit4] 2> 1485132 T2367 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/]
[junit4] 2> 1485134 T2367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1485135 T2367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1485135 T2367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1485136 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1485136 T2367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1485137 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1485137 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1485138 T2367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1485138 T2367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1485139 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1485162 T2367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1485163 T2367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19230/solr
[junit4] 2> 1485164 T2367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1485165 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1485168 T2407 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1890e79a name:ZooKeeperConnection Watcher:127.0.0.1:19230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1485169 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1485186 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1485199 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1485201 T2409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a34c24a name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1485202 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1485207 T2367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1486112 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1486114 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:61020_",
[junit4] 2> "base_url":"http://127.0.0.1:61020"}
[junit4] 2> 1486145 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1486145 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1486145 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1486210 T2367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27902_
[junit4] 2> 1486213 T2367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27902_
[junit4] 2> 1486216 T2396 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1486216 T2389 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1486217 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1486216 T2409 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1486217 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1486217 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1486233 T2410 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1486233 T2410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1486235 T2410 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1487651 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1487652 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:27902_",
[junit4] 2> "base_url":"http://127.0.0.1:27902"}
[junit4] 2> 1487653 T2390 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 1487653 T2390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1487674 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1487674 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1487674 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1488237 T2410 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1488237 T2410 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/collection1
[junit4] 2> 1488237 T2410 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1488238 T2410 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1488238 T2410 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1488239 T2410 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/collection1/'
[junit4] 2> 1488241 T2410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/collection1/lib/classes/' to classloader
[junit4] 2> 1488242 T2410 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/collection1/lib/README' to classloader
[junit4] 2> 1488306 T2410 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1488366 T2410 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1488467 T2410 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1488478 T2410 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1489113 T2410 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1489122 T2410 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1489125 T2410 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1489139 T2410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1489143 T2410 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1489147 T2410 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1489148 T2410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1489149 T2410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1489149 T2410 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1489150 T2410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1489150 T2410 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1489151 T2410 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1489151 T2410 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374412265213/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty1/
[junit4] 2> 1489151 T2410 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65368aae
[junit4] 2> 1489152 T2410 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty1
[junit4] 2> 1489152 T2410 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty1/index/
[junit4] 2> 1489152 T2410 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1489152 T2410 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty1/index
[junit4] 2> 1489155 T2410 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7658aa13 lockFactory=org.apache.lucene.store.NativeFSLockFactory@706e403c),segFN=segments_1,generation=1}
[junit4] 2> 1489155 T2410 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1489159 T2410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1489159 T2410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1489160 T2410 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1489160 T2410 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1489161 T2410 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1489161 T2410 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1489161 T2410 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1489161 T2410 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1489162 T2410 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1489175 T2410 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1489181 T2410 oass.SolrIndexSearcher.<init> Opening Searcher@140a5206 main
[junit4] 2> 1489182 T2410 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1489182 T2410 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1489185 T2411 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@140a5206 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1489186 T2410 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1489187 T2410 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27902 collection:collection1 shard:shard1
[junit4] 2> 1489187 T2410 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1489199 T2410 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1489219 T2410 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1489229 T2410 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1489229 T2410 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1489229 T2410 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27902/collection1/
[junit4] 2> 1489229 T2410 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1489230 T2410 oasc.SyncStrategy.syncToMe http://127.0.0.1:27902/collection1/ has no replicas
[junit4] 2> 1489230 T2410 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27902/collection1/ shard1
[junit4] 2> 1489230 T2410 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1490685 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1490848 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1490848 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1490848 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1490921 T2410 oasc.ZkController.register We are http://127.0.0.1:27902/collection1/ and leader is http://127.0.0.1:27902/collection1/
[junit4] 2> 1490921 T2410 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27902
[junit4] 2> 1490921 T2410 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1490922 T2410 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1490922 T2410 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1490924 T2410 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1490925 T2367 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1490926 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1490926 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1491223 T2367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1491227 T2367 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:45019
[junit4] 2> 1491228 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1491228 T2367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1491229 T2367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465
[junit4] 2> 1491229 T2367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/'
[junit4] 2> 1491267 T2367 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/solr.xml
[junit4] 2> 1491356 T2367 oasc.CoreContainer.<init> New CoreContainer 1981722869
[junit4] 2> 1491357 T2367 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/]
[junit4] 2> 1491359 T2367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1491359 T2367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1491360 T2367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1491360 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1491361 T2367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1491361 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1491361 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1491362 T2367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1491362 T2367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1491363 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1491381 T2367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1491382 T2367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19230/solr
[junit4] 2> 1491382 T2367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1491383 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1491388 T2423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@774c1fc7 name:ZooKeeperConnection Watcher:127.0.0.1:19230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1491389 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1491406 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1491419 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1491422 T2425 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7eb39a0 name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1491422 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1491434 T2367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1492401 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1492403 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:27902_",
[junit4] 2> "base_url":"http://127.0.0.1:27902"}
[junit4] 2> 1492447 T2367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45019_
[junit4] 2> 1492449 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1492450 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1492449 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1492450 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1492452 T2367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45019_
[junit4] 2> 1492464 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1492465 T2409 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1492465 T2396 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1492465 T2389 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1492465 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1492466 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1492466 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1492467 T2425 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1492489 T2426 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1492490 T2426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1492493 T2426 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1493957 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1493959 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:45019_",
[junit4] 2> "base_url":"http://127.0.0.1:45019"}
[junit4] 2> 1493959 T2390 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1493959 T2390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1493978 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1493978 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1493978 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1493978 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1494495 T2426 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1494495 T2426 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/collection1
[junit4] 2> 1494495 T2426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1494496 T2426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1494496 T2426 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1494497 T2426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/collection1/'
[junit4] 2> 1494498 T2426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/collection1/lib/README' to classloader
[junit4] 2> 1494498 T2426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/collection1/lib/classes/' to classloader
[junit4] 2> 1494539 T2426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1494626 T2426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1494727 T2426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1494738 T2426 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1495325 T2426 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1495337 T2426 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1495342 T2426 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1495357 T2426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1495362 T2426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1495367 T2426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1495368 T2426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1495369 T2426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1495369 T2426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1495370 T2426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1495370 T2426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1495371 T2426 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1495371 T2426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374412271465/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty2/
[junit4] 2> 1495371 T2426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65368aae
[junit4] 2> 1495372 T2426 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty2
[junit4] 2> 1495373 T2426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty2/index/
[junit4] 2> 1495373 T2426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1495373 T2426 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty2/index
[junit4] 2> 1495377 T2426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@389637e9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6736cd9d),segFN=segments_1,generation=1}
[junit4] 2> 1495378 T2426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1495382 T2426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1495383 T2426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1495383 T2426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1495384 T2426 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1495385 T2426 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1495385 T2426 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1495385 T2426 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1495386 T2426 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1495386 T2426 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1495404 T2426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1495411 T2426 oass.SolrIndexSearcher.<init> Opening Searcher@36421403 main
[junit4] 2> 1495413 T2426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1495413 T2426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1495417 T2427 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36421403 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1495419 T2426 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1495420 T2426 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45019 collection:collection1 shard:shard2
[junit4] 2> 1495420 T2426 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1495439 T2426 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1495459 T2426 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1495462 T2426 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1495463 T2426 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1495463 T2426 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45019/collection1/
[junit4] 2> 1495463 T2426 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1495463 T2426 oasc.SyncStrategy.syncToMe http://127.0.0.1:45019/collection1/ has no replicas
[junit4] 2> 1495464 T2426 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45019/collection1/ shard2
[junit4] 2> 1495464 T2426 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1495483 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1495501 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1495501 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1495501 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1495501 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1495522 T2426 oasc.ZkController.register We are http://127.0.0.1:45019/collection1/ and leader is http://127.0.0.1:45019/collection1/
[junit4] 2> 1495522 T2426 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45019
[junit4] 2> 1495522 T2426 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1495522 T2426 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1495523 T2426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1495525 T2426 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1495526 T2367 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1495526 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1495527 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1495875 T2367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1495878 T2367 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:12970
[junit4] 2> 1495879 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1495879 T2367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1495880 T2367 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065
[junit4] 2> 1495880 T2367 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/'
[junit4] 2> 1495923 T2367 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/solr.xml
[junit4] 2> 1496019 T2367 oasc.CoreContainer.<init> New CoreContainer 273337289
[junit4] 2> 1496020 T2367 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/]
[junit4] 2> 1496022 T2367 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1496022 T2367 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1496023 T2367 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1496023 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1496024 T2367 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1496024 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1496024 T2367 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1496025 T2367 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1496025 T2367 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1496026 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1496044 T2367 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1496045 T2367 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:19230/solr
[junit4] 2> 1496046 T2367 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1496047 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1496106 T2439 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29504327 name:ZooKeeperConnection Watcher:127.0.0.1:19230 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1496107 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1496110 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1496121 T2367 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1496124 T2441 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bb90be8 name:ZooKeeperConnection Watcher:127.0.0.1:19230/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1496124 T2367 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1496142 T2367 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1497016 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1497018 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:45019_",
[junit4] 2> "base_url":"http://127.0.0.1:45019"}
[junit4] 2> 1497059 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497059 T2441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497059 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497059 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497059 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497146 T2367 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12970_
[junit4] 2> 1497149 T2367 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12970_
[junit4] 2> 1497151 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497151 T2441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1497152 T2396 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1497152 T2409 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1497152 T2389 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1497153 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1497152 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1497153 T2425 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1497153 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1497154 T2441 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1497174 T2442 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1497175 T2442 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1497177 T2442 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1498564 T2390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1498566 T2390 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:12970_",
[junit4] 2> "base_url":"http://127.0.0.1:12970"}
[junit4] 2> 1498566 T2390 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1498566 T2390 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1498604 T2389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1498604 T2441 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1498604 T2409 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1498604 T2396 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1498604 T2425 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1499179 T2442 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1499179 T2442 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/collection1
[junit4] 2> 1499179 T2442 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1499180 T2442 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1499180 T2442 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1499181 T2442 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/collection1/'
[junit4] 2> 1499183 T2442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/collection1/lib/classes/' to classloader
[junit4] 2> 1499184 T2442 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/collection1/lib/README' to classloader
[junit4] 2> 1499248 T2442 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1499307 T2442 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1499408 T2442 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1499419 T2442 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1500054 T2442 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1500066 T2442 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1500071 T2442 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1500085 T2442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1500090 T2442 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1500095 T2442 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1500096 T2442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1500096 T2442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1500097 T2442 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1500098 T2442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1500098 T2442 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1500098 T2442 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1500099 T2442 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374412276065/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty3/
[junit4] 2> 1500099 T2442 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@65368aae
[junit4] 2> 1500100 T2442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty3
[junit4] 2> 1500100 T2442 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty3/index/
[junit4] 2> 1500100 T2442 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1500101 T2442 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374412259652/jetty3/index
[junit4] 2> 1500105 T2442 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7da77ca9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@667b23e2),segFN=segments_1,generation=1}
[junit4] 2> 1500105 T2442 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1500110 T2442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1500110 T2442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1500111 T2442 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1500112 T2442 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1500112 T2442 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1500113 T2442 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1500113 T2442 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1500114 T2442 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1500114 T2442 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1500131 T2442 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1500138 T2442 oass.SolrIndexSearcher.<init> Opening Searcher@73a53e08 main
[junit4] 2> 1500140 T2442 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1500140 T2442 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1500144 T2443 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73a53e08 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1500146 T2442 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1500146 T2442 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:12970 collection:collection1 shard:shard1
[junit4] 2> 1500147 T2442 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1500167 T2442 oasc.ZkController.register We are http://127.0.0.1:12970/collection1/ and leader is http://127.0.0.1:27902/collection1/
[junit4] 2> 1500168 T2442 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:12970
[junit4] 2> 1500168 T2442 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1500168 T2442 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C425 name=collection1 org.apache.solr.core.SolrCore@179d4440 url=http://127.0.0.1:12970/collection1 node=127.0.0.1:12970_ C425_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:12970_, base_url=http://127.0.0.1:12970}
[junit4] 2> 1500169 T2444 C425 P12970 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1500170 T2444 C425 P12970 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1500170 T2442 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1500171 T2444 C425 P12970 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1500171 T2444 C425 P12970 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1500172 T2367 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1500173 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1500173 T2444 C425 P12970 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1500173 T2367 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1500188 T2399 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1500530 T2367 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1500533 T2367 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:49115
[junit4] 2> 1500534 T2367 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1500535 T2367 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for sol
[...truncated too long message...]
pleText), n_d1=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), rnd_b=PostingsFormat(name=Memory doPackFST= false), intDefault=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), n_td1=PostingsFormat(name=MockSep), timestamp=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), id=PostingsFormat(name=Memory doPackFST= false), range_facet_sl=PostingsFormat(name=SimpleText), range_facet_si=PostingsFormat(name=MockSep), oddField_s=PostingsFormat(name=Memory doPackFST= false), sequence_i=PostingsFormat(name=Memory doPackFST= false), name=PostingsFormat(name=Memory doPackFST= false), foo_i=PostingsFormat(name=MockSep), regex_dup_B_s=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), multiDefault=PostingsFormat(name=MockSep), n_tf1=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), n_dt1=PostingsFormat(name=MockSep), genre_s=PostingsFormat(name=Memory doPackFST= false), author_t=PostingsFormat(name=MockSep), range_facet_l=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), n_ti1=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=SimpleText), _version_=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), val_i=PostingsFormat(name=Memory doPackFST= false), SubjectTerms_mfacet=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), series_t=PostingsFormat(name=MockSep), a_t=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), n_tdt1=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), regex_dup_A_s=PostingsFormat(name=Memory doPackFST= false), price=PostingsFormat(name=SimpleText), other_tl1=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), n_l1=Pulsing41(freqCutoff=19 minBlockSize=46 maxBlockSize=149), a_si=PostingsFormat(name=MockSep), inStock=PostingsFormat(name=Memory doPackFST= false)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=ru_RU, timezone=Etc/GMT+2
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=148140680,total=292683776
[junit4] 2> NOTE: All tests run in this JVM: [OverseerTest, AlternateDirectoryTest, SolrCoreTest, BinaryUpdateRequestHandlerTest, PreAnalyzedFieldTest, TestSolrQueryParserDefaultOperatorResource, TestBadConfig, MinimalSchemaTest, TestStressRecovery, TestSerializedLuceneMatchVersion, TestSearchPerf, ZkCLITest, ClusterStateUpdateTest, TestSweetSpotSimilarityFactory, BadComponentTest, LoggingHandlerTest, TestSurroundQueryParser, TestFieldTypeCollectionResource, RegexBoostProcessorTest, SliceStateUpdateTest, TestLMDirichletSimilarityFactory, TestFieldResource, TestSolrQueryParserResource, SchemaVersionSpecificBehaviorTest, QueryParsingTest, TestJoin, MoreLikeThisHandlerTest, AddSchemaFieldsUpdateProcessorFactoryTest, UnloadDistributedZkTest, TestGroupingSearch, TestAddFieldRealTimeGet, MBeansHandlerTest, SpatialFilterTest, AssignTest, SliceStateTest, TestLFUCache, DistributedSpellCheckComponentTest, SuggesterFSTTest, RecoveryZkTest, TestSchemaNameResource, TestCharFilters, JsonLoaderTest, TestDocSet, CircularListTest, SyncSliceTest, TestSolrXmlPersistence, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=8C9846E0944CD96C -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ru_RU -Dtests.timezone=Etc/GMT+2 -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([8C9846E0944CD96C]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=2889, name=recoveryCmdExecutor-1132-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([8C9846E0944CD96C]:0)
[junit4] Completed on J0 in 436.99s, 1 test, 2 errors <<< FAILURES!
[...truncated 581 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1247: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:890: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 869 ignored (2 assumptions)
Total time: 90 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 326 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/326/
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([B46EEC0187440AC0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest] at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391) at java.net.Socket.connect(Socket.java:579) at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127) at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
at __randomizedtesting.SeedInfo.seed([B46EEC0187440AC0]:0)
Build Log:
[...truncated 9628 lines...]
[junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
[junit4] 2> 1548713 T3182 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1548717 T3182 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZkTest-1374308280203
[junit4] 2> 1548719 T3182 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1548719 T3183 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1548820 T3182 oasc.ZkTestServer.run start zk server on port:11345
[junit4] 2> 1548821 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1548825 T3189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45f72fda name:ZooKeeperConnection Watcher:127.0.0.1:11345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1548826 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1548826 T3182 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1548869 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1548872 T3191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e11295 name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1548872 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1548872 T3182 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1548887 T3182 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1548890 T3182 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1548892 T3182 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1548895 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1548896 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1548906 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1548907 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1549011 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1549012 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1549014 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1549015 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1549018 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1549018 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1549026 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1549026 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1549029 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1549030 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1549032 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1549033 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1549040 T3182 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1549041 T3182 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1549323 T3182 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1549327 T3182 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11348
[junit4] 2> 1549328 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1549329 T3182 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1549329 T3182 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538
[junit4] 2> 1549329 T3182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/'
[junit4] 2> 1549364 T3182 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/solr.xml
[junit4] 2> 1549449 T3182 oasc.CoreContainer.<init> New CoreContainer 1706837251
[junit4] 2> 1549450 T3182 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/]
[junit4] 2> 1549452 T3182 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1549452 T3182 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1549453 T3182 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1549453 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1549453 T3182 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1549454 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1549454 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1549455 T3182 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1549455 T3182 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1549456 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1549473 T3182 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1549474 T3182 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11345/solr
[junit4] 2> 1549474 T3182 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1549475 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1549483 T3202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cf8b3d9 name:ZooKeeperConnection Watcher:127.0.0.1:11345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1549483 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1549495 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1549507 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1549514 T3204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c06e56c name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1549514 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1549516 T3182 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1549527 T3182 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1549536 T3182 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1549538 T3182 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11348_
[junit4] 2> 1549539 T3182 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11348_
[junit4] 2> 1549548 T3182 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1549561 T3182 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1549563 T3182 oasc.Overseer.start Overseer (id=90066667451842563-127.0.0.1:11348_-n_0000000000) starting
[junit4] 2> 1549567 T3182 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1549581 T3206 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1549581 T3182 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1549583 T3182 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1549585 T3182 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1549587 T3205 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1549592 T3207 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1549592 T3207 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1549593 T3207 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1551091 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1551092 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11348_",
[junit4] 2> "base_url":"http://127.0.0.1:11348"}
[junit4] 2> 1551092 T3205 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4] 2> 1551093 T3205 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1551110 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1551596 T3207 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1551596 T3207 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/collection1
[junit4] 2> 1551596 T3207 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1551597 T3207 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1551597 T3207 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1551598 T3207 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/collection1/'
[junit4] 2> 1551599 T3207 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/collection1/lib/README' to classloader
[junit4] 2> 1551600 T3207 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/collection1/lib/classes/' to classloader
[junit4] 2> 1551652 T3207 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1551710 T3207 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1551812 T3207 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1551822 T3207 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1552400 T3207 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1552412 T3207 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1552416 T3207 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1552428 T3207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1552432 T3207 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1552436 T3207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1552437 T3207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1552438 T3207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1552438 T3207 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1552439 T3207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1552439 T3207 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1552440 T3207 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1552440 T3207 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-controljetty-1374308280538/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/control/data/
[junit4] 2> 1552440 T3207 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cd4d8ac
[junit4] 2> 1552443 T3207 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/control/data
[junit4] 2> 1552443 T3207 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/control/data/index/
[junit4] 2> 1552444 T3207 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1552444 T3207 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/control/data/index
[junit4] 2> 1552449 T3207 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@3bdeccc8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1a6f880)),segFN=segments_1,generation=1}
[junit4] 2> 1552449 T3207 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1552453 T3207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1552453 T3207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1552454 T3207 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1552454 T3207 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1552455 T3207 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1552455 T3207 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1552456 T3207 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1552456 T3207 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1552456 T3207 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1552473 T3207 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1552480 T3207 oass.SolrIndexSearcher.<init> Opening Searcher@870b3e4 main
[junit4] 2> 1552482 T3207 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1552482 T3207 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1552485 T3208 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@870b3e4 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1552486 T3207 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1552487 T3207 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11348 collection:control_collection shard:shard1
[junit4] 2> 1552487 T3207 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1552502 T3207 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1552520 T3207 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1552522 T3207 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1552522 T3207 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1552523 T3207 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11348/collection1/
[junit4] 2> 1552523 T3207 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1552523 T3207 oasc.SyncStrategy.syncToMe http://127.0.0.1:11348/collection1/ has no replicas
[junit4] 2> 1552523 T3207 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11348/collection1/ shard1
[junit4] 2> 1552524 T3207 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1552615 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1552630 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1552682 T3207 oasc.ZkController.register We are http://127.0.0.1:11348/collection1/ and leader is http://127.0.0.1:11348/collection1/
[junit4] 2> 1552682 T3207 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11348
[junit4] 2> 1552682 T3207 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1552682 T3207 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1552683 T3207 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1552684 T3207 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1552685 T3182 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1552686 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1552686 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1552696 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4] 2> 1552697 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1552699 T3211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28b61c2f name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1552699 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1552700 T3182 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1552702 T3182 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1552970 T3182 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1552977 T3182 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11352
[junit4] 2> 1552978 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1552979 T3182 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1552979 T3182 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187
[junit4] 2> 1552980 T3182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/'
[junit4] 2> 1553015 T3182 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/solr.xml
[junit4] 2> 1553100 T3182 oasc.CoreContainer.<init> New CoreContainer 1579067146
[junit4] 2> 1553100 T3182 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/]
[junit4] 2> 1553102 T3182 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1553103 T3182 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1553103 T3182 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1553103 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1553104 T3182 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1553104 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1553105 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1553105 T3182 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1553106 T3182 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1553106 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1553121 T3182 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1553122 T3182 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11345/solr
[junit4] 2> 1553123 T3182 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1553124 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1553127 T3222 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f7a5bad name:ZooKeeperConnection Watcher:127.0.0.1:11345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1553128 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1553139 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1553149 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1553151 T3224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c3137b5 name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1553152 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1553161 T3182 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1554135 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1554136 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11348_",
[junit4] 2> "base_url":"http://127.0.0.1:11348"}
[junit4] 2> 1554197 T3182 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11352_
[junit4] 2> 1554199 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1554199 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1554199 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1554257 T3182 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11352_
[junit4] 2> 1554277 T3211 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1554277 T3204 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1554278 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1554277 T3224 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1554278 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1554278 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1554328 T3225 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1554328 T3225 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1554353 T3225 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1555761 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1555762 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11352_",
[junit4] 2> "base_url":"http://127.0.0.1:11352"}
[junit4] 2> 1555762 T3205 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4] 2> 1555763 T3205 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1555779 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1555779 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1555779 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1556355 T3225 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1556355 T3225 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/collection1
[junit4] 2> 1556355 T3225 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1556356 T3225 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1556356 T3225 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1556357 T3225 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/collection1/'
[junit4] 2> 1556358 T3225 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/collection1/lib/README' to classloader
[junit4] 2> 1556359 T3225 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/collection1/lib/classes/' to classloader
[junit4] 2> 1556411 T3225 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1556468 T3225 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1556570 T3225 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1556580 T3225 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1557149 T3225 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1557161 T3225 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1557165 T3225 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1557177 T3225 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1557182 T3225 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1557186 T3225 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1557187 T3225 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1557187 T3225 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1557187 T3225 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1557188 T3225 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1557188 T3225 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1557189 T3225 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1557189 T3225 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty1-1374308284187/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty1/
[junit4] 2> 1557189 T3225 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cd4d8ac
[junit4] 2> 1557192 T3225 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty1
[junit4] 2> 1557193 T3225 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty1/index/
[junit4] 2> 1557193 T3225 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1557193 T3225 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty1/index
[junit4] 2> 1557197 T3225 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@43694faa lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b407671)),segFN=segments_1,generation=1}
[junit4] 2> 1557197 T3225 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1557201 T3225 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1557202 T3225 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1557202 T3225 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1557203 T3225 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1557204 T3225 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1557204 T3225 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1557204 T3225 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1557205 T3225 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1557205 T3225 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1557221 T3225 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1557229 T3225 oass.SolrIndexSearcher.<init> Opening Searcher@2b132235 main
[junit4] 2> 1557230 T3225 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1557230 T3225 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1557234 T3226 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b132235 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1557236 T3225 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1557236 T3225 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11352 collection:collection1 shard:shard1
[junit4] 2> 1557237 T3225 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1557254 T3225 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1557279 T3225 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1557281 T3225 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1557282 T3225 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1557282 T3225 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11352/collection1/
[junit4] 2> 1557282 T3225 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1557282 T3225 oasc.SyncStrategy.syncToMe http://127.0.0.1:11352/collection1/ has no replicas
[junit4] 2> 1557283 T3225 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11352/collection1/ shard1
[junit4] 2> 1557283 T3225 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1557284 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1557288 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1557288 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1557288 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1558791 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1558808 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1558808 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1558808 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1558825 T3225 oasc.ZkController.register We are http://127.0.0.1:11352/collection1/ and leader is http://127.0.0.1:11352/collection1/
[junit4] 2> 1558825 T3225 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11352
[junit4] 2> 1558825 T3225 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1558825 T3225 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1558826 T3225 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1558827 T3225 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1558828 T3182 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1558829 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1558829 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1559114 T3182 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1559117 T3182 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11358
[junit4] 2> 1559118 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1559118 T3182 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1559119 T3182 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326
[junit4] 2> 1559119 T3182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/'
[junit4] 2> 1559155 T3182 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/solr.xml
[junit4] 2> 1559241 T3182 oasc.CoreContainer.<init> New CoreContainer 73082665
[junit4] 2> 1559242 T3182 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/]
[junit4] 2> 1559244 T3182 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1559244 T3182 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1559245 T3182 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1559245 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1559245 T3182 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1559246 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1559246 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1559247 T3182 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1559247 T3182 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1559248 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1559265 T3182 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1559266 T3182 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11345/solr
[junit4] 2> 1559266 T3182 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1559267 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1559337 T3238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@512d5b53 name:ZooKeeperConnection Watcher:127.0.0.1:11345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1559338 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1559376 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1559386 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1559388 T3240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c6b32cf name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1559388 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1559435 T3182 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1560312 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1560313 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11352_",
[junit4] 2> "base_url":"http://127.0.0.1:11352"}
[junit4] 2> 1560331 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1560331 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1560331 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1560331 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1560438 T3182 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11358_
[junit4] 2> 1560439 T3182 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11358_
[junit4] 2> 1560442 T3224 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1560442 T3240 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1560442 T3204 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1560442 T3211 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1560442 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1560442 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1560442 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1560443 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1560450 T3241 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1560450 T3241 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1560451 T3241 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1561836 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1561837 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11358_",
[junit4] 2> "base_url":"http://127.0.0.1:11358"}
[junit4] 2> 1561837 T3205 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1561837 T3205 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1561853 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1561854 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1561854 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1561854 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1562453 T3241 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1562453 T3241 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/collection1
[junit4] 2> 1562453 T3241 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1562454 T3241 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1562454 T3241 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1562455 T3241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/collection1/'
[junit4] 2> 1562456 T3241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/collection1/lib/README' to classloader
[junit4] 2> 1562457 T3241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/collection1/lib/classes/' to classloader
[junit4] 2> 1562509 T3241 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1562572 T3241 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1562674 T3241 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1562684 T3241 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1563242 T3241 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1563250 T3241 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1563253 T3241 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1563263 T3241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1563266 T3241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1563269 T3241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1563270 T3241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1563271 T3241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1563271 T3241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1563272 T3241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1563272 T3241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1563272 T3241 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1563272 T3241 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty2-1374308290326/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty2/
[junit4] 2> 1563272 T3241 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cd4d8ac
[junit4] 2> 1563274 T3241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty2
[junit4] 2> 1563275 T3241 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty2/index/
[junit4] 2> 1563275 T3241 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1563275 T3241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty2/index
[junit4] 2> 1563278 T3241 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@782bb7e3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2aad9e1d)),segFN=segments_1,generation=1}
[junit4] 2> 1563278 T3241 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1563281 T3241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1563281 T3241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1563281 T3241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1563282 T3241 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1563282 T3241 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1563282 T3241 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1563283 T3241 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1563283 T3241 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1563283 T3241 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1563294 T3241 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1563299 T3241 oass.SolrIndexSearcher.<init> Opening Searcher@5ba6252 main
[junit4] 2> 1563299 T3241 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1563300 T3241 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1563303 T3242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ba6252 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1563304 T3241 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1563304 T3241 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11358 collection:collection1 shard:shard2
[junit4] 2> 1563304 T3241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1563316 T3241 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1563336 T3241 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1563348 T3241 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1563348 T3241 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1563349 T3241 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:11358/collection1/
[junit4] 2> 1563349 T3241 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1563349 T3241 oasc.SyncStrategy.syncToMe http://127.0.0.1:11358/collection1/ has no replicas
[junit4] 2> 1563349 T3241 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:11358/collection1/ shard2
[junit4] 2> 1563350 T3241 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1563364 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1563369 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1563369 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1563369 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1563369 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1563414 T3241 oasc.ZkController.register We are http://127.0.0.1:11358/collection1/ and leader is http://127.0.0.1:11358/collection1/
[junit4] 2> 1563414 T3241 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11358
[junit4] 2> 1563414 T3241 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1563414 T3241 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1563415 T3241 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1563416 T3241 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1563417 T3182 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1563418 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1563418 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1563600 T3182 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1563602 T3182 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11361
[junit4] 2> 1563602 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1563602 T3182 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1563603 T3182 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910
[junit4] 2> 1563603 T3182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/'
[junit4] 2> 1563626 T3182 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/solr.xml
[junit4] 2> 1563681 T3182 oasc.CoreContainer.<init> New CoreContainer 2009998317
[junit4] 2> 1563682 T3182 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/]
[junit4] 2> 1563683 T3182 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1563684 T3182 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1563684 T3182 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1563684 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1563684 T3182 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1563685 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1563685 T3182 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1563685 T3182 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1563686 T3182 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1563686 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4] 2> 1563697 T3182 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1563697 T3182 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:11345/solr
[junit4] 2> 1563698 T3182 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1563698 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1563701 T3254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76d13556 name:ZooKeeperConnection Watcher:127.0.0.1:11345 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1563702 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1563704 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4] 2> 1563716 T3182 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1563718 T3256 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7cd7cb15 name:ZooKeeperConnection Watcher:127.0.0.1:11345/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1563718 T3182 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1563728 T3182 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1564731 T3182 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11361_
[junit4] 2> 1564758 T3182 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11361_
[junit4] 2> 1564761 T3224 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1564761 T3240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1564761 T3204 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1564761 T3256 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1564761 T3211 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1564762 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564761 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564761 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564762 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564777 T3257 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1564777 T3257 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1564778 T3257 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1564874 T3205 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1564875 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node2",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "roles":null,
[junit4] 2> "state":"active",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11358_",
[junit4] 2> "base_url":"http://127.0.0.1:11358"}
[junit4] 2> 1564878 T3205 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "core_node_name":"core_node3",
[junit4] 2> "numShards":"2",
[junit4] 2> "shard":null,
[junit4] 2> "roles":null,
[junit4] 2> "state":"down",
[junit4] 2> "shard_state":"active",
[junit4] 2> "core":"collection1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard_range":null,
[junit4] 2> "node_name":"127.0.0.1:11361_",
[junit4] 2> "base_url":"http://127.0.0.1:11361"}
[junit4] 2> 1564879 T3205 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1564879 T3205 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1564883 T3256 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564883 T3204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564883 T3211 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564883 T3240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1564883 T3224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1565780 T3257 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1565780 T3257 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/collection1
[junit4] 2> 1565780 T3257 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1565781 T3257 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1565781 T3257 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1565782 T3257 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/collection1/'
[junit4] 2> 1565784 T3257 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/collection1/lib/classes/' to classloader
[junit4] 2> 1565784 T3257 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/collection1/lib/README' to classloader
[junit4] 2> 1565836 T3257 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1565893 T3257 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1565995 T3257 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1566005 T3257 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1566580 T3257 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1566592 T3257 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1566596 T3257 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1566607 T3257 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1566612 T3257 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1566616 T3257 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1566617 T3257 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1566618 T3257 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1566618 T3257 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1566619 T3257 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1566619 T3257 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1566619 T3257 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 1566620 T3257 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty3-1374308294910/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty3/
[junit4] 2> 1566620 T3257 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cd4d8ac
[junit4] 2> 1566622 T3257 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty3
[junit4] 2> 1566623 T3257 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty3/index/
[junit4] 2> 1566623 T3257 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1566624 T3257 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZkTest-1374308280203/jetty3/index
[junit4] 2> 1566627 T3257 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@5cb52c5a lockFactory=org.apache.lucene.store.NativeFSLockFactory@9c1f9c4)),segFN=segments_1,generation=1}
[junit4] 2> 1566627 T3257 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1566632 T3257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1566632 T3257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1566633 T3257 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1566633 T3257 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1566634 T3257 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1566634 T3257 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1566635 T3257 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1566635 T3257 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1566636 T3257 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1566652 T3257 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1566659 T3257 oass.SolrIndexSearcher.<init> Opening Searcher@5369867 main
[junit4] 2> 1566660 T3257 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1566660 T3257 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1566664 T3258 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5369867 main{StandardDirectoryReader(segments_1:1)}
[junit4] 2> 1566666 T3257 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1566666 T3257 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:11361 collection:collection1 shard:shard1
[junit4] 2> 1566666 T3257 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4] 2> 1566686 T3257 oasc.ZkController.register We are http://127.0.0.1:11361/collection1/ and leader is http://127.0.0.1:11352/collection1/
[junit4] 2> 1566686 T3257 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:11361
[junit4] 2> 1566686 T3257 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1566686 T3257 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C1534 name=collection1 org.apache.solr.core.SolrCore@3e67038 url=http://127.0.0.1:11361/collection1 node=127.0.0.1:11361_ C1534_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:11361_, base_url=http://127.0.0.1:11361}
[junit4] 2> 1566687 T3259 C1534 P11361 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1566687 T3257 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1566688 T3259 C1534 P11361 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1566688 T3259 C1534 P11361 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1566688 T3259 C1534 P11361 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1566689 T3182 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
[junit4] 2> 1566689 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1566690 T3259 C1534 P11361 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1566690 T3182 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4] 2> 1566707 T3214 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1566972 T3182 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1566974 T3182 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11365
[junit4] 2> 1566975 T3182 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1566976 T3182 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1566976 T3182 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZkTest-jetty4-1374308298190
[junit4] 2> 1566977 T3182 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/./org.apache.so
[...truncated too long message...]
RequestDirector.java:480)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] 2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4] 2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_KW, timezone=PLT
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=16,threads=2,free=119594072,total=369164288
[junit4] 2> NOTE: All tests run in this JVM: [QueryEqualityTest, TestDocumentBuilder, StatelessScriptUpdateProcessorFactoryTest, TestSchemaNameResource, TestFaceting, TestSolrDeletionPolicy2, SpatialFilterTest, QueryParsingTest, StressHdfsTest, TermVectorComponentTest, DocumentBuilderTest, MinimalSchemaTest, TestSolrIndexConfig, TestStressRecovery, DistributedQueryElevationComponentTest, TestExtendedDismaxParser, BadComponentTest, SoftAutoCommitTest, TestFieldResource, ResourceLoaderTest, TestReplicationHandler, TestQueryUtils, JsonLoaderTest, TestSolrQueryParserDefaultOperatorResource, TestSchemaVersionResource, TestMaxScoreQueryParser, TestIBSimilarityFactory, TestFastWriter, WordBreakSolrSpellCheckerTest, TestAddFieldRealTimeGet, TestFunctionQuery, PeerSyncTest, TestDefaultSearchFieldResource, TestCopyFieldCollectionResource, SolrIndexSplitterTest, TestSolrQueryParserResource, CachingDirectoryFactoryTest, DocValuesTest, PluginInfoTest, TermsComponentTest, FieldMutatingUpdateProcessorTest, TestBadConfig, AnalysisAfterCoreReloadTest, TestSweetSpotSimilarityFactory, TestDynamicFieldResource, LukeRequestHandlerTest, InfoHandlerTest, ShardRoutingTest, TestConfig, ShowFileRequestHandlerTest, SliceStateUpdateTest, PreAnalyzedUpdateProcessorTest, ResponseLogComponentTest, SolrInfoMBeanTest, TestPhraseSuggestions, TestRealTimeGet, TestPseudoReturnFields, CoreAdminHandlerTest, TestSearchPerf, ParsingFieldUpdateProcessorsTest, ChaosMonkeyNothingIsSafeTest, ScriptEngineTest, TestWriterPerf, RegexBoostProcessorTest, UnloadDistributedZkTest, PreAnalyzedFieldTest, TestFuzzyAnalyzedSuggestions, TestSerializedLuceneMatchVersion, OverseerCollectionProcessorTest, TestReversedWildcardFilterFactory, TestLMDirichletSimilarityFactory, TestSurroundQueryParser, TestGroupingSearch, TestSolrCoreProperties, SliceStateTest, SuggesterFSTTest, OverseerTest, TestRandomDVFaceting, BasicDistributedZkTest]
[junit4] 2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicDistributedZkTest -Dtests.seed=B46EEC0187440AC0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_KW -Dtests.timezone=PLT -Dtests.file.encoding=US-ASCII
[junit4] ERROR 0.00s J0 | BasicDistributedZkTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:
[junit4] > 1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([B46EEC0187440AC0]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3697, name=recoveryCmdExecutor-1594-thread-1, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4] > at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
[junit4] > at java.net.Socket.connect(Socket.java:579)
[junit4] > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4] > at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:291)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4] > at __randomizedtesting.SeedInfo.seed([B46EEC0187440AC0]:0)
[junit4] Completed on J0 in 415.13s, 1 test, 2 errors <<< FAILURES!
[...truncated 479 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:396: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:369: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:181: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1247: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:890: There were test failures: 318 suites, 1342 tests, 2 suite-level errors, 565 ignored (3 assumptions)
Total time: 89 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 325 - Still Failing
Posted by Adrien Grand <jp...@gmail.com>.
I will dig.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org